overloading of internal priority calculations in agenda

overloading of internal priority calculations in agenda

[Adam Spiers]

Hi all,

I'm currently working on adding a feature to org-agenda which allows
manual ordering of entries in combination with the existing automatic
ordering (as dictated by `org-agenda-sorting-strategy').

During my investigations I noticed that while `org-get-priority' converts
[#B] style cookies into a numeric priority which is a multiple of
1000, further adjustments are made in functions like
`org-agenda-get-scheduled' before adding this numeric priority as a
text property on the entry:

     'priority (if habitp (org-habit-get-priority habitp)
                 (+ 99 diff (org-get-priority item)))

In this case `diff' refers to the number of days between now and when
the item was scheduled.  A slightly different calculation is made in
`org-agenda-get-timestamps':

     (org-add-props item props
       'priority (if habit?
                     (org-habit-get-priority (org-habit-parse-todo))
                   (org-get-priority item))

I further noticed that this overloading of the internal priority by
including timestamp and habit data causes disruption to the behaviour
I imagine most users would expect from `org-agenda-sorting-strategy'.
For example, if you have `priority-down' as the first entry in the
`agenda' section and `category-keep' as the second, then differences
in the SCHEDULED timestamp are included in the priority calculation
and can therefore prevent sorting of two adjacent [#B] items by
category.  This seems like a bug to me, or at least breaks the
Principle of Least Surprise.

I did some git archaelogy and found that the first ever git commit
4be4c562 (for release 4.12a) already includes
`org-agenda-sorting-strategy', but at that point, time-{up,down} were
the only time-related sorting criteria available.

I was also wondering where the magic 99 number above came from, and I
found that the same (first ever) commit introduced the following
priority calculation:

         (+ (- 5 diff) (org-get-priority txt))

Subsequently, commit 70b6cc5d (for release 5.10a) changes this to:

         (+ 94 (- 5 diff) (org-get-priority txt))

and then commit 69ec6258 (on 2016-11-25) changes it to

         (+ 99 diff (org-get-priority item))

Given that `org-agenda-sorting-strategy' now supports all manner of
sorting criteria, many of which are time-sensitive, I would like to
know if there is any reason not to remove this overloading of the
priority calculation, i.e. decoupling it to depend purely on the
result of `org-get-priority' and `org-habit-get-priority'?

If fact, perhaps we could go one step further and add support for new
habit-priority-{up,down} sorters to `org-agenda-sorting-strategy', so
that the priority-{up,down} sorters sort purely by the priority cookie
and nothing else?

Thanks!
Adam

Re: overloading of internal priority calculations in agenda

[Adam Spiers]

Hi again,

On Wed, Dec 02, 2020 at 02:20:53PM +0000, Adam Spiers wrote: 
>Hi all,
>
>I'm currently working on adding a feature to org-agenda which allows 
>manual ordering of entries in combination with the existing automatic 
>ordering (as dictated by `org-agenda-sorting-strategy'). 
>
>During my investigations I noticed that while `org-get-priority' converts 
>[#B] style cookies into a numeric priority which is a multiple of 
>1000, further adjustments are made in functions like 
>`org-agenda-get-scheduled' before adding this numeric priority as a 
>text property on the entry: 
>
>    'priority (if habitp (org-habit-get-priority habitp)
>                (+ 99 diff (org-get-priority item)))
>
>In this case `diff' refers to the number of days between now and when 
>the item was scheduled.  A slightly different calculation is made in 
>`org-agenda-get-timestamps': 
>
>    (org-add-props item props
>      'priority (if habit?
>                    (org-habit-get-priority (org-habit-parse-todo))
>                  (org-get-priority item))
>
>I further noticed that this overloading of the internal priority by 
>including timestamp and habit data causes disruption to the behaviour 
>I imagine most users would expect from `org-agenda-sorting-strategy'. 
>For example, if you have `priority-down' as the first entry in the 
>`agenda' section and `category-keep' as the second, then differences 
>in the SCHEDULED timestamp are included in the priority calculation 
>and can therefore prevent sorting of two adjacent [#B] items by 
>category.  This seems like a bug to me, or at least breaks the 
>Principle of Least Surprise. 

[snipped]

>Given that `org-agenda-sorting-strategy' now supports all manner of 
>sorting criteria, many of which are time-sensitive, I would like to 
>know if there is any reason not to remove this overloading of the 
>priority calculation, i.e. decoupling it to depend purely on the 
>result of `org-get-priority' and `org-habit-get-priority'? 
>
>If fact, perhaps we could go one step further and add support for new 
>habit-priority-{up,down} sorters to `org-agenda-sorting-strategy', so 
>that the priority-{up,down} sorters sort purely by the priority cookie 
>and nothing else? 

Gently bumping this as I didn't get any replies yet.  I would like to 
continue working on a solution, but obviously don't want to waste time 
on something which would be rejected. 

If it is considered important to preserve the exact behaviour 
currently offered by `org-agenda-sorting-strategy' then I would 
propose the following: 

- Keep the existing priority-{up,down} which combine priority cookies
   with timestamp data and the result from `org-habit-get-priority',
   but probably also deprecate it and remove it from the default value.

- Introduce new priority-cookie-{up,down} sorters which operate purely
   on [#A] and [#1] style priority cookies and nothing else.

This would facilitate decoupling of the sortable criteria whilst 
remaining backwards compatible. 

Does this sound reasonable?  I am keen to proceed very soon (ideally 
over the Xmas break).  I have already written some new ert tests for 
`org-agenda-sorting-strategy' which would be included in any submitted 
patches. 

Thanks!
Adam

Re: overloading of internal priority calculations in agenda

[Samuel Wales]

if my opinion is worth anything [perhaps not much here :]], i like
your proposals and the idea of being able to re-sort an existing
agenda assuming that is your goal.

i don't use any priority sorting except in user-customizable but it
makes sense to decouple them for those who do.  and i frequently want
to differently sort an existing agenda view.


On 12/22/20, Adam Spiers <orgmode@adamspiers.org> wrote:
> Hi again,
>
> On Wed, Dec 02, 2020 at 02:20:53PM +0000, Adam Spiers wrote:
>>Hi all,
>>
>>I'm currently working on adding a feature to org-agenda which allows
>>manual ordering of entries in combination with the existing automatic
>>ordering (as dictated by `org-agenda-sorting-strategy').
>>
>>During my investigations I noticed that while `org-get-priority' converts
>>[#B] style cookies into a numeric priority which is a multiple of
>>1000, further adjustments are made in functions like
>>`org-agenda-get-scheduled' before adding this numeric priority as a
>>text property on the entry:
>>
>>    'priority (if habitp (org-habit-get-priority habitp)
>>                (+ 99 diff (org-get-priority item)))
>>
>>In this case `diff' refers to the number of days between now and when
>>the item was scheduled.  A slightly different calculation is made in
>>`org-agenda-get-timestamps':
>>
>>    (org-add-props item props
>>      'priority (if habit?
>>                    (org-habit-get-priority (org-habit-parse-todo))
>>                  (org-get-priority item))
>>
>>I further noticed that this overloading of the internal priority by
>>including timestamp and habit data causes disruption to the behaviour
>>I imagine most users would expect from `org-agenda-sorting-strategy'.
>>For example, if you have `priority-down' as the first entry in the
>>`agenda' section and `category-keep' as the second, then differences
>>in the SCHEDULED timestamp are included in the priority calculation
>>and can therefore prevent sorting of two adjacent [#B] items by
>>category.  This seems like a bug to me, or at least breaks the
>>Principle of Least Surprise.
>
> [snipped]
>
>>Given that `org-agenda-sorting-strategy' now supports all manner of
>>sorting criteria, many of which are time-sensitive, I would like to
>>know if there is any reason not to remove this overloading of the
>>priority calculation, i.e. decoupling it to depend purely on the
>>result of `org-get-priority' and `org-habit-get-priority'?
>>
>>If fact, perhaps we could go one step further and add support for new
>>habit-priority-{up,down} sorters to `org-agenda-sorting-strategy', so
>>that the priority-{up,down} sorters sort purely by the priority cookie
>>and nothing else?
>
> Gently bumping this as I didn't get any replies yet.  I would like to
> continue working on a solution, but obviously don't want to waste time
> on something which would be rejected.
>
> If it is considered important to preserve the exact behaviour
> currently offered by `org-agenda-sorting-strategy' then I would
> propose the following:
>
> - Keep the existing priority-{up,down} which combine priority cookies
>    with timestamp data and the result from `org-habit-get-priority',
>    but probably also deprecate it and remove it from the default value.
>
> - Introduce new priority-cookie-{up,down} sorters which operate purely
>    on [#A] and [#1] style priority cookies and nothing else.
>
> This would facilitate decoupling of the sortable criteria whilst
> remaining backwards compatible.
>
> Does this sound reasonable?  I am keen to proceed very soon (ideally
> over the Xmas break).  I have already written some new ert tests for
> `org-agenda-sorting-strategy' which would be included in any submitted
> patches.
>
> Thanks!
> Adam
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: overloading of internal priority calculations in agenda

[Adam Spiers]

[-- Attachment #1: Type: text/plain, Size: 4152 bytes --]

Thanks a lot - appreciate the feedback!

On Tue, 22 Dec 2020 at 23:38, Samuel Wales <samologist@gmail.com> wrote:

> if my opinion is worth anything [perhaps not much here :]], i like
> your proposals and the idea of being able to re-sort an existing
> agenda assuming that is your goal.
>
> i don't use any priority sorting except in user-customizable but it
> makes sense to decouple them for those who do.  and i frequently want
> to differently sort an existing agenda view.
>
>
> On 12/22/20, Adam Spiers <orgmode@adamspiers.org> wrote:
> > Hi again,
> >
> > On Wed, Dec 02, 2020 at 02:20:53PM +0000, Adam Spiers wrote:
> >>Hi all,
> >>
> >>I'm currently working on adding a feature to org-agenda which allows
> >>manual ordering of entries in combination with the existing automatic
> >>ordering (as dictated by `org-agenda-sorting-strategy').
> >>
> >>During my investigations I noticed that while `org-get-priority' converts
> >>[#B] style cookies into a numeric priority which is a multiple of
> >>1000, further adjustments are made in functions like
> >>`org-agenda-get-scheduled' before adding this numeric priority as a
> >>text property on the entry:
> >>
> >>    'priority (if habitp (org-habit-get-priority habitp)
> >>                (+ 99 diff (org-get-priority item)))
> >>
> >>In this case `diff' refers to the number of days between now and when
> >>the item was scheduled.  A slightly different calculation is made in
> >>`org-agenda-get-timestamps':
> >>
> >>    (org-add-props item props
> >>      'priority (if habit?
> >>                    (org-habit-get-priority (org-habit-parse-todo))
> >>                  (org-get-priority item))
> >>
> >>I further noticed that this overloading of the internal priority by
> >>including timestamp and habit data causes disruption to the behaviour
> >>I imagine most users would expect from `org-agenda-sorting-strategy'.
> >>For example, if you have `priority-down' as the first entry in the
> >>`agenda' section and `category-keep' as the second, then differences
> >>in the SCHEDULED timestamp are included in the priority calculation
> >>and can therefore prevent sorting of two adjacent [#B] items by
> >>category.  This seems like a bug to me, or at least breaks the
> >>Principle of Least Surprise.
> >
> > [snipped]
> >
> >>Given that `org-agenda-sorting-strategy' now supports all manner of
> >>sorting criteria, many of which are time-sensitive, I would like to
> >>know if there is any reason not to remove this overloading of the
> >>priority calculation, i.e. decoupling it to depend purely on the
> >>result of `org-get-priority' and `org-habit-get-priority'?
> >>
> >>If fact, perhaps we could go one step further and add support for new
> >>habit-priority-{up,down} sorters to `org-agenda-sorting-strategy', so
> >>that the priority-{up,down} sorters sort purely by the priority cookie
> >>and nothing else?
> >
> > Gently bumping this as I didn't get any replies yet.  I would like to
> > continue working on a solution, but obviously don't want to waste time
> > on something which would be rejected.
> >
> > If it is considered important to preserve the exact behaviour
> > currently offered by `org-agenda-sorting-strategy' then I would
> > propose the following:
> >
> > - Keep the existing priority-{up,down} which combine priority cookies
> >    with timestamp data and the result from `org-habit-get-priority',
> >    but probably also deprecate it and remove it from the default value.
> >
> > - Introduce new priority-cookie-{up,down} sorters which operate purely
> >    on [#A] and [#1] style priority cookies and nothing else.
> >
> > This would facilitate decoupling of the sortable criteria whilst
> > remaining backwards compatible.
> >
> > Does this sound reasonable?  I am keen to proceed very soon (ideally
> > over the Xmas break).  I have already written some new ert tests for
> > `org-agenda-sorting-strategy' which would be included in any submitted
> > patches.
> >
> > Thanks!
> > Adam
> >
> >
>
>
> --
> The Kafka Pandemic
>
> Please learn what misopathy is.
>
> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>

[-- Attachment #2: Type: text/html, Size: 5405 bytes --]

Re: overloading of internal priority calculations in agenda

[Jack Kamm]

Hi Adam,

> I further noticed that this overloading of the internal priority by
> including timestamp and habit data causes disruption to the behaviour
> I imagine most users would expect from `org-agenda-sorting-strategy'.
> For example, if you have `priority-down' as the first entry in the
> `agenda' section and `category-keep' as the second, then differences
> in the SCHEDULED timestamp are included in the priority calculation
> and can therefore prevent sorting of two adjacent [#B] items by
> category.  This seems like a bug to me, or at least breaks the
> Principle of Least Surprise.

I just ran into this issue you highlight here.

In particular, I was trying to set the org-agenda-sorting-strategy to

(priority-down scheduled-down)

i.e., sorting by priority (highest first), and then within priority,
sorting by scheduled (most recent first).

However, the fact that the priority includes the scheduled timestamp
makes this sorting strategy impossible.

I agree this seems like a bug, in that it contradicts the written
documentation as far as I can tell (for example, the *Help* for
org-agenda-sorting-strategy mentions nothing of the fact that the
priority includes the scheduled timestamp, and I don't see anything
about it in the *Info* either).

I imagine that many have gotten used to the default behavior of sort by
highest priority, then by earliest scheduled timestamp, but we could
keep this default behavior by adding "scheduled-up" after
"priority-down" in org-agenda-sorting-strategy, as you allude.

Jack

Re: overloading of internal priority calculations in agenda

[Adam Spiers]

Thanks Jack for the helpful response and support of my assessment.

I do intend to fix this as part of my ongoing (but currently delayed)
work on improving agenda sorting and adding an option to manually sort.

On Tue, 9 Mar 2021 at 07:07, Jack Kamm <jackkamm@gmail.com> wrote:
>
> Hi Adam,
>
> > I further noticed that this overloading of the internal priority by
> > including timestamp and habit data causes disruption to the behaviour
> > I imagine most users would expect from `org-agenda-sorting-strategy'.
> > For example, if you have `priority-down' as the first entry in the
> > `agenda' section and `category-keep' as the second, then differences
> > in the SCHEDULED timestamp are included in the priority calculation
> > and can therefore prevent sorting of two adjacent [#B] items by
> > category.  This seems like a bug to me, or at least breaks the
> > Principle of Least Surprise.
>
> I just ran into this issue you highlight here.
>
> In particular, I was trying to set the org-agenda-sorting-strategy to
>
> (priority-down scheduled-down)
>
> i.e., sorting by priority (highest first), and then within priority,
> sorting by scheduled (most recent first).
>
> However, the fact that the priority includes the scheduled timestamp
> makes this sorting strategy impossible.
>
> I agree this seems like a bug, in that it contradicts the written
> documentation as far as I can tell (for example, the *Help* for
> org-agenda-sorting-strategy mentions nothing of the fact that the
> priority includes the scheduled timestamp, and I don't see anything
> about it in the *Info* either).
>
> I imagine that many have gotten used to the default behavior of sort by
> highest priority, then by earliest scheduled timestamp, but we could
> keep this default behavior by adding "scheduled-up" after
> "priority-down" in org-agenda-sorting-strategy, as you allude.
>
> Jack