overloading of internal priority calculations in agenda

[Adam Spiers <orgmode@adamspiers.org>]

Hi all,

I'm currently working on adding a feature to org-agenda which allows
manual ordering of entries in combination with the existing automatic
ordering (as dictated by `org-agenda-sorting-strategy').

During my investigations I noticed that while `org-get-priority' converts
[#B] style cookies into a numeric priority which is a multiple of
1000, further adjustments are made in functions like
`org-agenda-get-scheduled' before adding this numeric priority as a
text property on the entry:

     'priority (if habitp (org-habit-get-priority habitp)
                 (+ 99 diff (org-get-priority item)))

In this case `diff' refers to the number of days between now and when
the item was scheduled.  A slightly different calculation is made in
`org-agenda-get-timestamps':

     (org-add-props item props
       'priority (if habit?
                     (org-habit-get-priority (org-habit-parse-todo))
                   (org-get-priority item))

I further noticed that this overloading of the internal priority by
including timestamp and habit data causes disruption to the behaviour
I imagine most users would expect from `org-agenda-sorting-strategy'.
For example, if you have `priority-down' as the first entry in the
`agenda' section and `category-keep' as the second, then differences
in the SCHEDULED timestamp are included in the priority calculation
and can therefore prevent sorting of two adjacent [#B] items by
category.  This seems like a bug to me, or at least breaks the
Principle of Least Surprise.

I did some git archaelogy and found that the first ever git commit
4be4c562 (for release 4.12a) already includes
`org-agenda-sorting-strategy', but at that point, time-{up,down} were
the only time-related sorting criteria available.

I was also wondering where the magic 99 number above came from, and I
found that the same (first ever) commit introduced the following
priority calculation:

         (+ (- 5 diff) (org-get-priority txt))

Subsequently, commit 70b6cc5d (for release 5.10a) changes this to:

         (+ 94 (- 5 diff) (org-get-priority txt))

and then commit 69ec6258 (on 2016-11-25) changes it to

         (+ 99 diff (org-get-priority item))

Given that `org-agenda-sorting-strategy' now supports all manner of
sorting criteria, many of which are time-sensitive, I would like to
know if there is any reason not to remove this overloading of the
priority calculation, i.e. decoupling it to depend purely on the
result of `org-get-priority' and `org-habit-get-priority'?

If fact, perhaps we could go one step further and add support for new
habit-priority-{up,down} sorters to `org-agenda-sorting-strategy', so
that the priority-{up,down} sorters sort purely by the priority cookie
and nothing else?

Thanks!
Adam