[PATCH] Fix alphabetic sorting for headlines, tags

[PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

* org.el (org-sort-entries): Use collated sorting.
(org-tags-sort-function): Use collated sorting.
(org-string-collate-greaterp): Add helper-function to use as defcustom
option, since there is no ‘string-collate-greaterp’ in Emacs.

‘org-sort-entries’ and ‘org-tags-sort-function’ advertise alphabetic
sorting, but actually sort based only on character code.  This
produces non-alphabetic orderings of strings in non-ASCII locales.

E. g., German Umlauts “Ä Ü Ö” are alphabetically sorted as if they
were “A U O”, whereas sorting based on character-code will place them
after “Z”, which is unexpected.
---
 lisp/org.el | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 688e48bcc..d54c410b0 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3558,8 +3558,8 @@ (defcustom org-tags-sort-function nil
   :group 'org-tags
   :type '(choice
 	  (const :tag "No sorting" nil)
-	  (const :tag "Alphabetical" string<)
-	  (const :tag "Reverse alphabetical" string>)
+	  (const :tag "Alphabetical" string-collate-lessp)
+	  (const :tag "Reverse alphabetical" org-string-collate-greaterp)
 	  (function :tag "Custom function" nil)))
 
 (defvar org-tags-history nil
@@ -8803,7 +8803,7 @@ (defun org-sort-entries
 	     (t (error "Invalid sorting type `%c'" sorting-type))))
 	  nil
 	  (cond
-	   ((= dcst ?a) 'string<)
+	   ((= dcst ?a) 'string-collate-lessp)
 	   ((= dcst ?f)
 	    (or compare-func
 		(and interactive?
@@ -8913,6 +8913,12 @@ (defun org-context-p (&rest contexts)
 		    (org-in-item-p)))
       (goto-char pos))))
 
+(defun org-string-collate-greaterp (s1 s2 &optional locale ignore-case)
+  "Return t if S1 is greater than S2 in collation order.
+
+LOCALE and IGNORE-CASE are handled as in `string-collate-lessp'."
+  (not (string-collate-lessp s1 s2)))
+
 ;;;###autoload
 (defun org-run-like-in-org-mode (cmd)
   "Run a command, pretending that the current buffer is in Org mode.
-- 
2.16.1

[PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

* org.el (org-sort-entries): Use collated sorting.
(org-tags-sort-function): Use collated sorting.
(org-string-collate-greaterp): Add helper-function to use as defcustom
option, since there is no ‘string-collate-greaterp’ in Emacs.

‘org-sort-entries’ and ‘org-tags-sort-function’ advertise alphabetic
sorting, but actually sort based only on character code.  This
produces non-alphabetic orderings of strings in non-ASCII locales.

E. g., German Umlauts “Ä Ü Ö” are alphabetically sorted as if they
were “A U O”, whereas sorting based on character-code will place them
after “Z”, which is unexpected.
---
 lisp/org.el | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 688e48bcc..dc0611b87 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3558,8 +3558,8 @@ (defcustom org-tags-sort-function nil
   :group 'org-tags
   :type '(choice
 	  (const :tag "No sorting" nil)
-	  (const :tag "Alphabetical" string<)
-	  (const :tag "Reverse alphabetical" string>)
+	  (const :tag "Alphabetical" string-collate-lessp)
+	  (const :tag "Reverse alphabetical" org-string-collate-greaterp)
 	  (function :tag "Custom function" nil)))
 
 (defvar org-tags-history nil
@@ -8803,7 +8803,7 @@ (defun org-sort-entries
 	     (t (error "Invalid sorting type `%c'" sorting-type))))
 	  nil
 	  (cond
-	   ((= dcst ?a) 'string<)
+	   ((= dcst ?a) 'string-collate-lessp)
 	   ((= dcst ?f)
 	    (or compare-func
 		(and interactive?
@@ -8913,6 +8913,12 @@ (defun org-context-p (&rest contexts)
 		    (org-in-item-p)))
       (goto-char pos))))
 
+(defun org-string-collate-greaterp (s1 s2 &optional locale ignore-case)
+  "Return t if S1 is greater than S2 in collation order.
+
+LOCALE and IGNORE-CASE are handled as in `string-collate-lessp'."
+  (not (string-collate-lessp s1 s2 locale ignore-case)))
+
 ;;;###autoload
 (defun org-run-like-in-org-mode (cmd)
   "Run a command, pretending that the current buffer is in Org mode.
-- 
2.16.1

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Nicolas Goaziou]

Hello,

Sebastian Reuße <seb@wirrsal.net> writes:

> * org.el (org-sort-entries): Use collated sorting.
> (org-tags-sort-function): Use collated sorting.
> (org-string-collate-greaterp): Add helper-function to use as defcustom
> option, since there is no ‘string-collate-greaterp’ in Emacs.

Thank you.

However, Org 9.X still supports Emacs 24, so we cannot use
`string-collate-lessp without degrading gracefully to `string-lessp' in
these Emacsen.

We could add `org-string-collate-lessp' to "org-compat.el", which would
be an alias for `string-collate-lessp' on Emacs 25+ and to
`string-lessp' otherwise.

> +(defun org-string-collate-greaterp (s1 s2 &optional locale ignore-case)
> +  "Return t if S1 is greater than S2 in collation order.

Return non-nil if...

We would also need tests for that feature.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

Hello Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> However, Org 9.X still supports Emacs 24, so we cannot use
> `string-collate-lessp without degrading gracefully to `string-lessp'
> in these Emacsen.

> We could add `org-string-collate-lessp' to "org-compat.el", which
> would be an alias for `string-collate-lessp' on Emacs 25+ and to
> `string-lessp' otherwise.

Thanks, I would not have suspected collated sorting to be so recent. I
added a proxy function to «org-compat.el».

>> +(defun org-string-collate-greaterp (s1 s2 &optional locale ignore-case)
>> +  "Return t if S1 is greater than S2 in collation order.
>
> Return non-nil if...

Fixed.

> We would also need tests for that feature.

For coverage purposes I added a test for the «string-collate-greaterp»
wrapper.

I also considered adding a regression test for non-ASCII chars to
«test-org/sort-entries», but for stable results, one would have to
enforce some canonical locale. Unfortunately, it’s not possible to
change the locale at Emacs run-time; Emacs only seems to call
«setlocale()» once during initialization, so «wcscoll()» always collates
according to the initial value of «LC_COLLATE». A regression test would
thus require changes to the Makefile, and tests might yield different
results when run from inside an existing Emacs process, so I left the
«sort-entries» test as is.

Let me know what you think.

Kind regards,

SR

-- 
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay

[PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

* org.el (org-sort-entries): Use collated sorting.
(org-tags-sort-function): Use collated sorting.
(org-string-collate-greaterp): Add helper-function to use as defcustom
option, since there is no ‘string-collate-greaterp’ in Emacs.

* org-compat.el (org-string-collate-lessp): Add proxy to fall-back on
string-lessp when string-collate-lessp is missing (Emacs ≤ 24).

* test-org.el (test-org/string-collate-lessp): Add test.

‘org-sort-entries’ and ‘org-tags-sort-function’ advertise alphabetic
sorting, but actually sort based only on character code.  This
produces non-alphabetic orderings of strings in non-ASCII locales.

E. g., German Umlauts “Ä Ü Ö” are alphabetically sorted as if they
were “A U O”, whereas sorting based on character-code will place them
after “Z”, which is unexpected.
---
 lisp/org-compat.el       |  6 ++++++
 lisp/org.el              | 12 +++++++++---
 testing/lisp/test-org.el |  5 +++++
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 2553286e1..acd5c3e1e 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -118,6 +118,12 @@ (defvar org-table1-hline-regexp)
 	      (push (expand-file-name file dir) files)))))
       (nconc result (nreverse files)))))
 
+;; `string-collate-lessp' is new in Emacs 25.
+(defalias 'org-string-collate-lessp
+  (if (fboundp 'string-collate-lessp)
+      'string-collate-lessp
+    'string-lessp))
+
 \f
 ;;; Obsolete aliases (remove them after the next major release).
 
diff --git a/lisp/org.el b/lisp/org.el
index 688e48bcc..fbbeea80f 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3558,8 +3558,8 @@ (defcustom org-tags-sort-function nil
   :group 'org-tags
   :type '(choice
 	  (const :tag "No sorting" nil)
-	  (const :tag "Alphabetical" string<)
-	  (const :tag "Reverse alphabetical" string>)
+	  (const :tag "Alphabetical" org-string-collate-lessp)
+	  (const :tag "Reverse alphabetical" org-string-collate-greaterp)
 	  (function :tag "Custom function" nil)))
 
 (defvar org-tags-history nil
@@ -8803,7 +8803,7 @@ (defun org-sort-entries
 	     (t (error "Invalid sorting type `%c'" sorting-type))))
 	  nil
 	  (cond
-	   ((= dcst ?a) 'string<)
+	   ((= dcst ?a) 'org-string-collate-lessp)
 	   ((= dcst ?f)
 	    (or compare-func
 		(and interactive?
@@ -8913,6 +8913,12 @@ (defun org-context-p (&rest contexts)
 		    (org-in-item-p)))
       (goto-char pos))))
 
+;; Defined to provide a value for defcustom, since there is no
+;; string-collate-greaterp in Emacs.
+(defun org-string-collate-greaterp (s1 s2)
+  "Return non-nil if S1 is greater than S2 in collation order."
+  (not (org-string-collate-lessp s1 s2)))
+
 ;;;###autoload
 (defun org-run-like-in-org-mode (cmd)
   "Run a command, pretending that the current buffer is in Org mode.
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index cb21cda47..ec4535551 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2927,6 +2927,11 @@
 	    (org-sort-entries nil ?a)
 	    (buffer-string)))))
 
+(ert-deftest test-org/string-collate-greaterp ()
+  "Test `org-string-collate-greaterp' specifications."
+  (should (org-string-collate-greaterp "def" "abc"))
+  (should-not (org-string-collate-greaterp "abc" "def")))
+
 (ert-deftest test-org/file-contents ()
   "Test `org-file-contents' specifications."
   ;; Open files.
-- 
2.16.1

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Nicolas Goaziou]

Sebastian Reuße <seb@wirrsal.net> writes:

> I also considered adding a regression test for non-ASCII chars to
> «test-org/sort-entries», but for stable results, one would have to
> enforce some canonical locale. Unfortunately, it’s not possible to
> change the locale at Emacs run-time; Emacs only seems to call
> «setlocale()» once during initialization, so «wcscoll()» always collates
> according to the initial value of «LC_COLLATE». A regression test would
> thus require changes to the Makefile, and tests might yield different
> results when run from inside an existing Emacs process, so I left the
> «sort-entries» test as is.

It should be possible to redefine `string-collate-lessp', using
`cl-letf' so that is uses different locales as the optional argument.
WDYT?

The patch looks good. Thank you. It would be good to add an entry in
ORG-NEWS about it, however.

Then we can improve `org-table-sort-lines' accordingly. It can be done
in another patch, tho.

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Sebastian Reuße <seb@wirrsal.net> writes:
>
>> I also considered adding a regression test for non-ASCII chars to
>> «test-org/sort-entries», but for stable results, one would have to
>> enforce some canonical locale. Unfortunately, it’s not possible to
>> change the locale at Emacs run-time; Emacs only seems to call
>> «setlocale()» once during initialization, so «wcscoll()» always collates
>> according to the initial value of «LC_COLLATE». A regression test would
>> thus require changes to the Makefile, and tests might yield different
>> results when run from inside an existing Emacs process, so I left the
>> «sort-entries» test as is.
>
> It should be possible to redefine `string-collate-lessp', using
> `cl-letf' so that is uses different locales as the optional argument.
> WDYT?

That would work. Though it does tie up the test with an
implementation-detail. Still, probably better than no regression test at
all.

The test will error out if the data for the chosen locale isn’t actually
present on the system. I chose the «en_US» locale in the hopes that this
will frequently be installed. I’ll leave it up to you to decide whether
that is acceptable.

> The patch looks good. Thank you. It would be good to add an entry in
> ORG-NEWS about it, however.

Added.

> Then we can improve `org-table-sort-lines' accordingly. It can be done
> in another patch, tho.

That’s a good idea, I’ll look into it if I have some time by the end of
the week.

Kind regards,
SR

-- 
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay

[PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

* org.el (org-sort-entries): Use collated sorting.
(org-tags-sort-function): Use collated sorting.
(org-string-collate-greaterp): Add helper-function to use as defcustom
option, since there is no ‘string-collate-greaterp’ in Emacs.

* org-compat.el (org-string-collate-lessp): Add proxy to fall-back on
string-lessp when string-collate-lessp is missing (Emacs ≤ 24).

* test-org.el (test-org/string-collate-lessp): Add test.
(test-org/sort-entries): Add regression test for non-ASCII inputs.

‘org-sort-entries’ and ‘org-tags-sort-function’ advertise alphabetic
sorting, but actually sort based only on character code.  This
produces non-alphabetic orderings of strings in non-ASCII locales.

E. g., German Umlauts “Ä Ü Ö” are alphabetically sorted as if they
were “A U O”, whereas sorting based on character-code will place them
after “Z”, which is unexpected.
---
 etc/ORG-NEWS             |  5 +++++
 lisp/org-compat.el       |  6 ++++++
 lisp/org.el              | 12 +++++++++---
 testing/lisp/test-org.el | 21 +++++++++++++++++++++
 4 files changed, 41 insertions(+), 3 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index aedede201..9c12f8e2a 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -239,6 +239,11 @@ of these exporters will properly export to =irc:= links, which will
 open properly in irc clients from web browsers.
 
 *** ~org-comment-dwim~ (bound to =M-;=) now comments headings, if point is on a heading
+*** Alphabetic sorting in headings and tags now uses the locale’s sorting rules
+
+When sorting alphabetically, ~org-sort-entries~ and
+~org-tags-sort-function~ now sort according to the locale’s collation
+rules instead of by code-point.
 * Version 9.1
 
 ** Incompatible changes
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 2553286e1..acd5c3e1e 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -118,6 +118,12 @@ (defvar org-table1-hline-regexp)
 	      (push (expand-file-name file dir) files)))))
       (nconc result (nreverse files)))))
 
+;; `string-collate-lessp' is new in Emacs 25.
+(defalias 'org-string-collate-lessp
+  (if (fboundp 'string-collate-lessp)
+      'string-collate-lessp
+    'string-lessp))
+
 \f
 ;;; Obsolete aliases (remove them after the next major release).
 
diff --git a/lisp/org.el b/lisp/org.el
index 688e48bcc..fbbeea80f 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3558,8 +3558,8 @@ (defcustom org-tags-sort-function nil
   :group 'org-tags
   :type '(choice
 	  (const :tag "No sorting" nil)
-	  (const :tag "Alphabetical" string<)
-	  (const :tag "Reverse alphabetical" string>)
+	  (const :tag "Alphabetical" org-string-collate-lessp)
+	  (const :tag "Reverse alphabetical" org-string-collate-greaterp)
 	  (function :tag "Custom function" nil)))
 
 (defvar org-tags-history nil
@@ -8803,7 +8803,7 @@ (defun org-sort-entries
 	     (t (error "Invalid sorting type `%c'" sorting-type))))
 	  nil
 	  (cond
-	   ((= dcst ?a) 'string<)
+	   ((= dcst ?a) 'org-string-collate-lessp)
 	   ((= dcst ?f)
 	    (or compare-func
 		(and interactive?
@@ -8913,6 +8913,12 @@ (defun org-context-p (&rest contexts)
 		    (org-in-item-p)))
       (goto-char pos))))
 
+;; Defined to provide a value for defcustom, since there is no
+;; string-collate-greaterp in Emacs.
+(defun org-string-collate-greaterp (s1 s2)
+  "Return non-nil if S1 is greater than S2 in collation order."
+  (not (org-string-collate-lessp s1 s2)))
+
 ;;;###autoload
 (defun org-run-like-in-org-mode (cmd)
   "Run a command, pretending that the current buffer is in Org mode.
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index cb21cda47..dcf097e69 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2737,6 +2737,22 @@
 	  (org-test-with-temp-text "\n* def\n* xyz\n* abc\n"
 	    (org-sort-entries nil ?A)
 	    (buffer-string))))
+  ;; Sort alphabetically (with non-ASCII input). Rebinds `string-collate-lessp'
+  ;; to enforce a canonical locale during testing.
+  (let ((original-string-collate-lessp (symbol-function 'string-collate-lessp)))
+    (cl-letf (((symbol-function 'string-collate-lessp)
+	       (lambda (s1 s2) (funcall original-string-collate-lessp
+					s1 s2 "en_US.utf-8"))))
+      (should
+       (equal "\n* äa\n* ab\n* z\n"
+	      (org-test-with-temp-text "\n* ab\n* z\n* äa\n"
+	        (org-sort-entries nil ?a)
+	        (buffer-string))))
+      (should
+       (equal "\n* z\n* äb\n* aa\n"
+	      (org-test-with-temp-text "\n* äb\n* z\n* aa\n"
+	        (org-sort-entries nil ?A)
+		(buffer-string))))))
   ;; Sort numerically.
   (should
    (equal "\n* 1\n* 2\n* 10\n"
@@ -2927,6 +2943,11 @@
 	    (org-sort-entries nil ?a)
 	    (buffer-string)))))
 
+(ert-deftest test-org/string-collate-greaterp ()
+  "Test `org-string-collate-greaterp' specifications."
+  (should (org-string-collate-greaterp "def" "abc"))
+  (should-not (org-string-collate-greaterp "abc" "def")))
+
 (ert-deftest test-org/file-contents ()
   "Test `org-file-contents' specifications."
   ;; Open files.
-- 
2.16.1

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Nicolas Goaziou]

Hello,

Sebastian Reuße <seb@wirrsal.net> writes:

> * org.el (org-sort-entries): Use collated sorting.
> (org-tags-sort-function): Use collated sorting.
> (org-string-collate-greaterp): Add helper-function to use as defcustom
> option, since there is no ‘string-collate-greaterp’ in Emacs.
>
> * org-compat.el (org-string-collate-lessp): Add proxy to fall-back on
> string-lessp when string-collate-lessp is missing (Emacs ≤ 24).
>
> * test-org.el (test-org/string-collate-lessp): Add test.
> (test-org/sort-entries): Add regression test for non-ASCII inputs.
>
> ‘org-sort-entries’ and ‘org-tags-sort-function’ advertise alphabetic
> sorting, but actually sort based only on character code.  This
> produces non-alphabetic orderings of strings in non-ASCII locales.
>
> E. g., German Umlauts “Ä Ü Ö” are alphabetically sorted as if they
> were “A U O”, whereas sorting based on character-code will place them
> after “Z”, which is unexpected.

Applied. Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Nicolas Goaziou]

Sebastian Reuße <seb@wirrsal.net> writes:

> The test will error out if the data for the chosen locale isn’t actually
> present on the system. I chose the «en_US» locale in the hopes that this
> will frequently be installed. I’ll leave it up to you to decide whether
> that is acceptable.

I used "C" locale instead. I think it is available on every system. So
basically, the test checks if we are not using string< for comparison.

> That’s a good idea, I’ll look into it if I have some time by the end of
> the week.

Thank you!

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Sebastian Reuße]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Sebastian Reuße <seb@wirrsal.net> writes:

>> The test will error out if the data for the chosen locale isn’t
>> actually present on the system. I chose the «en_US» locale in the
>> hopes that this will frequently be installed. I’ll leave it up to you
>> to decide whether that is acceptable.

> I used "C" locale instead. I think it is available on every system. So
> basically, the test checks if we are not using string< for comparison.

It looks like «string-collate-lessp», when used with the POSIX locale,
is equivalent to «string<». The examples I tried out all came up the
same, and [1] (headline «LC_COLLATE Category in the POSIX Locale») says
the POSIX collation ordering is the same as the ASCII codeset. glibc
adheres to this too (cf. /usr/share/i18n/locales/POSIX).

[1] <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html#tag_07_03>

In that case the extension to the test case might be better left out.

When I was looking into this yesterday I noticed that glibc ships a
locale definition file that specifies collation according to ISO 14651
(which most language locales derive from), but unfortunately the POSIX
locale doesn’t use that.

Kind regards,
SR

--
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay

Re: [PATCH] Fix alphabetic sorting for headlines, tags

[Nicolas Goaziou]

Sebastian Reuße <seb@wirrsal.net> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Sebastian Reuße <seb@wirrsal.net> writes:
>
>>> The test will error out if the data for the chosen locale isn’t
>>> actually present on the system. I chose the «en_US» locale in the
>>> hopes that this will frequently be installed. I’ll leave it up to you
>>> to decide whether that is acceptable.
>
>> I used "C" locale instead. I think it is available on every system. So
>> basically, the test checks if we are not using string< for comparison.
>
> It looks like «string-collate-lessp», when used with the POSIX locale,
> is equivalent to «string<». The examples I tried out all came up the
> same, and [1] (headline «LC_COLLATE Category in the POSIX Locale») says
> the POSIX collation ordering is the same as the ASCII codeset. glibc
> adheres to this too (cf. /usr/share/i18n/locales/POSIX).

True. Somehow, I thought the would differ outside the ASCII range.

> In that case the extension to the test case might be better left out.

I'm going to remove this test. Thanks for the heads up.