Re: test-org-table/sort-lines: Failing test on macOS

[Ihor Radchenko <yantar92@posteo.net>]

Max Nikulin <manikulin@gmail.com> writes:

> On 22/11/2022 08:14, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>>> 2. `org-sort-list'
>>>> 5. `org-sort-entries'
>>> `downcase' is used, not proper case folding, so a potential issue
>> 
>> `downcase' is used to determine user input about sorting type.
>> Not for sorting itself.
>
> See case-func variable. Its initialization depends on the IGNORE-CASE 
> argument. Strings to sort are passed either through `identity' or 
> through `downcase'.

Thanks for the pointer.
Now, I am getting more confused though.
Do we even need to use `string-collate-lessp' then?

Eli even argued that `string-collate-lessp' is strictly worse compared
to more predictable approach. See
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59275#40

Do you remember any cases when users actually demanded locale-specific
sorting?

>>> IGNORE-CASE argument is not used, perhaps `downcase' is hidden in the code.
>> 
>> I feel like we are slightly miscommunicating here.
>> I mostly tried to list the uses of libc-sensitive sorting. Not
>> specifically cases when we try to ignore the case.
>> 
>> The problem is not limited to case-sensitive comparisons. Some systems
>> may fail to implement specific locales and thus sorting may downgrade to
>> simple string-lessp.
>
> When case folding is not involved, I consider `string-lessp' as a 
> graceful degradation. Despite locale rules are not applied, strings are 
> mostly sorted. Exceptions exist, but usually order is reasonable.
>
> Completely disregarding IGNORE-CASE argument of `string-collate-lessp' 
> on MacOS (that is not a heavily stripped embedded OS) is a bad surprise 
> for me.

It was a surprise for me as well. Should be at least a bit more clear
now as I updated the docstring of `string-collate-lessp'.

However, I feel a bit lost about what to do on Org side.
We can put a disclaimer in the manual and all that, but it still feels
too complex.

>>>> 6. Agenda sorting, when alphabetical sorting is involved
>>>
>>> `string-lessp' and `downcase' so even more severe locale-related issues
>>> might be expected.
>> 
>> Could you please elaborate?
>
> I admit that `downcase' may be an acceptable workaround since 
> `string-collate-lessp' may not work IGNORE-CASE, but I believe, when 
> available, `string-collate-lessp' should be the preferred option for 
> sorting.

As I pointed above, Eli has an opposite opinion.
I feel that my understanding of the topic is not sufficient to judge.
Maybe we should ask users? (But who is even aware about these things
happening under the hood?)

> I have an idea of a compatibility wrapper for `string-collate-lessp' 
> with special treatment of ignoring case and bad libc implementation. 
> Apply `downcase' before passing arguments to `string-lessp'. It should 
> provide consistency, best user experience when locales works properly, 
> and graceful degradation otherwise. I hope, it is acceptable for Org 
> even though such trick is undesired for Emacs due to performance reasons.

Macro idea sounds reasonable. Though I am still unsure which direction
we need to go.

> However I am afraid of compatibility shims after
>
> d3a9c424b 2022-08-16 17:15:27 +0800 Ihor Radchenko: org-encode-time: 
> Refactor into top-level `defmacro'

What do you refer to?

> I do not like that Emacs relies on locale support (and timezone as well) 
> in libc. It becomes a problem as soon as more than one locale should be 
> used in simultaneously. I agree that there are enough complications and 
> sometimes locale depends on the document (e.g. #+LANGUAGE:), sometimes 
> specific locale even restricted to a part of a document. It is tricky to 
> handle such cases, but current limitations are too strict (and defective 
> `string-collate-lessp' on MacOS is an example).

The question is what can be done and, more importantly, how much effort
will it take to implement and maintain an alternative.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>