bug: org-table-sort-lines does only case sensitive alphabetical sorting

bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Heikki Lehvaslaiho]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

org-sort has an optional argument with-case. By default its sorting ignores
the case. When org-sort is called within a table, it calls
org-table-sort-line with the argument. org-table-sort-line defines a let
variable sort-fold-case (not with-case) but fails to use either variable
when sorting and always sorts with case.

This is with the very latest org:
Org mode version 9.1.13 (9.1.13-elpa @
/Users/helehvas/.emacs.d/elpa/org-20180702/)

     -Heikki

Heikki Lehväslaiho - skype:heikki_lehvaslaiho cell: +358 40 850 6640
http://about.me/heikki

[-- Attachment #2: Type: text/html, Size: 1571 bytes --]

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Nicolas Goaziou]

Hello,

Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:

> org-sort has an optional argument with-case. By default its sorting ignores
> the case. When org-sort is called within a table, it calls
> org-table-sort-line with the argument. org-table-sort-line defines a let
> variable sort-fold-case (not with-case) but fails to use either variable
> when sorting and always sorts with case.

IIRC, it was fixed in master branch. Could you test it?

Thank you.

Regards,

-- 
Nicolas Goaziou

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Heikki Lehvaslaiho]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

I can not see any change in function with the new code. Alphabetical
sorting is fixedly case sensitive.

I am definitely running the new code for org-table-sort-lines at the head
of the master branch. I can see there have been changes. Universal argument
does not change the behaviour, either.

I am running this OS X High Sierra with English as a default language, so
the problem should not be the locale.


     -Heikki

Heikki Lehväslaiho - skype:heikki_lehvaslaiho cell: +358 40 850 6640
http://about.me/heikki

On 4 July 2018 at 19:16, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Hello,
>
> Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:
>
> > org-sort has an optional argument with-case. By default its sorting
> ignores
> > the case. When org-sort is called within a table, it calls
> > org-table-sort-line with the argument. org-table-sort-line defines a let
> > variable sort-fold-case (not with-case) but fails to use either variable
> > when sorting and always sorts with case.
>
> IIRC, it was fixed in master branch. Could you test it?
>
> Thank you.
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 2029 bytes --]

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Nicolas Goaziou]

Hello,

Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:

> I can not see any change in function with the new code. Alphabetical
> sorting is fixedly case sensitive.
>
> I am definitely running the new code for org-table-sort-lines at the head
> of the master branch. I can see there have been changes. Universal argument
> does not change the behaviour, either.

OK. I'm Cc'ing Sebastian, who fixed table sorting recently.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Sebastian Reuße]

Hello Nicolas, hello Heikki,

> Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:

>> I can not see any change in function with the new code. Alphabetical
>> sorting is fixedly case sensitive.

Is your Emacs≤24? Emacs (surprisingly) only gained a facility for
alphabetic (locale-sensitive) sorting with ver. 25. Before that, we will
resort to using «string-lessp», which only sorts lexicographically (i.e,
based on the binary character value, which means it is always
case-sensitive on Unicode systems).

If you are using GNU Emacs≥25, what locale are you using? FI_fi and most
of the other western locales are based on ISO-14651, so you should
always get case-*in*sensitive orderings, which is what I get with Emacs
26.1. So, evaluating the following should all yield «t»:

    (string-collate-lessp "a" "x" "de_DE.utf-8" nil)
    (string-collate-lessp "a" "X" "de_DE.utf-8" t)
    (string-collate-lessp "a" "x" "fi_FI.utf-8" nil)
    (string-collate-lessp "a" "X" "fi_FI.utf-8" t)

Kind regards,

Sebastian

-- 
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Heikki Lehvaslaiho]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

Thanks for replies,

I am using:
GNU Emacs 26.1 (build 1, x86_64-apple-darwin14.5.0, NS appkit-1348.17
Version 10.10.5 (Build 14F2511)) of 2018-05-31


This is what I get:

    (string-collate-lessp "a" "x" "de_DE.utf-8" nil) ; t
    (string-collate-lessp "a" "X" "de_DE.utf-8" t) ; nil
    (string-collate-lessp "a" "x" "fi_FI.utf-8" nil) ; t
    (string-collate-lessp "a" "X" "fi_FI.utf-8" t) ; nil


I think I have lost the sense how Emacs language environments work
after struggling for years with buffers getting automatically
converted to Latin-1. I started setting all possible variables to
'UTF-8 and finally something did it.

Now I tried again various values to see if they affect the org table
sorting:

(set-language-environment 'utf-8)
(set-language-environment 'English)
(set-language-environment 'Swedish)

(setq locale-coding-system 'utf-8)
(setq locale-coding-system 'fi_FI.utf-8)
(setq locale-coding-system 'de_DE.utf-8)
(setq locale-coding-system 'en_GB.utf-8)

Unfortunately none of these made any difference. Any suggestions?

I am testing sorting in a simple one column table:

| a |
| c |
| B |
| D |




     -Heikki

Heikki Lehväslaiho - skype:heikki_lehvaslaiho cell: +358 40 850 6640
http://about.me/heikki

On 14 July 2018 at 15:31, Sebastian Reuße <seb@wirrsal.net> wrote:

> Hello Nicolas, hello Heikki,
>
> > Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:
>
> >> I can not see any change in function with the new code. Alphabetical
> >> sorting is fixedly case sensitive.
>
> Is your Emacs≤24? Emacs (surprisingly) only gained a facility for
> alphabetic (locale-sensitive) sorting with ver. 25. Before that, we will
> resort to using «string-lessp», which only sorts lexicographically (i.e,
> based on the binary character value, which means it is always
> case-sensitive on Unicode systems).
>
> If you are using GNU Emacs≥25, what locale are you using? FI_fi and most
> of the other western locales are based on ISO-14651, so you should
> always get case-*in*sensitive orderings, which is what I get with Emacs
> 26.1. So, evaluating the following should all yield «t»:
>
>     (string-collate-lessp "a" "x" "de_DE.utf-8" nil)
>     (string-collate-lessp "a" "X" "de_DE.utf-8" t)
>     (string-collate-lessp "a" "x" "fi_FI.utf-8" nil)
>     (string-collate-lessp "a" "X" "fi_FI.utf-8" t)
>
> Kind regards,
>
> Sebastian
>
> --
> Insane cobra split the wood
> Trader of the lowland breed
> Call a jittney, drive away
> In the slipstream we will stay
>

[-- Attachment #2: Type: text/html, Size: 3992 bytes --]

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Sebastian Reuße]

Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:

> x86_64-apple-darwin14.5.0

Looking at the C source of ‘string-collate-lessp’ (src/fns.c), it looks
like there exists a case where Emacs itself will fall back on
‘string-lessp’ even when ‘string-collate-lessp’ is invoked.

    #else  /* !__STDC_ISO_10646__, !WINDOWSNT */
     return Fstring_lessp (s1, s2);
    #endif /* !__STDC_ISO_10646__, !WINDOWSNT */

I can’t tell what the first macro evaluates to on your platform. Might
this be the cause?

An indication for whether your Emacs uses the locale at all may be
testing the following (presuming you have the German locale data
installed and compiled):

    (string-collate-lessp "ß" "ssa" "de_DE.utf-8") ; t
    (string-lessp "ß" "ssa")                       ; nil

This is based on the German ‘ß’ ligature being sorted as if it were
equal to the string ‘ss’ (you can probably construct something analogous
using fi_FI). If the first call returns nil, your Emacs isn’t using the
locale data at all.

Kind regards,
SR

-- 
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Heikki Lehvaslaiho]

[-- Attachment #1: Type: text/plain, Size: 2433 bytes --]

My emacs is from homebrew without any manual settings on locale. I assume a
number of locale settings come with it. I have not had any problems before
this.

OS X does not have LANG envvar set. This computer came with Finnish
language set and I switched to English (Australian) for interface. This has
lead to somewhat weird locale setting at OS:

$ defaults read .GlobalPreferences AppleLanguages
(
    "en-FI"
)

Apart from ssa, the output from string-collate-lessp seems to be what is to
be expected:

(string-collate-lessp "a" "x" "de_DE.utf-8" nil) ; t
(string-collate-lessp "a" "X" "de_DE.utf-8" nil) ; nil
(string-collate-lessp "ß" "ssa" "de_DE.utf-8") ; nil
(string-lessp "ß" "ssa")                       ; nil
(string-collate-lessp "ä" "ö" "fi_FI.utf-8") ; t
(string-collate-lessp "Ä" "ö" "fi_FI.utf-8") ; t

I can not able to comment on C code, but it really seems that alphabetic
sorting is fixed to case-sensitive mode.




     -Heikki

Heikki Lehväslaiho - skype:heikki_lehvaslaiho cell: +358 40 850 6640
http://about.me/heikki

On 17 July 2018 at 09:59, Sebastian Reuße <seb@wirrsal.net> wrote:

>
> Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:
>
> > x86_64-apple-darwin14.5.0
>
> Looking at the C source of ‘string-collate-lessp’ (src/fns.c), it looks
> like there exists a case where Emacs itself will fall back on
> ‘string-lessp’ even when ‘string-collate-lessp’ is invoked.
>
>     #else  /* !__STDC_ISO_10646__, !WINDOWSNT */
>      return Fstring_lessp (s1, s2);
>     #endif /* !__STDC_ISO_10646__, !WINDOWSNT */
>
> I can’t tell what the first macro evaluates to on your platform. Might
> this be the cause?
>
> An indication for whether your Emacs uses the locale at all may be
> testing the following (presuming you have the German locale data
> installed and compiled):
>
>     (string-collate-lessp "ß" "ssa" "de_DE.utf-8") ; t
>     (string-lessp "ß" "ssa")                       ; nil
>
> This is based on the German ‘ß’ ligature being sorted as if it were
> equal to the string ‘ss’ (you can probably construct something analogous
> using fi_FI). If the first call returns nil, your Emacs isn’t using the
> locale data at all.
>
> Kind regards,
> SR
>
> --
> Insane cobra split the wood
> Trader of the lowland breed
> Call a jittney, drive away
> In the slipstream we will stay
>

[-- Attachment #2: Type: text/html, Size: 3905 bytes --]

Re: bug: org-table-sort-lines does only case sensitive alphabetical sorting

[Sebastian Reuße]

Heikki Lehvaslaiho <heikki.lehvaslaiho@gmail.com> writes:

> (string-collate-lessp "ß" "ssa" "de_DE.utf-8") ; nil
> (string-lessp "ß" "ssa")                       ; nil

This shows us that your Emacs installation does not support alphabetic
sorting at all, regardless of case, and instead only sorts
lexicographically (i.e., regardless of locale). This likely happens due
to your platform not implementing wide characters in terms of ISO 10646.
It is likely not related to your language environment, because the
implementation of ‘string-collate-lessp’ ignores the language
environment when a locale string is passed.

Kind regards,
SR

-- 
Insane cobra split the wood
Trader of the lowland breed
Call a jittney, drive away
In the slipstream we will stay