COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Tom Alexander]

It seems that COUNTER-SET[1] is not being honored when exporting to utf-8 for alphabetical lists even though it is honored for numeric lists. When exporting to html, COUNTER-SET is honored for both.

Test document:
```
# An ordered list starting at 3
1. [@3] foo


# An ordered list starting at 1
m. bar


# An ordered list starting at 11
m. [@k] baz
```

Launching emacs with: (Setting org-list-allow-alphabetical is necessary or else the alphabetical lists will become paragraphs)
```
emacs -q --eval '(setq org-list-allow-alphabetical t)' /tmp/test.org
```

When exporting to html you get (edited to remove whitespace for clarity):
```
<ol class="org-ol"><li value="3">foo</li></ol>
<ol class="org-ol"><li>bar</li></ol>
<ol class="org-ol"><li value="11">baz</li></ol>
```

But when exporting to utf-8 you get: (whitespace removed again)
```
3. foo
m. bar
m. baz
```

Whereas I would expect:  (whitespace removed again)
```
3. foo
m. bar
k. baz
```

On a slightly related note: it seems the COUNTER-SET[1] allows single-letter values even when org-list-allow-alphabetical is nil. I don't think that is going to hurt anyone but I figured I should mention it in case its a bug (test doc: `1. [@k] foo` is a plain list starting at 11 even when org-list-allow-alphabetical is nil).

[1] https://orgmode.org/worg/org-syntax.html#Items

Emacs 29.1, Org-mode version 9.7-pre (release_9.6.8-781-gc70354)

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Ihor Radchenko]

"Tom Alexander" <tom@fizz.buzz> writes:

> It seems that COUNTER-SET[1] is not being honored when exporting to utf-8 for alphabetical lists even though it is honored for numeric lists. When exporting to html, COUNTER-SET is honored for both.
>
> Test document:
> ```
> # An ordered list starting at 3
> 1. [@3] foo
>
>
> # An ordered list starting at 1
> m. bar
>
>
> # An ordered list starting at 11
> m. [@k] baz
> ```
> ...
> But when exporting to utf-8 you get: (whitespace removed again)
> ```
> 3. foo
> m. bar
> m. baz
> ```
>
> Whereas I would expect:  (whitespace removed again)
> ```
> 3. foo
> m. bar
> k. baz
> ```

Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=26f1cb77a

However, what we do for alphabetical bullets on export is slightly
different:

3. foo
1. bar
11. baz

Other export backends do not preserve alphabetical bullets. Numbers are
always used.

> On a slightly related note: it seems the COUNTER-SET[1] allows single-letter values even when org-list-allow-alphabetical is nil. I don't think that is going to hurt anyone but I figured I should mention it in case its a bug (test doc: `1. [@k] foo` is a plain list starting at 11 even when org-list-allow-alphabetical is nil).
>
> [1] https://orgmode.org/worg/org-syntax.html#Items

org-syntax.html#Items describes what the parser does (`org-element-item-parser').
`org-list-allow-alphabetical' should be changed to t in future and
obsoleted. We aim to reduce config-dependent Org syntax in the long
term.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Tom Alexander]

Thanks!

> We aim to reduce config-dependent Org syntax in the long term.

Thats wonderful news! Sometimes this stuff can really surprise you. For example, the structure of the document created by running `echo "1. foo\n 1.bar\n        1.baz\n\t1.lorem"` changes based on the user's **tab-width**!!

If tab-width is less than 8 then this is:
```text
1. foo
  1. bar
    1. baz
  2. lorem
```

If tab-width is 8 then this is:
```text
1. foo
  1. bar
    1. baz
    2. lorem
```

and if tab-width is greater than 8 this is:
```text
1. foo
  1. bar
    1. baz
      1. lorem
```

Absolute madness! I always considered tab-width to be a personal aesthetic choice and not something that would functionally change how documents other people wrote will be parsed.

Idk if its been discussed, but personally if I were given dictatorship over org-mode I would take all of these emacs variables that are defined outside of the document, and instead of having them influence org-mode directly, I would *only* use them to pre-populate values for in-buffer settings templates.

For example, if a user had set `org-odd-levels-only` then I wouldn't have that impact ANY existing document they open, but if they open a new document then I would have it auto-insert `#+STARTUP: odd` at the top of the fresh document.

Otherwise it seems like org-mode is unsuitable for multi-person collaboration without dictating the contents of everyone's `.emacs` file.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc

[PATCH] [DISCUSSION] Force `tab-width' to be 8 in Org mode (was: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

"Tom Alexander" <tom@fizz.buzz> writes:

> Thanks!
>
>> We aim to reduce config-dependent Org syntax in the long term.
>
> Thats wonderful news! Sometimes this stuff can really surprise you. For example, the structure of the document created by running `echo "1. foo\n 1.bar\n        1.baz\n\t1.lorem"` changes based on the user's **tab-width**!!
> ...
> Absolute madness! I always considered tab-width to be a personal aesthetic choice and not something that would functionally change how documents other people wrote will be parsed.

I agree.
See the attached tentative patches for Org mode and org-syntax.

This is a breaking change, so I encourage people who might be affected
reply if they have objections.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-org.el-org-mode-Force-tab-width-to-be-8.patch --]
[-- Type: text/x-patch, Size: 4177 bytes --]

From 423cf9aef588ed93cbc06d6033feaf6bf1a91dfc Mon Sep 17 00:00:00 2001
Message-ID: <423cf9aef588ed93cbc06d6033feaf6bf1a91dfc.1697796750.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Fri, 20 Oct 2023 13:10:30 +0300
Subject: [PATCH] * lisp/org.el (org-mode): Force `tab-width' to be 8

* lisp/org-macs.el (org-current-text-column): Assert `tab-width' to be
8 to ensure consistency of the parser across user configurations.
* etc/ORG-NEWS (~tab-width~ value is now assumed to be 8): Document
the breaking change.

This breaking change is made to standardize Org mode format for list
items.  With variable `tab-width', indentation in lists may depend on
user settings leading to inconsistent Org documents when open by
different users.

Link: https://orgmode.org/list/2c9a6cbd-21c0-45bf-8fbb-4f7eccac4ae7@app.fastmail.com
---
 etc/ORG-NEWS     | 34 ++++++++++++++++++++++++++++++++++
 lisp/org-macs.el | 13 ++++++++++---
 lisp/org.el      |  3 +++
 3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 78b75b578..ea8ebd3af 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -13,6 +13,40 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
 ** Important announcements and breaking changes
+*** ~tab-width~ value is now assumed to be 8
+
+Org mode now assumes tab width to be 8 characters, when calculating
+list and other indentation.  ~tab-width~ is also set to 8 when Org
+major mode is loaded.
+
+This is done to improve consistency of the markup for lists, where
+indentation affects list items.
+
+Users with non-default values of ~tab-width~ should avoid overriding
+the value of 8 set by Org mode.  If the custom ~tab-width~ value is
+_smaller_ than 8, the existing Org documents can be converted to the
+new standard tab width using the following helper command:
+
+#+begin_src emacs-lisp
+(defun org-compat-adjust-tab-width-in-buffer (old-width)
+  "Adjust visual indentation from `tab-width' equal OLD-WIDTH to 8."
+  (interactive "nOld `tab-width': ")
+  (cl-assert (derived-mode-p 'org-mode))
+  (unless (= old-width 8)
+    (org-with-wide-buffer
+     (goto-char (point-min))
+     (let (bound
+	   (repl (if (< old-width 8)
+		     (make-string old-width ?\s)
+                   (concat "\t" (make-string (- old-width 8) ?\s)))))
+       (while (re-search-forward "^ *\t" nil t)
+	 (skip-chars-forward " \t")
+	 (setq bound (point-marker))
+	 (forward-line 0)
+	 (while (search-forward "\t" bound t)
+	   (replace-match repl)))))))
+#+end_src
+
 *** New export option ~org-export-expand-links~
 
 The new option makes Org expand environment variables in link and INCLUDE paths.
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index fd0f508c5..1fd7dd704 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -1148,9 +1148,16 @@ (defun org-string-width (string &optional pixels)
             (/ pixel-width symbol-width)))))))
 
 (defmacro org-current-text-column ()
-  "Like `current-column' but ignore display properties."
-  `(string-width (buffer-substring-no-properties
-                  (line-beginning-position) (point))))
+  "Like `current-column' but ignore display properties.
+Throw an error when `tab-width' is not 8.
+
+This function forces `tab-width' value because it is used as a part of
+the parser, to ensure parser consistency when calculating list
+indentation."
+  `(progn
+     (cl-assert (= 8 tab-width))
+     (string-width (buffer-substring-no-properties
+                    (line-beginning-position) (point)))))
 
 (defun org-not-nil (v)
   "If V not nil, and also not the string \"nil\", then return V.
diff --git a/lisp/org.el b/lisp/org.el
index bda64bb6c..671dfbe38 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -4846,6 +4846,9 @@ (define-derived-mode org-mode outline-mode "Org"
 
 \\{org-mode-map}"
   (setq-local org-mode-loading t)
+  ;; Force tab width - indentation is significant in lists, so we need
+  ;; to make sure that it is consistent across configurations.
+  (setq-local tab-width 8)
   (org-load-modules-maybe)
   (when org-agenda-file-menu-enabled
     (org-install-agenda-files-menu))
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-org-syntax.org-Indentation-Clarify-that-tabs-occupy-.patch --]
[-- Type: text/x-patch, Size: 1248 bytes --]

From 1ce9ef2f8e475bcb8b9be7367bb30aa0983da6ea Mon Sep 17 00:00:00 2001
Message-ID: <1ce9ef2f8e475bcb8b9be7367bb30aa0983da6ea.1697637104.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 18 Oct 2023 16:51:26 +0300
Subject: [PATCH] * org-syntax.org (Indentation): Clarify that tabs occupy 8
 characters

---
 org-syntax.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/org-syntax.org b/org-syntax.org
index 9e9b0420..3089ddcf 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -240,7 +240,9 @@ ** Indentation
 Indentation consists of a series of space and tab characters at the
 beginning of a line.  Most elements can be indentated, with the
 exception of [[#Headings][headings]], [[#Inlinetasks][inlinetasks]], [[#Footnote_Definitions][footnote definitions]], and [[#Diary_Sexp][diary
-sexps]].  Indentation is only syntactically meaningful in plain lists.
+sexps]].  Indentation is only syntactically meaningful in plain lists,
+where indentation is calculated assuming that space characters occupy
+a single character and tab characters occupy 8 characters.
 
 The common indentation of all the lines within an element is
 discarded.  This also applies to single-line elements.
-- 
2.42.0


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] [DISCUSSION] Force `tab-width' to be 8 in Org mode (was: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter)

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> Absolute madness! I always considered tab-width to be a personal aesthetic choice and not something that would functionally change how documents other people wrote will be parsed.
>
> I agree.
> See the attached tentative patches for Org mode and org-syntax.
>
> This is a breaking change, so I encourage people who might be affected
> reply if they have objections.
>
> Subject: [PATCH] * lisp/org.el (org-mode): Force `tab-width' to be 8

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=315417582

Also, adjusted org-syntax document.
https://git.sr.ht/~bzg/worg/commit/72134bba

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Ihor Radchenko]

"Tom Alexander" <tom@fizz.buzz> writes:

> Idk if its been discussed, but personally if I were given dictatorship over org-mode I would take all of these emacs variables that are defined outside of the document, and instead of having them influence org-mode directly, I would *only* use them to pre-populate values for in-buffer settings templates.

> For example, if a user had set `org-odd-levels-only` then I wouldn't have that impact ANY existing document they open, but if they open a new document then I would have it auto-insert `#+STARTUP: odd` at the top of the fresh document.

That would be very annoying for many users.
What might be better is providing a way to "unload" the parser-affecting
non-standard settings into Org file header, so that the file can be
shared properly with other people.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Stacey Marshall]

Regarding https://list.orgmode.org/orgmode/36a62fbf-6484-456f-9537-a7aa40530068@app.fastmail.com/

Was the intention with this change to remove alphabetical lists from
text (ascii) exporter?

----
#+begin_src 
,#+title: Test document

,#+options: toc:nil author:nil
,* test of alphabetical export

,#+begin_src emacs-lisp :exports both :eval yes
(format "%S\n%S\norg-list-allow-alphabetical: %S\n"
	(emacs-version) (org-version) org-list-allow-alphabetical)
,#+end_src

a. Point A
b. Point B
c. Point C
d. [@z] Point Z
#+end_src

Latest 9.7 alpha results in the alphabetical labels being replaced
with numbers::
#+begin_example
1 test of alphabetical export
=============================

  ,----
  | (format "%S\n%S\norg-list-allow-alphabetical: %S\n"
  | 	(emacs-version) (org-version) org-list-allow-alphabetical)
  `----

  ,----
  | "GNU Emacs 29.1 (build 1, aarch64-apple-darwin23.3.0, Carbon Version 170 AppKit 2487.4)
  |  of 2024-02-26"
  | "9.7-pre"
  | org-list-allow-alphabetical: t
  `----


  1. Point A
  2. Point B
  3. Point C
  26. Point Z
#+end_example

Removing the change to ox-ascii.el restores it, but leaves the original issue:

#+begin_src diff :eval no :exports code
$ git diff lisp/ox-ascii.el
diff --git a/lisp/ox-ascii.el b/lisp/ox-ascii.el
index db4356ec6..78641f1db 100644
--- a/lisp/ox-ascii.el
+++ b/lisp/ox-ascii.el
@@ -1485,7 +1485,7 @@ contextual information."
                                      struct
                                      (org-list-prevs-alist struct)
                                      (org-list-parents-alist struct)))))))
-              (replace-regexp-in-string "[0-9A-Za-z]+" num bul)))
+              (replace-regexp-in-string "[0-9]+" num bul)))
            (_ (let ((bul (org-list-bullet-string
                           (org-element-property :bullet item))))
                 ;; Change bullets into more visible form if UTF-8 is active.
#+end_src

#+begin_example
1 test of alphabetical export
=============================

  ,----
  | (format "%S\n%S\norg-list-allow-alphabetical: %S\n"
  | 	(emacs-version) (org-version) org-list-allow-alphabetical)
  `----

  ,----
  | "GNU Emacs 29.1 (build 1, aarch64-apple-darwin23.3.0, Carbon Version 170 AppKit 2487.4)
  |  of 2024-02-26"
  | "9.7-pre"
  | org-list-allow-alphabetical: t
  `----


  a. Point A
  b. Point B
  c. Point C
  d. Point Z
#+end_example

I must admit that I had not noticed that other exporters converted
the letter to a number.  Going forward I shall stop using them.

If it is obsoleted or deprecated then a mention to etc/ORG-NEWS
would be appreciated.

yours faithfully,

Stacey Marshall

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Ihor Radchenko]

Stacey Marshall <stacey.marshall@gmail.com> writes:

> Was the intention with this change to remove alphabetical lists from
> text (ascii) exporter?

The intention was to make ascii exporter aware about alphabetical lists
instead of behaving in undefined way.

> ...
> a. Point A
> b. Point B
> c. Point C
> d. [@z] Point Z
> #+end_src
>
> Latest 9.7 alpha results in the alphabetical labels being replaced
> with numbers::
> ..

Yes.

> I must admit that I had not noticed that other exporters converted
> the letter to a number.  Going forward I shall stop using them.
>
> If it is obsoleted or deprecated then a mention to etc/ORG-NEWS
> would be appreciated.

There is nothing to obsolete.
ox-ascii did not process alphabetical lists in any way, which was a bug.
All other exporters convert alphabetical lists into numbered lists.
The bug has been fixed, and the behaviour is now consistent with the rest
of exporters.

If you wish alphabetical lists to be exported specially, feel free to
post a feature request.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

[Stacey Marshall]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

> There is nothing to obsolete.
> ox-ascii did not process alphabetical lists in any way, which was a bug.

OK, thanks for verification.

Stace

On Sat, 6 Apr 2024 at 13:02, Ihor Radchenko <yantar92@posteo.net> wrote:

> Stacey Marshall <stacey.marshall@gmail.com> writes:
>
> > Was the intention with this change to remove alphabetical lists from
> > text (ascii) exporter?
>
> The intention was to make ascii exporter aware about alphabetical lists
> instead of behaving in undefined way.
>
> > ...
> > a. Point A
> > b. Point B
> > c. Point C
> > d. [@z] Point Z
> > #+end_src
> >
> > Latest 9.7 alpha results in the alphabetical labels being replaced
> > with numbers::
> > ..
>
> Yes.
>
> > I must admit that I had not noticed that other exporters converted
> > the letter to a number.  Going forward I shall stop using them.
> >
> > If it is obsoleted or deprecated then a mention to etc/ORG-NEWS
> > would be appreciated.
>
> There is nothing to obsolete.
> ox-ascii did not process alphabetical lists in any way, which was a bug.
> All other exporters convert alphabetical lists into numbered lists.
> The bug has been fixed, and the behaviour is now consistent with the rest
> of exporters.
>
> If you wish alphabetical lists to be exported specially, feel free to
> post a feature request.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>

[-- Attachment #2: Type: text/html, Size: 2412 bytes --]