c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

This commit may cause random failures when
org-emphasis-regexp-components is changed by user. org-emph-re is
calculated according to org-emphasis-regexp-components. Changing
org-emphasis-regexp-components can make "(when (looking-at org-emph-re)"
in parsers return nil. The emphasised text will still be fontified, but
not available in the parsed buffer.

Maybe we need to move the logic for org-emph-re from org.el to
org-element.el?

Also, there is org-emphasis-alist. It is even defcustom, but ignored by
org-element.el.

Best,
Ihor

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> This commit may cause random failures when
> org-emphasis-regexp-components is changed by user.

This is not supported anyway.

> org-emph-re is calculated according to org-emphasis-regexp-components.
> Changing org-emphasis-regexp-components can make "(when (looking-at
> org-emph-re)" in parsers return nil. The emphasised text will still be
> fontified, but not available in the parsed buffer.

That’s exactly my point. The syntax is not meant to be configurable.
I wrote a patch also removing ‘org-emph-re’ depedency from
"org-element.el", but I was delayed. I just applied it.

> Maybe we need to move the logic for org-emph-re from org.el to
> org-element.el?

‘org-emph-re’ has some limitations which do not belong to syntax
definition. There’s no point in adding it in "org-element.el".

The grand scheme is to remove most "org.el" dependencies from
"org-element.el", and move the others.

> Also, there is org-emphasis-alist. It is even defcustom, but ignored by
> org-element.el.

This variable is a defcustom for the faces, not the markers. I.e., it is
not meant to add, remove, or change emphasis markup, but rather alter
how they appear. IMO, this should be removed altogether: it’s up to
a theme to set such a thing.

Regards,
-- 
Nicolas Goaziou

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> This commit may cause random failures when
>> org-emphasis-regexp-components is changed by user.
>
> This is not supported anyway.

Yeah. Though I have seen people changing this variable.
Maybe we should change defvar to defconst?

>> org-emph-re is calculated according to org-emphasis-regexp-components.
>> Changing org-emphasis-regexp-components can make "(when (looking-at
>> org-emph-re)" in parsers return nil. The emphasised text will still be
>> fontified, but not available in the parsed buffer.
>
> That’s exactly my point. The syntax is not meant to be configurable.
> I wrote a patch also removing ‘org-emph-re’ depedency from
> "org-element.el", but I was delayed. I just applied it.

Thanks!

>> Maybe we need to move the logic for org-emph-re from org.el to
>> org-element.el?
>
> ‘org-emph-re’ has some limitations which do not belong to syntax
> definition. There’s no point in adding it in "org-element.el".
>
> The grand scheme is to remove most "org.el" dependencies from
> "org-element.el", and move the others.

That would be great. I was thinking about unifying the grammar better.
Things like org-set-regexps-and-options define part of the grammar
non-transparently outside org-element. As a result, the new org-persist
library can be potentially broken if the user changes grammar between
Emacs session (e.g. org-off-levels-only, org-todo-keywords, etc). Keeping
all the variables that change the grammar in one place would be helpful.

Maybe we could save the complete grammar state right inside output of
org-element-parse-buffer. In the updated element cache code, I
introduced properties into org-data element. Maybe we can keep all the
important variables for the buffer grammar in org-data? That way, the
job of org-set-regexps-and-options will be done by
org-element-org-data-parser. Moreover, users exporting using ox-org will
be able to create self-sufficient Org files that can be shared without
relying on the same configuration in init.el.

>> Also, there is org-emphasis-alist. It is even defcustom, but ignored by
>> org-element.el.
>
> This variable is a defcustom for the faces, not the markers. I.e., it is
> not meant to add, remove, or change emphasis markup, but rather alter
> how they appear. IMO, this should be removed altogether: it’s up to
> a theme to set such a thing.

Unless I miss something, org-emphasis-alist is used in org-emphasise.
Though it just another reason to remove it.

Best,
Ihor

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Max Nikulin]

On 15/11/2021 22:20, Ihor Radchenko wrote:
> Nicolas Goaziou writes:
>> Ihor Radchenko writes:
>>
>>> This commit may cause random failures when
>>> org-emphasis-regexp-components is changed by user.
>>
>> This is not supported anyway.
> 
> Yeah. Though I have seen people changing this variable.
> Maybe we should change defvar to defconst?

Better docs and some restriction on defcustom values were discussed earlier:
https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon, 
19 Apr 2021 11:27:07 +0200

Sorry, I have not prepared a patch. I am not confident with defcustom 
fine tuning and have not experimented with it since that time.

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> Better docs and some restriction on defcustom values were discussed earlier:
> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon, 
> 19 Apr 2021 11:27:07 +0200
>
> Sorry, I have not prepared a patch. I am not confident with defcustom 
> fine tuning and have not experimented with it since that time.

Maybe something like the attached?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-emphasis-alist-Update-defcustom-making-emphasis-.patch --]
[-- Type: text/x-diff, Size: 2397 bytes --]

From 8057cdb57f6600443b3605c1e7f00a30bea2a9ea Mon Sep 17 00:00:00 2001
Message-Id: <8057cdb57f6600443b3605c1e7f00a30bea2a9ea.1637048505.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Tue, 16 Nov 2021 15:40:35 +0800
Subject: [PATCH] org-emphasis-alist: Update defcustom making emphasis
 characters constant

* lisp/org.el (org-emphasis-alist): Mention that emphasis characters
should not be changed by user.  Update the defcustom type accordingly.
---
 lisp/org.el | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 603b57279..7af5e26c6 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3833,18 +3833,47 @@ (defcustom org-emphasis-alist
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+The characters in the alist must not be changed.  They do not affect
+the actual Org syntax, just fontification.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
-  :type '(repeat
+  :type '(list
 	  (list
-	   (string :tag "Marker character")
+	   (const "*")
+	   (choice
+	    (face :tag "Bold emphasis face")
+	    (plist :tag "Bold emphasis face property list")))
+          (list
+	   (const "/")
+	   (choice
+	    (face :tag "Italic emphasis face")
+	    (plist :tag "Italic emphasis face property list")))
+          (list
+	   (const "_")
+	   (choice
+	    (face :tag "Underline emphasis face")
+	    (plist :tag "Underline emphasis face property list")))
+          (list
+	   (const "=")
+	   (choice
+	    (face :tag "Verbatim emphasis face")
+	    (plist :tag "Verbatim emphasis face property list"))
+	   (const verbatim))
+          (list
+	   (const "~")
+	   (choice
+	    (face :tag "Code emphasis face")
+	    (plist :tag "Code emphasis face property list"))
+	   (const verbatim))
+          (list
+	   (const "+")
 	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+	    (face :tag "Strike-through emphasis face")
+	    (plist :tag "Strike-through emphasis face property list")))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.32.0

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Samuel Wales]

hmm i will have to try not setting o-e-r-c [ordinary user].  probably
a lot of users do.

might be useful to know whether a default regexp change could satisfy
everybody?  in my case i remove " and , from third re.  i also have a
note "org-set-emph-re is what you are supposed to use, but it is
undocumented".  for some reason.  that might be obsolete,


On 11/16/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> Max Nikulin <manikulin@gmail.com> writes:
>
>> Better docs and some restriction on defcustom values were discussed
>> earlier:
>> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
>> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon,
>> 19 Apr 2021 11:27:07 +0200
>>
>> Sorry, I have not prepared a patch. I am not confident with defcustom
>> fine tuning and have not experimented with it since that time.
>
> Maybe something like the attached?
>
> Best,
> Ihor
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Samuel Wales]

i should point out that my changes are old and i don't know if they
have already been done

On 11/16/21, Samuel Wales <samologist@gmail.com> wrote:
> hmm i will have to try not setting o-e-r-c [ordinary user].  probably
> a lot of users do.
>
> might be useful to know whether a default regexp change could satisfy
> everybody?  in my case i remove " and , from third re.  i also have a
> note "org-set-emph-re is what you are supposed to use, but it is
> undocumented".  for some reason.  that might be obsolete,
>
>
> On 11/16/21, Ihor Radchenko <yantar92@gmail.com> wrote:
>> Max Nikulin <manikulin@gmail.com> writes:
>>
>>> Better docs and some restriction on defcustom values were discussed
>>> earlier:
>>> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
>>> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon,
>>> 19 Apr 2021 11:27:07 +0200
>>>
>>> Sorry, I have not prepared a patch. I am not confident with defcustom
>>> fine tuning and have not experimented with it since that time.
>>
>> Maybe something like the attached?
>>
>> Best,
>> Ihor
>>
>>
>
>
> --
> The Kafka Pandemic
>
> Please learn what misopathy is.
> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Max Nikulin]

On 17/11/2021 04:56, Samuel Wales wrote:
> might be useful to know whether a default regexp change could satisfy
> everybody?  in my case i remove " and , from third re.

Samuel, I have seen your next message, but it still may be helpful to 
have an example (for a case if you will meet the problem again).

These regexps will always fail under some conditions, since it is not 
strict markup. An example is emphasis terminated inside link target

/A link [[https://orgmode.org/?oops=true][Org Mode]]

On 16/11/2021 14:43, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Better docs and some restriction on defcustom values were discussed earlier:
>> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
>> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon,
>> 19 Apr 2021 11:27:07 +0200
>>
>> Sorry, I have not prepared a patch. I am not confident with defcustom
>> fine tuning and have not experimented with it since that time.
> 
> Maybe something like the attached?

Thank you, Ihor. Another reason why I have not tried to do it myself is 
that it is necessary to test behavior for users who customized markers. 
The change should not be fatal for them. I have not checked it with you 
patch yet.

I was considering some way to warn users if improper customization is 
detected (unexpected marker is noticed). It should be noticeable to make 
user aware of export issues but not too annoying.

> +The characters in the alist must not be changed.  They do not affect
> +the actual Org syntax, just fontification.

Since this is known point of abuses, maybe stronger words are appropriate.

Do not change makers and do not add new ones to use custom markers for 
existing styles or to introduce new styles. Org syntax is not meant to 
be configurable and such modifications will not work with export. If you 
are interested in fontification of custom markup inside Emacs only, 
there are other ways to achieve desired effect.

In addition, I do not like the following phrase in the manual:

> To narrow down the list of available markup syntax, you can customize org-emphasis-alist.

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Samuel Wales]

i think i found it useful, long ago, when it was ok/tolerated to
change the var, and probably still, to /"do this"/ and /this,/.

although the latter seems weird to me now so i must not now understand
what i was doing.

i think changing the var was at least in a faq or on worg or so, so it
might be needed to compare user's value with standard-value.


On 11/17/21, Max Nikulin <manikulin@gmail.com> wrote:
> On 17/11/2021 04:56, Samuel Wales wrote:
>> might be useful to know whether a default regexp change could satisfy
>> everybody?  in my case i remove " and , from third re.
>
> Samuel, I have seen your next message, but it still may be helpful to
> have an example (for a case if you will meet the problem again).
>
> These regexps will always fail under some conditions, since it is not
> strict markup. An example is emphasis terminated inside link target
>
> /A link [[https://orgmode.org/?oops=true][Org Mode]]
>
> On 16/11/2021 14:43, Ihor Radchenko wrote:
>> Max Nikulin writes:
>>
>>> Better docs and some restriction on defcustom values were discussed
>>> earlier:
>>> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
>>> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon,
>>> 19 Apr 2021 11:27:07 +0200
>>>
>>> Sorry, I have not prepared a patch. I am not confident with defcustom
>>> fine tuning and have not experimented with it since that time.
>>
>> Maybe something like the attached?
>
> Thank you, Ihor. Another reason why I have not tried to do it myself is
> that it is necessary to test behavior for users who customized markers.
> The change should not be fatal for them. I have not checked it with you
> patch yet.
>
> I was considering some way to warn users if improper customization is
> detected (unexpected marker is noticed). It should be noticeable to make
> user aware of export issues but not too annoying.
>
>> +The characters in the alist must not be changed.  They do not affect
>> +the actual Org syntax, just fontification.
>
> Since this is known point of abuses, maybe stronger words are appropriate.
>
> Do not change makers and do not add new ones to use custom markers for
> existing styles or to introduce new styles. Org syntax is not meant to
> be configurable and such modifications will not work with export. If you
> are interested in fontification of custom markup inside Emacs only,
> there are other ways to achieve desired effect.
>
> In addition, I do not like the following phrase in the manual:
>
>> To narrow down the list of available markup syntax, you can customize
>> org-emphasis-alist.
>
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2569 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> These regexps will always fail under some conditions, since it is not 
> strict markup. An example is emphasis terminated inside link target
>
> /A link [[https://orgmode.org/?oops=true][Org Mode]]

Your example actually reveals a much bigger issue with current element
parser. Even though the link is fontified by Org, if you run
org-element-context on the link, it will return nil. The parsed sentence
will look like:

<begin italic>/A link [[https://orgmode.org/<end italic><begin ordinary
text - it is not a link!>?oops=true][Org Mode]]<end of ordinary text>

The reason behind is partially regexps used to detect emphasised text
and partially the way our parser works - no intersected objects are
allowed.

My intuition says that the current parser behaviour is not correct. It
would make more sense to prioritise link over italics. However, it would
require a major change in the parser - instead of a single pass, the
parser may parse different types of objects sequentially. The emphasis
objects should come last avoiding the markers to have different parents.

Nicolas, WDYT?

>> Maybe something like the attached?
>
> Thank you, Ihor. Another reason why I have not tried to do it myself is 
> that it is necessary to test behavior for users who customized markers. 
> The change should not be fatal for them. I have not checked it with you 
> patch yet.
>
> I was considering some way to warn users if improper customization is 
> detected (unexpected marker is noticed). It should be noticeable to make 
> user aware of export issues but not too annoying.
>
>> +The characters in the alist must not be changed.  They do not affect
>> +the actual Org syntax, just fontification.
>
> Since this is known point of abuses, maybe stronger words are appropriate.
>
> Do not change makers and do not add new ones to use custom markers for 
> existing styles or to introduce new styles. Org syntax is not meant to 
> be configurable and such modifications will not work with export. If you 
> are interested in fontification of custom markup inside Emacs only, 
> there are other ways to achieve desired effect.
> ...
> In addition, I do not like the following phrase in the manual:
>
>> To narrow down the list of available markup syntax, you can customize org-emphasis-alist.

I updated the patch dropping your last suggested sentence in the
docstring. I find it not very helpful for the user. We should either say
nothing (as in the patch) or give a concrete reference how to "achieve
the desired effect".

Best,
Ihor

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-emphasis-alist-Update-defcustom-making-emphasis-.patch --]
[-- Type: text/x-diff, Size: 3276 bytes --]

From c950768bc08b4cae07ba3cb451d8af0abfec7dc8 Mon Sep 17 00:00:00 2001
Message-Id: <c950768bc08b4cae07ba3cb451d8af0abfec7dc8.1637237807.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Tue, 16 Nov 2021 15:40:35 +0800
Subject: [PATCH] org-emphasis-alist: Update defcustom making emphasis
 characters constant

* lisp/org.el (org-emphasis-alist): Mention that emphasis characters
should not be changed by user.  Update the defcustom type accordingly.
* doc/org-manual.org (Emphasis and Monospace): Update the referece to
`org-emphasis-alist'.
---
 doc/org-manual.org |  5 +++--
 lisp/org.el        | 41 ++++++++++++++++++++++++++++++++++++-----
 2 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index c4bb7ef30..9fea2c42e 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -10814,9 +10814,10 @@ ** Emphasis and Monospace
 exported verbatim.
 
 #+vindex: org-fontify-emphasized-text
+#+vindex: org-emphasis-alist
 To turn off fontification for marked up text, you can set
-~org-fontify-emphasized-text~ to ~nil~.  To narrow down the list of
-available markup syntax, you can customize ~org-emphasis-alist~.
+~org-fontify-emphasized-text~ to ~nil~.  For more fine-tuned
+fontification, you can customize ~org-emphasis-alist~.
 
 ** Subscripts and Superscripts
 :PROPERTIES:
diff --git a/lisp/org.el b/lisp/org.el
index 603b57279..00ac00ab3 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3833,18 +3833,49 @@ (defcustom org-emphasis-alist
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
-  :type '(repeat
+  :type '(list
 	  (list
-	   (string :tag "Marker character")
+	   (const "*")
+	   (choice
+	    (face :tag "Bold emphasis face")
+	    (plist :tag "Bold emphasis face property list")))
+          (list
+	   (const "/")
+	   (choice
+	    (face :tag "Italic emphasis face")
+	    (plist :tag "Italic emphasis face property list")))
+          (list
+	   (const "_")
+	   (choice
+	    (face :tag "Underline emphasis face")
+	    (plist :tag "Underline emphasis face property list")))
+          (list
+	   (const "=")
+	   (choice
+	    (face :tag "Verbatim emphasis face")
+	    (plist :tag "Verbatim emphasis face property list"))
+	   (const verbatim))
+          (list
+	   (const "~")
+	   (choice
+	    (face :tag "Code emphasis face")
+	    (plist :tag "Code emphasis face property list"))
+	   (const verbatim))
+          (list
+	   (const "+")
 	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+	    (face :tag "Strike-through emphasis face")
+	    (plist :tag "Strike-through emphasis face property list")))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.32.0

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> My intuition says that the current parser behaviour is not correct. It
> would make more sense to prioritise link over italics. However, it would
> require a major change in the parser - instead of a single pass, the
> parser may parse different types of objects sequentially. The emphasis
> objects should come last avoiding the markers to have different parents.
>
> Nicolas, WDYT?

I disagree. Priority should be given to the first object being started.
This is, IMO, the only sane way to handle syntax.

Regards,
-- 
Nicolas Goaziou

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> My intuition says that the current parser behaviour is not correct. It
>> would make more sense to prioritise link over italics. However, it would
>> require a major change in the parser - instead of a single pass, the
>> parser may parse different types of objects sequentially. The emphasis
>> objects should come last avoiding the markers to have different parents.
>>
>> Nicolas, WDYT?
>
> I disagree. Priority should be given to the first object being started.
> This is, IMO, the only sane way to handle syntax.

Multi-pass may indeed over-complicate the syntax.
However, it is not clear then how to handle situations like

/A link [[https://orgmode.org/?oops=true][Org Mode]]
or
/A code ~user/?my-user-variable~ with slash/
or
/A verbatim =text/.= with slash/

Should we modify the closing-re for emphasis?

>> (seq (not space)
>>      (group ,mark)
>>      (or (any space ?- ?. ?, ?\; ?: ?! ?? ?' ?\" ?\) ?\} ?\\ ?\[)
>>          line-end))

Best,
Ihor

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> However, it is not clear then how to handle situations like
>
> /A link [[https://orgmode.org/?oops=true][Org Mode]]
> or
> /A code ~user/?my-user-variable~ with slash/
> or
> /A verbatim =text/.= with slash/

You can use a zero-width space to help Org sorting out the ambiguity,
for example (| denotes the zero-width space):

  /|A link [[https://orgmode.org/?oops=true][Org Mode]]

  /A code ~user|/?my-user-variable~ with slash/

> Should we modify the closing-re for emphasis?
>
>>> (seq (not space)
>>>      (group ,mark)
>>>      (or (any space ?- ?. ?, ?\; ?: ?! ?? ?' ?\" ?\) ?\} ?\\ ?\[)
>>>          line-end))

I don't think it is necessary. /word/? seems a valid sentence closing.

Regards,
-- 
Nicolas Goaziou

[PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> You can use a zero-width space to help Org sorting out the ambiguity,
> for example (| denotes the zero-width space):
>
>   /|A link [[https://orgmode.org/?oops=true][Org Mode]]
>
>   /A code ~user|/?my-user-variable~ with slash/

Makes sense. Maybe we should also mention it in the Markup section of
the manual? I attached a tentative patch.

Also, part of the problem with
/|A link [[https://orgmode.org/?oops=true][Org Mode]]
is that the link is emphasised despite not being parser as a link by
org-element. I attached a patch for our link/emphasis fontification that
makes sure that fontification is always consistent with the parser. The
patch might hit the performance slightly, but I do not see obvious
slowdown using my 15Mb notes file.

Best,
Ihor

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-manual.org-Clarify-how-to-handle-markup-ambiguit.patch --]
[-- Type: text/x-diff, Size: 1410 bytes --]

From 3b4a857582e848e9688a49c01b853ed577dd4935 Mon Sep 17 00:00:00 2001
Message-Id: <3b4a857582e848e9688a49c01b853ed577dd4935.1637321577.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 19 Nov 2021 19:27:56 +0800
Subject: [PATCH] org-manual.org: Clarify how to handle markup ambiguity

* doc/org-manual.org (Emphasis and Monospace): Advice users to insert
zero width space when Org does not parse emphasized text correctly.
---
 doc/org-manual.org | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index a38dbec4a..1db993d3d 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -10818,6 +10818,18 @@ ** Emphasis and Monospace
 ~org-fontify-emphasized-text~ to ~nil~.  To narrow down the list of
 available markup syntax, you can customize ~org-emphasis-alist~.
 
+Sometimes, Org mode has difficulties recognising markup.  It usually
+happens when markup marker symbols are present inside verbatim or code
+blocks:
+
+#+begin_example
+/The whole line is supposed to be marked italic, but the following
+~user/?variable~ contains italics =/= marker and confuses Org parser/.
+#+end_example
+
+You can use zero width space to help Org sorting out the ambiguity.
+See [[*Escape Character]] for more details.
+
 ** Subscripts and Superscripts
 :PROPERTIES:
 :DESCRIPTION: Simple syntax for raising/lowering text.
-- 
2.32.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-org.el-Make-emphasis-and-link-fontification-consiste.patch --]
[-- Type: text/x-diff, Size: 4619 bytes --]

From d5413e772fe6aedb8a8aa094f98c96fc20b82d3a Mon Sep 17 00:00:00 2001
Message-Id: <d5413e772fe6aedb8a8aa094f98c96fc20b82d3a.1637321613.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 19 Nov 2021 19:13:54 +0800
Subject: [PATCH] org.el: Make emphasis and link fontification consistent with
 parser

* lisp/org.el (org-do-emphasis-faces):
(org-activate-links): Do not fontify just based on approximate
regexps.  Make sure the current object is emphasis.
---
 lisp/org.el | 62 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 33 insertions(+), 29 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index cb1b58c51..d9f073100 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -5106,12 +5106,15 @@ (defun org-do-emphasis-faces (limit)
 			       (looking-at-p org-outline-regexp-bol))))
 		   ;; Match full emphasis markup regexp.
 		   (looking-at (if verbatim? org-verbatim-re org-emph-re))
-		   ;; Do not span over paragraph boundaries.
-		   (not (string-match-p org-element-paragraph-separate
-					(match-string 2)))
-		   ;; Do not span over cells in table rows.
-		   (not (and (save-match-data (org-match-line "[ \t]*|"))
-			     (string-match-p "|" (match-string 4))))))
+                   ;; Verify that we are at the right object.
+                   (let ((object (save-excursion
+                                   (save-match-data
+                                     (goto-char (match-beginning 2))
+                                     (org-element-context)))))
+                     (and (memq (org-element-type object)
+                                '(bold italic verbatim code strike-through))
+                          (eq (match-beginning 2)
+                              (org-element-property :begin object))))))
 	    (pcase-let ((`(,_ ,face ,_) (assoc marker org-emphasis-alist))
 			(m (if org-hide-emphasis-markers 4 2)))
 	      (font-lock-prepend-text-property
@@ -5206,7 +5209,7 @@ (defun org-activate-links (limit)
 				 (eq 'org-tag face))))))
 	  (let* ((link-object (save-excursion
 				(goto-char start)
-				(save-match-data (org-element-link-parser))))
+				(save-match-data (org-element-context))))
 		 (link (org-element-property :raw-link link-object))
 		 (type (org-element-property :type link-object))
 		 (path (org-element-property :path link-object))
@@ -5229,29 +5232,30 @@ (defun org-activate-links (limit)
 					((and (pred functionp) f) (funcall f))
 					(_ `(:uri ,link)))
 			'font-lock-multiline t)))
-	    (org-remove-flyspell-overlays-in start end)
-	    (org-rear-nonsticky-at end)
-	    (if (not (eq 'bracket style))
-		(progn
+            (when (eq (org-element-type link-object) 'link)
+              (org-remove-flyspell-overlays-in start end)
+	      (org-rear-nonsticky-at end)
+	      (if (not (eq 'bracket style))
+		  (progn
+                    (add-face-text-property start end face-property)
+		    (add-text-properties start end properties))
+	        ;; Handle invisible parts in bracket links.
+	        (remove-text-properties start end '(invisible nil))
+	        (let ((hidden
+		       (append `(invisible
+			         ,(or (org-link-get-parameter type :display)
+				      'org-link))
+			       properties)))
+		  (add-text-properties start visible-start hidden)
                   (add-face-text-property start end face-property)
-		  (add-text-properties start end properties))
-	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
-	      (let ((hidden
-		     (append `(invisible
-			       ,(or (org-link-get-parameter type :display)
-				    'org-link))
-			     properties)))
-		(add-text-properties start visible-start hidden)
-                (add-face-text-property start end face-property)
-		(add-text-properties visible-start visible-end properties)
-		(add-text-properties visible-end end hidden)
-		(org-rear-nonsticky-at visible-start)
-		(org-rear-nonsticky-at visible-end)))
-	    (let ((f (org-link-get-parameter type :activate-func)))
-	      (when (functionp f)
-		(funcall f start end path (eq style 'bracket))))
-	    (throw :exit t)))))		;signal success
+		  (add-text-properties visible-start visible-end properties)
+		  (add-text-properties visible-end end hidden)
+		  (org-rear-nonsticky-at visible-start)
+		  (org-rear-nonsticky-at visible-end)))
+	      (let ((f (org-link-get-parameter type :activate-func)))
+	        (when (functionp f)
+		  (funcall f start end path (eq style 'bracket))))
+	      (throw :exit t))))))		;signal success
     nil))
 
 (defun org-activate-code (limit)
-- 
2.32.0

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> * doc/org-manual.org (Emphasis and Monospace): Advice users to insert
> zero width space when Org does not parse emphasized text correctly.
> ---

[...]

> +Sometimes, Org mode has difficulties recognising markup.  It usually
> +happens when markup marker symbols are present inside verbatim or code
> +blocks:

I disagree with the wording. Org mode has no difficulties recognizing
markup nor does it parses text incorrectly. This is similar to the well
known surprise:

  #+begin_example
  * something
  #+end_example

Here, we are simply fooled by the fontification process.

Otherwise, the patch looks good to me.

> +                   ;; Verify that we are at the right object.
> +                   (let ((object (save-excursion
> +                                   (save-match-data
> +                                     (goto-char (match-beginning 2))
> +                                     (org-element-context)))))
> +                     (and (memq (org-element-type object)
> +                                '(bold italic verbatim code strike-through))
> +                          (eq (match-beginning 2)
> +                              (org-element-property :begin object))))))

I sympathize with the idea. As long as fontification does not rely on
the parser, we will face issues like the one at stake. So, ultimately,
Org will hopefully move in that direction.

However, I'm not convinced making such local changes is going to help us
in the long run. It may be more fruitful to think this evolution
globally, as it involves a fair bit of rewriting of the fontification
process. For example, it may only be necessary to parse the part of the
Org document being fontified only once, and then apply all fontification
functions to the resulting tree rather than have each of them calling
the parser, like the above.

In any case, I think fontification deserves a dedicated thread, in
addition to some love.

Regards,
-- 
Nicolas Goaziou

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> +Sometimes, Org mode has difficulties recognising markup.  It usually
>> +happens when markup marker symbols are present inside verbatim or code
>> +blocks:
>
> I disagree with the wording. Org mode has no difficulties recognizing
> markup nor does it parses text incorrectly. This is similar to the well
> known surprise:
>
>   #+begin_example
>   * something
>   #+end_example
>
> Here, we are simply fooled by the fontification process.
>
> Otherwise, the patch looks good to me.

I updated the patch. If you have no objections on the new wording, I
will push it to main.

>> +                   ;; Verify that we are at the right object.
>> +                   (let ((object (save-excursion
>> +                                   (save-match-data
>> +                                     (goto-char (match-beginning 2))
>> +                                     (org-element-context)))))
>> +                     (and (memq (org-element-type object)
>> +                                '(bold italic verbatim code strike-through))
>> +                          (eq (match-beginning 2)
>> +                              (org-element-property :begin object))))))
>
> I sympathize with the idea. As long as fontification does not rely on
> the parser, we will face issues like the one at stake. So, ultimately,
> Org will hopefully move in that direction.
>
> However, I'm not convinced making such local changes is going to help us
> in the long run. It may be more fruitful to think this evolution
> globally, as it involves a fair bit of rewriting of the fontification
> process. For example, it may only be necessary to parse the part of the
> Org document being fontified only once, and then apply all fontification
> functions to the resulting tree rather than have each of them calling
> the parser, like the above.
>
> In any case, I think fontification deserves a dedicated thread, in
> addition to some love.

Ok. I will create a new discussion thread on fontification.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-manual.org-Clarify-how-to-handle-markup-ambiguit.patch --]
[-- Type: text/x-diff, Size: 1379 bytes --]

From a1a497a80578669ef1e96700aa592aadd8d0d7ec Mon Sep 17 00:00:00 2001
Message-Id: <a1a497a80578669ef1e96700aa592aadd8d0d7ec.1637329857.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 19 Nov 2021 19:27:56 +0800
Subject: [PATCH] org-manual.org: Clarify how to handle markup ambiguity

* doc/org-manual.org (Emphasis and Monospace): Advice users to insert
zero width space when Org does not parse emphasized text correctly.
---
 doc/org-manual.org | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index a38dbec4a..9615209a1 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -10818,6 +10818,17 @@ ** Emphasis and Monospace
 ~org-fontify-emphasized-text~ to ~nil~.  To narrow down the list of
 available markup syntax, you can customize ~org-emphasis-alist~.
 
+=*=, =/=, =_=, ===, and =~= symbols inside =verbatim= or ~code~ can
+sometimes produce unexpected markup.  For example,
+
+#+begin_example
+/The whole line is supposed to be marked italic, but the following
+~user/?variable~ contains italics =/= marker and confuses Org parser/.
+#+end_example
+
+You can use zero width space to help Org sorting out the ambiguity.
+See [[*Escape Character]] for more details.
+
 ** Subscripts and Superscripts
 :PROPERTIES:
 :DESCRIPTION: Simple syntax for raising/lowering text.
-- 
2.32.0

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Max Nikulin]

On 18/11/2021 05:44, Samuel Wales wrote:
> i think i found it useful, long ago, when it was ok/tolerated to
> change the var, and probably still, to /"do this"/ and /this,/.

D. Knuth in "The TeXbook" put punctuation outside of emphasized text, 
however he mentioned that accordingly to old manuals semicolon should be 
typed in the same font as previous word.

On 18/11/2021 19:25, Ihor Radchenko wrote:
> I updated the patch dropping your last suggested sentence in the
> docstring. I find it not very helpful for the user. We should either say
> nothing (as in the patch) or give a concrete reference how to "achieve
> the desired effect".

I just have never tried to add custom decorations, so it is difficult 
for me to express it by a concise phrase. I even do not know if the 
following recipe could be recommended

https://emacs.stackexchange.com/questions/35626/how-to-make-my-own-org-mode-text-emphasis-work-again

 From other messages I had an impression that it is still possible to 
create custom markup with new marker but actually it was broken years 
ago: 
https://list.orgmode.org/orgmode/CAN9DXH2Xwf+9GGcPrpXFeTFUMeCrxUHtyjduDAYG0Z49FpaJ4g@mail.gmail.com/T/#u
Bug: org-emphasis-alist not fully applied [9.1.1 (9.1.1-1-g80cbf9-elpa @ 
x:/folder/user/.emacs.d/elpa/org-20170918/)] 2017-09-19  4:16

It makes details of implementation less important.

To my taste fixed order of markers (:type list) is too rigid. If a user 
reordered a couple of items then multi-value customization form becomes 
single multiline input area. :set function might sort items, but I do 
not like such solution.

Since I have not found a way to improve customization interface with 
reasonable efforts, I believe that the patch is acceptable. It should be 
an improvement, but not an outstanding one.

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Max Nikulin]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On 16/11/2021 14:43, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Better docs and some restriction on defcustom values were discussed earlier:
>> https://list.orgmode.org/87k0oyd3pw.fsf@nicolasgoaziou.fr/
>> Nicolas Goaziou. Re: Using backticks for the inline code delimeter? Mon,
>> 19 Apr 2021 11:27:07 +0200
>>
>> Sorry, I have not prepared a patch. I am not confident with defcustom
>> fine tuning and have not experimented with it since that time.
> 
> Maybe something like the attached?

My draft version is attached. Ihor, thank you for inspiration. Feel free 
to improve it. I hope, it makes problem more apparent to user who tries 
to customize markers. Have I missed some undesired side effects?

Unfortunately with "set" type in defcustom "catch inappropriate" item is 
not hidden for valid configuration.

I do not mind if you commit any variant.

[-- Attachment #2: restrict-org-emphasis-alist.patch --]
[-- Type: text/x-patch, Size: 3680 bytes --]

diff --git a/doc/org-manual.org b/doc/org-manual.org
index a38dbec4a..b62c52e61 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -10814,9 +10814,11 @@ and verbatim string is not processed for Org specific syntax; it is
 exported verbatim.
 
 #+vindex: org-fontify-emphasized-text
+#+vindex: org-emphasis-alist
 To turn off fontification for marked up text, you can set
-~org-fontify-emphasized-text~ to ~nil~.  To narrow down the list of
-available markup syntax, you can customize ~org-emphasis-alist~.
+~org-fontify-emphasized-text~ to ~nil~.  For more fine-tuned
+fontification consider themes. It is possible to customize
+~org-emphasis-alist~ variable, but it will be deprecated.
 
 ** Subscripts and Superscripts
 :PROPERTIES:
diff --git a/lisp/org.el b/lisp/org.el
index cb1b58c51..ea62ae0b2 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3771,6 +3771,25 @@ newline      The maximum number of newlines allowed in an emphasis exp.
 
 You need to reload Org or to restart Emacs after setting this.")
 
+(defun org-set-emphasis-alist (var value)
+  "Set VAR (`org-emphasis-alist') to VALUE and check it for ignored characters.
+Warn user that Org syntax can not be extended with new emphasis markers
+if such attempt is detected.  The function is intended for :set argument
+of `defcustom'."
+  (set var value)
+  (let ((unsupported
+	 (delq nil
+	       (mapcar
+                (lambda (entry)
+                  (let ((marker (car entry)))
+                    (unless (member marker '("*" "/" "_" "=" "~" "+")) marker)))
+                value))))
+    (when unsupported
+      (message "Warning! Unsupported markup characters '%s' detected in `%s'"
+	       (mapconcat #'identity unsupported " ")
+	       (symbol-name var))))
+  value)
+
 (defcustom org-emphasis-alist
   '(("*" bold)
     ("/" italic)
@@ -3779,23 +3798,41 @@ You need to reload Org or to restart Emacs after setting this.")
     ("~" org-code verbatim)
     ("+" (:strike-through t)))
   "Alist of characters and faces to emphasize text.
+Warning! This variable will be deprecated in favor of themes.
+
 Text starting and ending with a special character will be emphasized,
 for example *bold*, _underlined_ and /italic/.  This variable sets the
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
+  :set #'org-set-emphasis-alist
   :type '(repeat
-	  (list
-	   (string :tag "Marker character")
-	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+          (group
+           (choice
+            :tag "Marker"
+            (const :tag "*Bold*" "*")
+            (const :tag "/Italic/" "/")
+            (const :tag "_Underline_" "_")
+            (const :tag "+Strike-through+" "+")
+            (const :tag "=Verbatim=" "=")
+            (const :tag "~Code~" "~")
+            ;; To warn users that it does not work.
+            (string :tag "Unsupported ignored character"))
+           (choice
+            :tag "Font"
+            (face :tag "Face")
+            (plist :tag "Property list"))
+           (option (const verbatim)))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> I updated the patch. If you have no objections on the new wording, I
> will push it to main.

Thanks for the update, and apologies in advance for being bold, as
I have some additional comments about it.

> * doc/org-manual.org (Emphasis and Monospace): Advice users to insert
> zero width space when Org does not parse emphasized text correctly.

Org _does_ parse emphasized text correctly. It may be seen as
unintuitive, but it's really a fontification problem. Anyway, this is
just a commit message…

> +=*=, =/=, =_=, ===, and =~= symbols inside =verbatim= or ~code~ can
> +sometimes produce unexpected markup.  

OK, but it's not limited to symbols within verbatim or code. What about
something like:

  Sometimes, when marked text also contains the marker character itself,
  the result may be unsettling.

  ...example follows (see below)...

> +#+begin_example
> +/The whole line is supposed to be marked italic, but the following
> +~user/?variable~ contains italics =/= marker and confuses Org parser/.
> +#+end_example

The whole line is not supposed to be marked as italic, as long as we
follow Org syntax. And the parser is not confused at all. The user may
be, however.

I suggest:

  /One may expect this whole sentence to be italicized, but the
  following ~user/?variable~ contains =/= character, which effectively
  stops emphasis there./

> +You can use zero width space to help Org sorting out the ambiguity.
> +See [[*Escape Character]] for more details.

Thinking about it a bit more, you might be right: we may slightly change
the closing part of the emphasis regexp, e.g.:

  (seq
   (not space)
   (group ,mark)
   (or (any space ?- ?')
       (and (any ?. ?, ?\; ?: ?! ?? ?\" ?\) ?\} ?\\ ?\[) (or space line-end))
       line-end))

The logic behind this is that in regular text, we assume usual
punctuation rules apply.

My concern is that the more complicated is the rule, the more difficult
it is to predict. Also, we introduce new corner case, e.g.,

  Woot! I just released Org *10*.0!

So, I'm not totally convinced it is worth the trouble.

Regards,
-- 
Nicolas Goaziou

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Thanks for the update, and apologies in advance for being bold, as
> I have some additional comments about it.

Constructive critics and suggestions are always welcome. And we do not
have pressing deadlines here :)

>> * doc/org-manual.org (Emphasis and Monospace): Advice users to insert
>> zero width space when Org does not parse emphasized text correctly.
>
> Org _does_ parse emphasized text correctly. It may be seen as
> unintuitive, but it's really a fontification problem. Anyway, this is
> just a commit message…

Agree. It just that the example in the patch _feels_ wrong considering
intuitive definition of verbatim borrowed from LaTeX.

Commit messages are also important, especially years later. I updated
the commit message in the attached new version of the patch.

> Thinking about it a bit more, you might be right: we may slightly change
> the closing part of the emphasis regexp, e.g.:
>
>   (seq
>    (not space)
>    (group ,mark)
>    (or (any space ?- ?')
>        (and (any ?. ?, ?\; ?: ?! ?? ?\" ?\) ?\} ?\\ ?\[) (or space line-end))
>        line-end))
>
> The logic behind this is that in regular text, we assume usual
> punctuation rules apply.

This will fail for "*Bold*?!" or "/Italics/!!!"

Also, is there any reason why we are not simply using punctuation
character class instead of listing punctuation chars explicitly (and
only for English)? What about "_你叫什么名字_？"

Maybe just

(seq
 (not space)
 (group ,mark)
 (0+ (in punctuation))
 (or space line-end))

> My concern is that the more complicated is the rule, the more difficult
> it is to predict. Also, we introduce new corner case, e.g.,
>
>   Woot! I just released Org *10*.0!
>
> So, I'm not totally convinced it is worth the trouble.

I am not sure if "Org *10*.0" is a good general example. It is probably
one of those cases when users want fine control over emphasis and must
use zero width space.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-manual.org-Clarify-how-to-handle-markup-ambiguit.patch --]
[-- Type: text/x-diff, Size: 1380 bytes --]

From 9ad522e8d1f1184ef097611fc30b326b08d5b432 Mon Sep 17 00:00:00 2001
Message-Id: <9ad522e8d1f1184ef097611fc30b326b08d5b432.1637486504.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 19 Nov 2021 19:27:56 +0800
Subject: [PATCH] org-manual.org: Clarify how to handle markup ambiguity

* doc/org-manual.org (Emphasis and Monospace): Advice users to insert
zero width space to force Org ignore emphasis markers.
---
 doc/org-manual.org | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 1d0213934..19f42fc77 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -10818,6 +10818,18 @@ ** Emphasis and Monospace
 ~org-fontify-emphasized-text~ to ~nil~.  To narrow down the list of
 available markup syntax, you can customize ~org-emphasis-alist~.
 
+Sometimes, when marked text also contains the marker character itself,
+the result may be unsettling.  For example,
+
+#+begin_example
+/One may expect this whole sentence to be italicized, but the
+following ~user/?variable~ contains =/= character, which effectively
+stops emphasis there./
+#+end_example
+
+You can use zero width space to help Org sorting out the ambiguity.
+See [[*Escape Character]] for more details.
+
 ** Subscripts and Superscripts
 :PROPERTIES:
 :DESCRIPTION: Simple syntax for raising/lowering text.
-- 
2.32.0

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> My draft version is attached. Ihor, thank you for inspiration. Feel free 
> to improve it. I hope, it makes problem more apparent to user who tries 
> to customize markers. Have I missed some undesired side effects?

Your version looks a lot better. Thanks!

> Unfortunately with "set" type in defcustom "catch inappropriate" item is 
> not hidden for valid configuration.

I am not very familiar with :set customisation. Could you elaborate what
you mean? I do not understand what "catch inappropriate" refers to.

> + ... It is possible to customize
> +~org-emphasis-alist~ variable, but it will be deprecated.

We probably do not need to mention to-be-deprecated variables in manual.
It will only confuse users.

Maybe we should also take a chance and deprecate org-emphasis-alist now?
We can introduce org-bold/italics/... faces and use them in
org-emphasis-alist default value.

Best,
Ihor

Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Max Nikulin]

On 21/11/2021 17:01, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> My draft version is attached. Ihor, thank you for inspiration. Feel free
>> to improve it. I hope, it makes problem more apparent to user who tries
>> to customize markers. Have I missed some undesired side effects?
> 
>> Unfortunately with "set" type in defcustom "catch inappropriate" item is
>> not hidden for valid configuration.
 >
> I am not very familiar with :set customisation. Could you elaborate what
> you mean? I do not understand what "catch inappropriate" refers to.

I am afraid of confusion of ":set" defcustom parameter and "set" type in 
":type" parameter.

I mean "set" instead of "repeat" (or "list" in your patch). It is 
collection of *unique* items. Other variants allow to add several 
entries for the same marker, e.g. "*". Set does not allow users to add 
duplicated entries. I started from your description of allowed entries
and changed rigid "list" type to "set" and added (approximately)

           (list :tag "Unsupported ignored"
	   (string :tag "Marker")
	   (choice :tag "Font"
	    (face :tag "Face")
	    (plist :tag "Property list"))
	   (option (const verbatim))

to notify user that some part of configuration does not work, it is 
always shown in `customize-variable' interface, even as a disabled 
option for default configuration.

With repeat, "Unsupported" options are not present in customization 
buffer, unless user has something like

(custom-set-variables
   '(org-emphasis-alist
      (quote
        (
	("*" bold)
	("_" underline)
	("=" org-verbatim verbatim)
	("/" italic)
	("~" org-code verbatim)
	("!" org-todo) ;; does not work
	("+" (:strike-through t)))))
   )

It is possible to add "seq" entry to catch list entries having 
completely invalid structure, but I decided that it is redundant.

So with "repeat" type user may notice "Unsupported ignored" option in 
customization interface. I think, it is not enough, so I added ":set" 
function that issues a warning during load time. Invalid configuration 
may still remain hidden in the case of `setq' instead of 
`custom-set-variables' in the init file or `eval-after-load' hook. I am 
afraid, nothing could be done to reliably detect the last case.

>> + ... It is possible to customize
>> +~org-emphasis-alist~ variable, but it will be deprecated.
> 
> We probably do not need to mention to-be-deprecated variables in manual.
> It will only confuse users.

I do not insist. My intention is to inform users that wide spread 
recipes should be avoided.

> Maybe we should also take a chance and deprecate org-emphasis-alist now?
> We can introduce org-bold/italics/... faces and use them in
> org-emphasis-alist default value.

I do not mind. I have never experimented with themes, so I would leave 
this part of work to someone else.

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> Commit messages are also important, especially years later. I updated
> the commit message in the attached new version of the patch.

Note I'm not saying commit messages are not important. I just won't
spend energy on the wording there.

>> Thinking about it a bit more, you might be right: we may slightly change
>> the closing part of the emphasis regexp, e.g.:
>>
>>   (seq
>>    (not space)
>>    (group ,mark)
>>    (or (any space ?- ?')
>>        (and (any ?. ?, ?\; ?: ?! ?? ?\" ?\) ?\} ?\\ ?\[) (or space line-end))
>>        line-end))
>>
>> The logic behind this is that in regular text, we assume usual
>> punctuation rules apply.
>
> This will fail for "*Bold*?!" or "/Italics/!!!"

Of course. Any regexp will fail somehow.

> Also, is there any reason why we are not simply using punctuation
> character class instead of listing punctuation chars explicitly (and
> only for English)? What about "_你叫什么名字_？"
>
> Maybe just
>
> (seq
>  (not space)
>  (group ,mark)
>  (0+ (in punctuation))
>  (or space line-end))

Historically, Org only focused on ASCII. But it makes sense to extend
the allowed punctuation characters, indeed.

This is orthogonal to OP's issue, however.

>> My concern is that the more complicated is the rule, the more difficult
>> it is to predict. Also, we introduce new corner case, e.g.,
>>
>>   Woot! I just released Org *10*.0!
>>
>> So, I'm not totally convinced it is worth the trouble.
>
> I am not sure if "Org *10*.0" is a good general example. It is probably
> one of those cases when users want fine control over emphasis and must
> use zero width space.

This is simply the first example that crossed my mind. My point is that
changing the regexp substantially may not be rewarding, ultimately.

> +Sometimes, when marked text also contains the marker character itself,
> +the result may be unsettling.  For example,
> +
> +#+begin_example
> +/One may expect this whole sentence to be italicized, but the
> +following ~user/?variable~ contains =/= character, which effectively
> +stops emphasis there./
> +#+end_example
> +
> +You can use zero width space to help Org sorting out the ambiguity.
> +See [[*Escape Character]] for more details.

LGTM!

Regards,
-- 
Nicolas Goaziou

Re: [PATCH] Re: c47b535bb origin/main org-element: Remove dependency on ‘org-emphasis-regexp-components’

[Ihor Radchenko]

>> I am not sure if "Org *10*.0" is a good general example. It is probably
>> one of those cases when users want fine control over emphasis and must
>> use zero width space.
>
> This is simply the first example that crossed my mind. My point is that
> changing the regexp substantially may not be rewarding, ultimately.

Agree. At least not until we find a clearly better variant.

Yet, I do not like the current state of things. Especially with regards
to mixing emphasis and hidden parts of links. Situations like in
/A link [[https://orgmode.org/?oops=true][Org Mode]]
or like reported in
https://list.orgmode.org/orgmode/87pmtqp79s.fsf@web.de/ can be very hard
to notice and cause "mysterious" Org failures.

>> +Sometimes, when marked text also contains the marker character itself,
>> +the result may be unsettling.  For example,
[ 8 more citation lines. Click/Enter to show. ]
>> +
>> +#+begin_example
>> +/One may expect this whole sentence to be italicized, but the
>> +following ~user/?variable~ contains =/= character, which effectively
>> +stops emphasis there./
>> +#+end_example
>> +
>> +You can use zero width space to help Org sorting out the ambiguity.
>> +See [[*Escape Character]] for more details.
>
> LGTM!

Applied.

Best,
Ihor

[PATCH] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Max Nikulin]

[-- Attachment #1: Type: text/plain, Size: 1989 bytes --]

On 21/11/2021 17:01, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> My draft version is attached. Ihor, thank you for inspiration. Feel free
>> to improve it. I hope, it makes problem more apparent to user who tries
>> to customize markers.

A patch is attached. I have dropped changes in documentation since I am 
not the author of them.

For init file (entries may be in arbitrary order)

(custom-set-variables
   '(org-emphasis-alist
      (quote
        (
	("*" bold)
	("_" underline)
	("=" org-verbatim verbatim)
	("/" italic)
	("~" org-code verbatim)
	("!" org-todo) ;; does not work
	("+" (:strike-through t)))))
   )

emacs --eval "(customize-variable 'org-emphasis-alist)"

looks as the following

Hide Org Emphasis Alist:
[INS] [DEL] :
             Marker: [Value Menu] *Bold*
             Font: [Value Menu] Face: (sample) bold 
 
 

             [ ] verbatim 
 
 

[INS] [DEL] :
             Marker: [Value Menu] _Underline_
             Font: [Value Menu] Face: (sample) underline 
 
 

             [ ] verbatim 
 
 

[INS] [DEL] :
             Marker: [Value Menu] =Verbatim=
             Font: [Value Menu] Face: (sample) org-verbatim 
 
 

             [X] verbatim
[INS] [DEL] :
             Marker: [Value Menu] /Italic/
             Font: [Value Menu] Face: (sample) italic 
 
 

             [ ] verbatim 
 
 

[INS] [DEL] :
             Marker: [Value Menu] ~Code~
             Font: [Value Menu] Face: (sample) org-code 
 
 

             [X] verbatim
[INS] [DEL] :
             Marker: [Value Menu] Unsupported ignored character: ! 
 
 

             Font: [Value Menu] Face: (sample) org-todo 
 
 

             [ ] verbatim 
 
 

[INS] [DEL] :
             Marker: [Value Menu] +Strike-through+
             Font: [Value Menu] Property list:
                         [INS] [DEL] :
                         Key: :strike-through 
 
 

                         Value: t 
 
 

             [INS]
             [ ] verbatim 
 
 

[INS]
    [ State ]: SAVED and set.

[-- Attachment #2: 0001-org.el-Warning-for-unsupported-markers-in-org-set-em.patch --]
[-- Type: text/x-patch, Size: 3361 bytes --]

From 1ee52f7ffc1039fd442775e9267403f1dca86b88 Mon Sep 17 00:00:00 2001
From: Max Nikulin <manikulin@gmail.com>
Date: Mon, 22 Nov 2021 23:56:15 +0700
Subject: [PATCH] org.el: Warning for unsupported markers in
 `org-set-emphasis-alist'

* lisp/org.el (org-emphasis-alist, org-set-emphasis-alist): Change
custom variable type definition and add :set parameter to warn users
that non-standard marker characters are ignored.

Attempts to introduce new markers have been discussed enough times to
add some code that should prevent wasting of time.

Unfortunately there is no way to issue warning for e.g. `setq'.
---
 lisp/org.el | 47 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 41 insertions(+), 6 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index df3d139c7..1a65b6db8 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3771,6 +3771,25 @@ newline      The maximum number of newlines allowed in an emphasis exp.
 
 You need to reload Org or to restart Emacs after setting this.")
 
+(defun org-set-emphasis-alist (var value)
+  "Set VAR (`org-emphasis-alist') to VALUE and check it for ignored characters.
+Warn user that Org syntax can not be extended with new emphasis markers
+if such attempt is detected.  The function is intended for :set argument
+of `defcustom'."
+  (set var value)
+  (let ((unsupported
+	 (delq nil
+	       (mapcar
+                (lambda (entry)
+                  (let ((marker (car entry)))
+                    (unless (member marker '("*" "/" "_" "=" "~" "+")) marker)))
+                value))))
+    (when unsupported
+      (message "Warning! Unsupported markup characters '%s' detected in `%s'"
+	       (mapconcat #'identity unsupported " ")
+	       (symbol-name var))))
+  value)
+
 (defcustom org-emphasis-alist
   '(("*" bold)
     ("/" italic)
@@ -3784,18 +3803,34 @@ for example *bold*, _underlined_ and /italic/.  This variable sets the
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
+  :set #'org-set-emphasis-alist
   :type '(repeat
-	  (list
-	   (string :tag "Marker character")
-	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+          (group
+           (choice
+            :tag "Marker"
+            (const :tag "*Bold*" "*")
+            (const :tag "/Italic/" "/")
+            (const :tag "_Underline_" "_")
+            (const :tag "+Strike-through+" "+")
+            (const :tag "=Verbatim=" "=")
+            (const :tag "~Code~" "~")
+            ;; To warn users that it does not work.
+            (string :tag "Unsupported ignored character"))
+           (choice
+            :tag "Font"
+            (face :tag "Face")
+            (plist :tag "Property list"))
+           (option (const verbatim)))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.25.1

org parser and priorities of inline elements

[Max Nikulin]

On 21/11/2021 16:28, Ihor Radchenko wrote:
> 
> Also, is there any reason why we are not simply using punctuation
> character class instead of listing punctuation chars explicitly (and
> only for English)? What about "_你叫什么名字_？"

It seems punctuation character class is too broad. E.g.
     ¿ INVERTED QUESTION MARK
normally appears before words, while "?" is usually after them. I do not 
see anything special in
     (category-set-mnemonics (char-category-set ?¿))
that may help to discriminate such cases.

An example that confuses fontification but not parser:
     : false [[http://te.st/dir?b-=&a=-][verbatim]] fontification
It is a simplified example, original one:
Chris Hunt. Bug: Tildes in URL impact visible link text
Sun, 27 Dec 2020 11:44:07 -0500.
https://list.orgmode.org/CAH+Wm4-_XHUZKFTf=ZtbfnCPvQWkbEoeGs8EpYm+8SPmu8LHFg@mail.gmail.com/

Nicolas Goaziou. Thu, 18 Nov 2021 13:35:19 +0100.
https://list.orgmode.org/87y25l8wvs.fsf@nicolasgoaziou.fr
> Ihor Radchenko writes:
> 
>> My intuition says that the current parser behaviour is not correct. It
>> would make more sense to prioritise link over italics. However, it would
>> require a major change in the parser - instead of a single pass, the
>> parser may parse different types of objects sequentially. The emphasis
>> objects should come last avoiding the markers to have different parents.
> 
> I disagree. Priority should be given to the first object being started.
> This is, IMO, the only sane way to handle syntax.

Origin of such expectation is not only TeX that changes category of 
characters for argument of verbatim commands. In markdown links and code 
have higher priorities than emphasis as well:

     echo 'A _b `c_ d` e_ f' | pandoc -f markdown -t html -
     <p>A <em>b <code>c_ d</code> e</em> f</p>

Org:
     A _b =c_ d= e_ f
export result (it is more concise and easier to read than output of 
`org-element-parse-secondary-string'):
     <p>
     A <span class="underline">b =c</span> d= e_ f
     </p>

Link in markdown:

     echo 'A _b c <https://orgmode.org/index.htm_?k=v> d e_ f' \
          | pandoc -f markdown -t html -
     <p>A <em>b c <a href="https://orgmode.org/index.htm_?k=v" 
class="uri">https://orgmode.org/index.htm_?k=v</a> d e</em> f</p>

Org:
     <p>
     A <span class="underline">b /c &lt;<a 
href="https://orgmode.org/index.htm">https://orgmode.org/index.htm</a></span>?k=v&gt; 
d/ e_ f
     </p>

I can not estimate efforts necessary to implement priorities of objects 
(verbatim - link - emphasis) in org-elements parser since I have not 
looked into its code. Comparing the following snippets, I might naively 
expect some kind of backtracking:

- A /b *c +d e+ f*    g/ h
- A /b *c +d    f* e+    h

I admit that I can be wrong and "first wins" approach handles buffer of 
incomplete parsed entities in a different way.

P.S. In reStructured text simple nesting is not allowed, maybe it is 
possible to use replacements.

Re: org parser and priorities of inline elements

[Nicolas Goaziou]

Hello,

Max Nikulin <manikulin@gmail.com> writes:

> I can not estimate efforts necessary to implement priorities of
> objects (verbatim - link - emphasis) in org-elements parser since
> I have not looked into its code. Comparing the following snippets,
> I might naively expect some kind of backtracking:
>
> - A /b *c +d e+ f*    g/ h
> - A /b *c +d    f* e+    h
>
> I admit that I can be wrong and "first wins" approach handles buffer
> of incomplete parsed entities in a different way.

I don't see any incentive to change the order objects are parsed, once
you know how Org does it. This is just a red herring. What is useful,
however, is to fontify them the way Org sees them.

Regards,
-- 
Nicolas Goaziou

Re: [PATCH] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> On 21/11/2021 17:01, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>> My draft version is attached. Ihor, thank you for inspiration. Feel free
>>> to improve it. I hope, it makes problem more apparent to user who tries
>>> to customize markers.
>
> A patch is attached. I have dropped changes in documentation since I am 
> not the author of them.

Sorry for the late reply.

I have reviewed the patch, and I'd like to suggest a new version with the
following changes:
1. Use `set-default-toplevel-value' instead of `set' that might be
   quirky in some scenarios.
2. Use `warn' instead of `message' as more alarming.
3. Remove verbatim in ("=" org-verbatim verbatim), ("~" org-code
   verbatim), and the :type spec.  AFAIU, they are unused. But can you
   please double-check?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-org.el-Warn-about-unsupported-markers-in-org-set-.patch --]
[-- Type: text/x-patch, Size: 3710 bytes --]

From 919fc426f298755886f6f3df22ce9670a7cf67c6 Mon Sep 17 00:00:00 2001
Message-Id: <919fc426f298755886f6f3df22ce9670a7cf67c6.1667544252.git.yantar92@posteo.net>
From: Max Nikulin <manikulin@gmail.com>
Date: Mon, 22 Nov 2021 23:56:15 +0700
Subject: [PATCH v3] org.el: Warn about unsupported markers in
 `org-set-emphasis-alist'

* lisp/org.el (org-emphasis-alist, org-set-emphasis-alist): Change
custom variable type definition and add :set parameter to warn users
that non-standard marker characters are ignored.  Remove unused third
list entry from the default value.

Attempts to introduce new markers have been discussed enough times to
add some code that should prevent wasting of time.

Unfortunately there is no way to issue warning for e.g. `setq'.

Link: https://orgmode.org/list/878rxoa6lk.fsf@localhost
---
 lisp/org.el | 50 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 42 insertions(+), 8 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index d8708f8f2..43be34daf 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3628,12 +3628,31 @@ (defvar org-emphasis-regexp-components
 
 You need to reload Org or to restart Emacs after setting this.")
 
+(defun org-set-emphasis-alist (var value)
+  "Set VAR (`org-emphasis-alist') to VALUE.
+Warn user that Org syntax can not be extended with new emphasis
+markers if such attempt is detected.  The function is intended for
+:set argument of `defcustom'."
+  (set-default-toplevel-value var value)
+  (let ((unsupported
+	 (delq nil
+	       (mapcar
+                (lambda (entry)
+                  (let ((marker (car entry)))
+                    (unless (member marker '("*" "/" "_" "=" "~" "+")) marker)))
+                value))))
+    (when unsupported
+      (warn "Unsupported markup characters '%s' detected in `%s'"
+	    (mapconcat #'identity unsupported " ")
+	    (symbol-name var))))
+  value)
+
 (defcustom org-emphasis-alist
   '(("*" bold)
     ("/" italic)
     ("_" underline)
-    ("=" org-verbatim verbatim)
-    ("~" org-code verbatim)
+    ("=" org-verbatim)
+    ("~" org-code)
     ("+" (:strike-through t)))
   "Alist of characters and faces to emphasize text.
 Text starting and ending with a special character will be emphasized,
@@ -3641,18 +3660,33 @@ (defcustom org-emphasis-alist
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
+  :set #'org-set-emphasis-alist
   :type '(repeat
-	  (list
-	   (string :tag "Marker character")
-	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+          (group
+           (choice
+            :tag "Marker"
+            (const :tag "*Bold*" "*")
+            (const :tag "/Italic/" "/")
+            (const :tag "_Underline_" "_")
+            (const :tag "+Strike-through+" "+")
+            (const :tag "=Verbatim=" "=")
+            (const :tag "~Code~" "~")
+            ;; To warn users that it does not work.
+            (string :tag "Unsupported ignored character"))
+           (choice
+            :tag "Font"
+            (face :tag "Face")
+            (plist :tag "Property list")))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Max Nikulin]

On 04/11/2022 13:53, Ihor Radchenko wrote:
> 
> I have reviewed the patch, and I'd like to suggest a new version with the
> following changes:

Great! I have not tried your patch in action, but I do not see 
substantial changes.

> 1. Use `set-default-toplevel-value' instead of `set' that might be
>     quirky in some scenarios.

Almost certainly it is an improvement. Docs in the Emacs repo have been 
updated, but the published variant still suggests `set-default' 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Variable-Definitions.html

> 2. Use `warn' instead of `message' as more alarming.

I do not mind.

> 3. Remove verbatim in ("=" org-verbatim verbatim), ("~" org-code
>     verbatim), and the :type spec.  AFAIU, they are unused. But can you
>     please double-check?

It seems, before the following commit, verbatim was used to suppress 
flyspell, but now "~" and "=" are hardcoded.

9fb2e047d 2016-12-08 09:44:26 +0100 Nicolas Goaziou: Split `org-emph-re' 
and `org-verbatim-re'

I think, verbatim option should not be removed from `defcustom' :type 
just now. For some users it might cause fallback to raw lisp value in 
easy customization UI. Perhaps another warning should be added to the 
`org-set-emphasis-alist' validator and the option should be labeled as 
unused in the :type definition.

Re: [PATCH] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

>> 3. Remove verbatim in ("=" org-verbatim verbatim), ("~" org-code
>>     verbatim), and the :type spec.  AFAIU, they are unused. But can you
>>     please double-check?
>
> It seems, before the following commit, verbatim was used to suppress 
> flyspell, but now "~" and "=" are hardcoded.
>
> 9fb2e047d 2016-12-08 09:44:26 +0100 Nicolas Goaziou: Split `org-emph-re' 
> and `org-verbatim-re'
>
> I think, verbatim option should not be removed from `defcustom' :type 
> just now. For some users it might cause fallback to raw lisp value in 
> easy customization UI. Perhaps another warning should be added to the 
> `org-set-emphasis-alist' validator and the option should be labeled as 
> unused in the :type definition.

Makes sense.
What about the attached patch?
The idea is to tag the constant as deprecated.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v4-0001-org.el-Warn-about-unsupported-markers-in-org-set-.patch --]
[-- Type: text/x-patch, Size: 3784 bytes --]

From 19f636e0d76dac3e1ca133adfac8bf97dfd52a68 Mon Sep 17 00:00:00 2001
Message-Id: <19f636e0d76dac3e1ca133adfac8bf97dfd52a68.1667636440.git.yantar92@posteo.net>
From: Max Nikulin <manikulin@gmail.com>
Date: Mon, 22 Nov 2021 23:56:15 +0700
Subject: [PATCH v4] org.el: Warn about unsupported markers in
 `org-set-emphasis-alist'

* lisp/org.el (org-emphasis-alist, org-set-emphasis-alist): Change
custom variable type definition and add :set parameter to warn users
that non-standard marker characters are ignored.  Remove unused third
list entry from the default value.

Attempts to introduce new markers have been discussed enough times to
add some code that should prevent wasting of time.

Unfortunately there is no way to issue warning for e.g. `setq'.

Link: https://orgmode.org/list/878rxoa6lk.fsf@localhost
---
 lisp/org.el | 51 +++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 43 insertions(+), 8 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index d8708f8f2..84ed78ef7 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3628,12 +3628,31 @@ (defvar org-emphasis-regexp-components
 
 You need to reload Org or to restart Emacs after setting this.")
 
+(defun org-set-emphasis-alist (var value)
+  "Set VAR (`org-emphasis-alist') to VALUE.
+Warn user that Org syntax can not be extended with new emphasis
+markers if such attempt is detected.  The function is intended for
+:set argument of `defcustom'."
+  (set-default-toplevel-value var value)
+  (let ((unsupported
+	 (delq nil
+	       (mapcar
+                (lambda (entry)
+                  (let ((marker (car entry)))
+                    (unless (member marker '("*" "/" "_" "=" "~" "+")) marker)))
+                value))))
+    (when unsupported
+      (warn "Unsupported markup characters '%s' detected in `%s'"
+	    (mapconcat #'identity unsupported " ")
+	    (symbol-name var))))
+  value)
+
 (defcustom org-emphasis-alist
   '(("*" bold)
     ("/" italic)
     ("_" underline)
-    ("=" org-verbatim verbatim)
-    ("~" org-code verbatim)
+    ("=" org-verbatim)
+    ("~" org-code)
     ("+" (:strike-through t)))
   "Alist of characters and faces to emphasize text.
 Text starting and ending with a special character will be emphasized,
@@ -3641,18 +3660,34 @@ (defcustom org-emphasis-alist
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
   :version "24.4"
   :package-version '(Org . "8.0")
+  :set #'org-set-emphasis-alist
   :type '(repeat
-	  (list
-	   (string :tag "Marker character")
-	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+          (group
+           (choice
+            :tag "Marker"
+            (const :tag "*Bold*" "*")
+            (const :tag "/Italic/" "/")
+            (const :tag "_Underline_" "_")
+            (const :tag "+Strike-through+" "+")
+            (const :tag "=Verbatim=" "=")
+            (const :tag "~Code~" "~")
+            ;; To warn users that it does not work.
+            (string :tag "Unsupported ignored character"))
+           (choice
+            :tag "Font"
+            (face :tag "Face")
+            (plist :tag "Property list"))
+           (option (const :tag "Deprecated ignored constant" verbatim)))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I think, verbatim option should not be removed from `defcustom' :type 
>> just now. For some users it might cause fallback to raw lisp value in 
>> easy customization UI. Perhaps another warning should be added to the 
>> `org-set-emphasis-alist' validator and the option should be labeled as 
>> unused in the :type definition.
>
> Makes sense.
> What about the attached patch?
> The idea is to tag the constant as deprecated.

I further improved the patch adding variable watcher. This should catch
everything. See the attached.

Let me know if there are any objections.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v5-0001-org.el-Warn-about-unsupported-markers-in-org-set-.patch --]
[-- Type: text/x-patch, Size: 4414 bytes --]

From 78c61c6cc6fe95ab4d661c86ee8b3902a499cd0e Mon Sep 17 00:00:00 2001
Message-Id: <78c61c6cc6fe95ab4d661c86ee8b3902a499cd0e.1675335125.git.yantar92@posteo.net>
From: Max Nikulin <manikulin@gmail.com>
Date: Mon, 22 Nov 2021 23:56:15 +0700
Subject: [PATCH v5] org.el: Warn about unsupported markers in
 `org-set-emphasis-alist'

* lisp/org.el (org-emphasis-alist): Change custom variable type
definition.  Remove unused third list entry from the default value.

(org-emphasis-alist--check-value): New function used to warn the user
when unsupported value is set.  The function is used via
`add-variable-watcher'.

Attempts to introduce new markers have been discussed enough times to
add some code that should prevent wasting of time.

Link: https://orgmode.org/list/878rxoa6lk.fsf@localhost
---
 lisp/org.el | 56 ++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 45 insertions(+), 11 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 1947c63a8..bc879e5d7 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3635,8 +3635,27 @@ (defvar org-verbatim-re nil
 
 (defvar org-emphasis-regexp-components) ; defined just below
 (defvar org-emphasis-alist) ; defined just below
+(defun org-emphasis-alist--check-value (symbol value &optional operation _)
+  "Verify the VALUE of `org-emphasis-alist' SYMBOL.
+Issue a warning when the value is not supported.
+Optional argument OPERATION is intended for `add-variable-watcher'
+usage, which see."
+  (when (memq operation '(set let))
+    (let ((unsupported
+	   (delq nil
+	         (mapcar
+                  (lambda (entry)
+                    (let ((marker (car entry)))
+                      (unless (member marker '("*" "/" "_" "=" "~" "+")) marker)))
+                  value))))
+      (when unsupported
+        (warn "Unsupported markup characters '%s' detected in `%s'"
+	      (mapconcat #'identity unsupported " ")
+	      (symbol-name symbol))))))
 (defun org-set-emph-re (var val)
-  "Set variable and compute the emphasis regular expression."
+  "Set VAR to VAL and compute the emphasis regular expression.
+The function is intended for :set argument of `defcustom' for
+`org-emphasis-alist'."
   (set-default-toplevel-value var val)
   (when (and (boundp 'org-emphasis-alist)
 	     (boundp 'org-emphasis-regexp-components)
@@ -3674,12 +3693,13 @@ (defvar org-emphasis-regexp-components
 
 You need to reload Org or to restart Emacs after setting this.")
 
+(add-variable-watcher 'org-emphasis-alist #'org-emphasis-alist--check-value)
 (defcustom org-emphasis-alist
   '(("*" bold)
     ("/" italic)
     ("_" underline)
-    ("=" org-verbatim verbatim)
-    ("~" org-code verbatim)
+    ("=" org-verbatim)
+    ("~" org-code)
     ("+" (:strike-through t)))
   "Alist of characters and faces to emphasize text.
 Text starting and ending with a special character will be emphasized,
@@ -3687,18 +3707,32 @@ (defcustom org-emphasis-alist
 marker characters and the face to be used by font-lock for highlighting
 in Org buffers.
 
+Do not change the characters and do not add new ones to use custom
+markers for existing styles or to introduce new styles.  Org syntax is
+not meant to be configurable and such modifications will not work with
+export.
+
 You need to reload Org or to restart Emacs after customizing this."
   :group 'org-appearance
   :set 'org-set-emph-re
-  :version "24.4"
-  :package-version '(Org . "8.0")
+  :package-version '(Org . "9.7")
   :type '(repeat
-	  (list
-	   (string :tag "Marker character")
-	   (choice
-	    (face :tag "Font-lock-face")
-	    (plist :tag "Face property list"))
-	   (option (const verbatim)))))
+          (group
+           (choice
+            :tag "Marker"
+            (const :tag "*Bold*" "*")
+            (const :tag "/Italic/" "/")
+            (const :tag "_Underline_" "_")
+            (const :tag "+Strike-through+" "+")
+            (const :tag "=Verbatim=" "=")
+            (const :tag "~Code~" "~")
+            ;; To warn users that it does not work.
+            (string :tag "Unsupported ignored character"))
+           (choice
+            :tag "Font"
+            (face :tag "Face")
+            (plist :tag "Property list"))
+           (option (const :tag "Deprecated ignored constant" verbatim)))))
 
 (defvar org-protecting-blocks '("src" "example" "export")
   "Blocks that contain text that is quoted, i.e. not processed as Org syntax.
-- 
2.39.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Max Nikulin]

On 02/02/2023 17:53, Ihor Radchenko wrote:
> I further improved the patch adding variable watcher. This should catch
> everything. See the attached.

`add-variable-watcher' may be fragile for lists, but it should catch 
most obvious attempts to modify the list. To my surprise 
`add-variable-watcher' is used in just a few places in Emacs sources.

> From: Max Nikulin<manikulin@gmail.com>

Taking into account your modifications, I think you can reset the commit 
author.

> +(defun org-emphasis-alist--check-value (symbol value &optional operation _)
> +  "Verify the VALUE of `org-emphasis-alist' SYMBOL.
> +Issue a warning when the value is not supported.
> +Optional argument OPERATION is intended for `add-variable-watcher'
> +usage, which see."
> +  (when (memq operation '(set let))

I am unsure concerning let. Perhaps the user needs something strange if 
let is used in some code instead of usual setq in init.el, so behavior 
may be too noisy.

> +           (option (const :tag "Deprecated ignored constant" verbatim)))))

Looking into easy customization UI I have found current tag text rather 
confusing. What about "Verbatim (deprecated and ignored)"?

Thank you for further improvements of the patch. I will not object if 
you apply the patch as is.

Are you still going to fix the manual a bit?
Ihor Radchenko to emacs-orgmode… Re: c47b535bb origin/main org-element: 
Remove dependency on ‘org-emphasis-regexp-components’ Thu, 18 Nov 2021 
20:25:33 +0800. https://list.orgmode.org/87tug93b2a.fsf@localhost

Re: [PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist'

[Max Nikulin]

On 02/02/2023 17:53, Ihor Radchenko wrote:
>   (defun org-set-emph-re (var val)
> -  "Set variable and compute the emphasis regular expression."
> +  "Set VAR to VAL and compute the emphasis regular expression.
> +The function is intended for :set argument of `defcustom' for
> +`org-emphasis-alist'."
>     (set-default-toplevel-value var val)
...
> +(add-variable-watcher 'org-emphasis-alist #'org-emphasis-alist--check-value)

Thinking more I have realized that I am in doubts if 
`add-variable-watcher' is appropriate tool in this particular case. The 
only way to get some effect from change of `org-emphasis-alist' is to 
call `org-set-emph-re' (its :set function), so it should be enough to 
call `org-emphasis-alist--check-value' from `org-set-emph-re'.

I have no idea what should be considered as best practice: should 
`set-default-toplevel-value' be combined with changes of `org-emph-re' 
as it is done in current code or `org-set-emph-re' should be split into 
the function that modifies `org-emph-re' (so it can be called 
separately) and a tiny setter (that may be defined as lambda this case) 
that calls the new function and `set-default-toplevel-value'.

Should we obsolete org-emphasis-alist? (was: [PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist')

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> +(add-variable-watcher 'org-emphasis-alist #'org-emphasis-alist--check-value)
>
> Thinking more I have realized that I am in doubts if 
> `add-variable-watcher' is appropriate tool in this particular case. The 
> only way to get some effect from change of `org-emphasis-alist' is to 
> call `org-set-emph-re' (its :set function), so it should be enough to 
> call `org-emphasis-alist--check-value' from `org-set-emph-re'.

Upon looking closer, `org-set-emph-re' is not affected by the value of
`org-emphasis-alist' with a single exception when the value is nil.

> I have no idea what should be considered as best practice: should 
> `set-default-toplevel-value' be combined with changes of `org-emph-re' 
> as it is done in current code or `org-set-emph-re' should be split into 
> the function that modifies `org-emph-re' (so it can be called 
> separately) and a tiny setter (that may be defined as lambda this case) 
> that calls the new function and `set-default-toplevel-value'.

I now reviewed the usage of `org-emphasis-alist' and setting/changing
the value has the following effects:

1. nil (and only nil) value makes `org-emph-re' and `org-verbatim-re'
   nil. Otherwise, these regexps are not affected.

   Also, `org-emph-re' only affects fontification.
   `org-verbarim-re' affects fontification and
   `org-in-verbatim-emphasis'. The latter should be fixed to use
   org-element API.

2. The value affects `org-mode-transpose-word-syntax-table'. Should be
   fixed - parser is not affected by `org-emphasis-alist'.

3. Fontification is affected in `org-do-emphasis-faces'.

4. Emphasis selection dialogue in `org-emphasize' is only using the
   emphasis from the alist.

That's all.

I think that we should simply obsolete `org-emphasis-alist' and
introduce individual faces for emphasis.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Should we obsolete org-emphasis-alist? (was: [PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist')

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

Hi Ihor,

> I think that we should simply obsolete `org-emphasis-alist’ and
> introduce individual faces for emphasis.

I’m in favour of this approach.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: Should we obsolete org-emphasis-alist? (was: [PATCH v5] org.el: Warning for unsupported markers in `org-set-emphasis-alist')

[Max Nikulin]

On 07/02/2023 17:47, Ihor Radchenko wrote:
> 
> Upon looking closer, `org-set-emph-re' is not affected by the value of
> `org-emphasis-alist' with a single exception when the value is nil.

You are right, it is result of the following commit:

9fb2e047d 2016-12-08 09:44:26 +0100 Nicolas Goaziou: Split `org-emph-re' 
and `org-verbatim-re'

> I think that we should simply obsolete `org-emphasis-alist' and
> introduce individual faces for emphasis.

I agree with your conclusions and proposal.

If you are close to merging fontification based on org-element then we 
may just discard this patch. Otherwise it may be merged with minimal 
modifications:
- Do not add the watcher
- Instead of `org-set-emph-re' use a lambda calling new 
`org-emphasis-alist--check-value` as the :set attribute of `defcustom'.
- Fix description of verbatim tag.

Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 21/11/2021 16:28, Ihor Radchenko wrote:
>> 
>> Also, is there any reason why we are not simply using punctuation
>> character class instead of listing punctuation chars explicitly (and
>> only for English)? What about "_你叫什么名字_？"
>
> It seems punctuation character class is too broad. E.g.
>      ¿ INVERTED QUESTION MARK
> normally appears before words, while "?" is usually after them. I do not 
> see anything special in
>      (category-set-mnemonics (char-category-set ?¿))
> that may help to discriminate such cases.

The last resort is define-category where we can manage exceptions.
But I think that even without distinguishing ?¿, we can improve the
situation for CJK users a lot.

We can probably split character categories into "left", "right", and
"neutral" with "(" being "left" example, ")" being "right" example, and
" " being "neutral" example.
We start from using the information we can extract from Unicode data and
modify it as necessary.

Then, emphasis will be defined as PRE MARKER ... MARKER POST with
PRE = left+neutral category
POST = right+neutral category

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Tom Gillespie]

Hi Ihor,
   Thank you for looping me in. Best,
Tom

The way I have implemented this is by maintaining an explicit list of
characters that are safe for pre markup and another for post markup.

It is not possible to use unicode punctuation for this because there
are a variety of punctuation marks that cannot appear in that position
and be considered markup, those include @, #, % to name just a few.

Therefore, if we want to do this we commit to extending and then
maintaining the lists of valid pre and post markup delimiters as
special cases.

Note also this could produce changes from current behavior because
things that previously tokenized as a series of words connected by
e.g. underscores could become markup.

The alternative would be (as usual in these cases) for the user to
add a zero width space or something like that between the end of the
markup marker and the symbol they want to follow the markup. This
solution is (trivially) backward compatible, and works for all chars
regardless of whether org-mode has blessed them as sanctioned marks.

My inclination would be not to make this change because there are a
potentially infinite number of future "left right neutral" marks
that we would have to maintain and would occasionally have to field
requests from users to add them, and those solutions would not work
with older versions of org.

Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Ihor Radchenko]

Tom Gillespie <tgbugs@gmail.com> writes:

> The way I have implemented this is by maintaining an explicit list of
> characters that are safe for pre markup and another for post markup.
>
> It is not possible to use unicode punctuation for this because there
> are a variety of punctuation marks that cannot appear in that position
> and be considered markup, those include @, #, % to name just a few.

Not that bad.
Unicode standard defines the following categories (I listed those that
might be of use):

Pc = Punctuation, connector
Pd = Punctuation, dash
Ps = Punctuation, open
Pe = Punctuation, close
Pi = Punctuation, initial quote (may behave like Ps or Pe depending on usage)
Pf = Punctuation, final quote (may behave like Ps or Pe depending on usage)
Po = Punctuation, other
Zs = Separator, space
Zl = Separator, line
Zp = Separator, paragraph

We currently use the following:
PRE  = <bol> <space> - ( ' " {
POST = <space> - . ; : ! ? ' " ) } \ [

At least, ({ have
(get-char-code-property ?{ 'general-category) ;=> Ps (punctuation, open)

We might probably generalize to
PRE  = Zs Zl Pc Pd Ps Pi ' "
POST = Zs Zl Pc Pd Pe Pf . ; : ! ? ' " \ [

Though we need to take care excluding zero-width spaces.

I can find https://www.unicode.org/review/pr-23.html that defines
punctuation terminals like .;:!?
It looks like it is adopted, via special properties:
https://www.unicode.org/reports/tr44/#STerm and
https://www.unicode.org/reports/tr44/#Terminal_Punctuation

Emacs does not support them though (yet?).

> Therefore, if we want to do this we commit to extending and then
> maintaining the lists of valid pre and post markup delimiters as
> special cases.

We certainly do not want to do this. It is out of scope of Org, when
Unicode can be of use.

> Note also this could produce changes from current behavior because
> things that previously tokenized as a series of words connected by
> e.g. underscores could become markup.

Indeed. And we should study the feedback.
However, most scenarios that will change will involve non-standard
Unicode markup characters. The odds are low that users will use such
Unicode at markup boundary and _also expect markup to be ignored_. At
the end, it is the current ASCII limitation plus partially arbitrary
choice of boundaries that keep some users confused (we are getting bug
reports about confusing markup from time to time).

Of course, we can, as usual, provide a linter to catch such scenarios
and warn in the ORG_NEWS.

I do believe that better Unicode support will benefit many Org users
that use non-Latin scripts. 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Tom Gillespie]

> We might probably generalize to
> PRE  = Zs Zl Pc Pd Ps Pi ' "
> POST = Zs Zl Pc Pd Pe Pf . ; : ! ? ' " \ [

If this works I think it is reasonable. We might want to
specify what to do in cases where an org implementation
might not fully support unicode, and might want to do a
review of related issues in syntax with respect to ascii
vs unicode, because iirc there is some ambiguity in
the current syntax doc.

For example, I'm pretty sure that I'm mixing and matching
unicode and ascii whitespace in the tokenizer I have in Racket.

> Though we need to take care excluding zero-width spaces.

Ya, I removed a comment to this effect in the paragraph about
the usual alternate solution.

> Emacs does not support them though (yet?).

Racket has full support for the latest unicode standards iirc,
so I will see if I can leverage that support for testing in laundry.

> At the end, it is the current ASCII limitation plus partially arbitrary
> choice of boundaries that keep some users confused (we are getting bug
> reports about confusing markup from time to time).

Ya, it would be good to try to generalize the affordance if possible since
users of text in non-ascii languages have certain valid expectations. Hopefully,
the unicode consortium has managed to cover the categories we need.

Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Ihor Radchenko]

Tom Gillespie <tgbugs@gmail.com> writes:

>> We might probably generalize to
>> PRE  = Zs Zl Pc Pd Ps Pi ' "
>> POST = Zs Zl Pc Pd Pe Pf . ; : ! ? ' " \ [
>
> If this works I think it is reasonable. We might want to
> specify what to do in cases where an org implementation
> might not fully support unicode,

Just fall back to ASCII subset? If the implementation does not support
unicode, it probably cannot properly work with UTF-encoded documents
anyway.

> ...and might want to do a
> review of related issues in syntax with respect to ascii
> vs unicode, because iirc there is some ambiguity in
> the current syntax doc.
> For example, I'm pretty sure that I'm mixing and matching
> unicode and ascii whitespace in the tokenizer I have in Racket.

Feel free to open new bug reports about such ambiguities.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>