Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

[Tom Gillespie <tgbugs@gmail.com>]

Hi Ihor,
   Thank you for looping me in. Best,
Tom

The way I have implemented this is by maintaining an explicit list of
characters that are safe for pre markup and another for post markup.

It is not possible to use unicode punctuation for this because there
are a variety of punctuation marks that cannot appear in that position
and be considered markup, those include @, #, % to name just a few.

Therefore, if we want to do this we commit to extending and then
maintaining the lists of valid pre and post markup delimiters as
special cases.

Note also this could produce changes from current behavior because
things that previously tokenized as a series of words connected by
e.g. underscores could become markup.

The alternative would be (as usual in these cases) for the user to
add a zero width space or something like that between the end of the
markup marker and the symbol they want to follow the markup. This
solution is (trivially) backward compatible, and works for all chars
regardless of whether org-mode has blessed them as sanctioned marks.

My inclination would be not to make this change because there are a
potentially infinite number of future "left right neutral" marks
that we would have to maintain and would occasionally have to field
requests from users to add them, and those solutions would not work
with older versions of org.