[BUG] Emphasis markers should be considered word constituents (was: [BUG] SPACE jumps to tag in header line after hidden emphasis marker [9.7-pre (release_9.6.7-562-g5b6268 @ /home/jschmidt/work/org-mode/lisp/)])

[Ihor Radchenko <yantar92@posteo.net>]

> Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:
> ...
>> Well, I have one not-so-minor nit here: With that commit you cannot 
>> insert text *before* some text having hidden emphasis without breaking 
>> the emphasis.  More concretely (in an empty org-mode buffer):
>>

    (setq org-hide-emphasis-markers t)

    *foo*<point>
    
>> ...
>>   C-<left>    ;; left-word
>>   SPC	     ;; org-self-insert-command
>>
>> The last SPC breaks the emphasis around the middle "test", which it did 
>> not previously.  Previously, the SPC would just move the whole 
>> emphasized word one space to the right.  Which I somehow got used to, 
>> and probably many others as well.

The root cause revealed by my change is that Org mode does not always
consider emphasis markers to be word boundaries.

Note how when you do not hide emphasis markers, C-<left> will end up in

     *<point>foo*

When emphasis markers are not hidden, things were the same, except that
Emacs adjusted point right to sticky invisible text in the direction of
point movement, leading to inconsistent behaviour between hiding and not
hiding emphasis markers.

Yet, Org sometimes does treat emphasis markers as word constituents:
At least, `org-fill-element' and `org-transpose-words' temporarily turn
emphasis markers into parts of words.

The situation is clearly a bit chaotic.

One of the reasons why we have such a mess is because *_/~+ are not
always representing emphasis. Proper distinction between *_/~+ that are
and aren't part of actual emphasis would require a custom
`syntax-propertize-function' that would query Org element API to
determine the context.

Unfortunately, using `syntax-propertize-function' it is not very
practical yet.

We need parser context down to object level, while we only cache parser
state down to element (paragraph) level. So, querying element API in
this case will lead to significant performance degradation, with
quadratic (N_objects_in_paragraph^2) complexity in some cases.

I will leave this bug report open until we have a better caching and can
reasonably implement custom `syntax-propertize-function'.

Or, if any, please post arguments against making emphasis markers word
constituents.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>