[BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[Lucas Viana]

[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]

Dear list,

I'm not sure if this is a general issue with fill-paragraph or with the way
org hides emphasis markers, but I'm facing a problem with bare emacs + org.

Steps to reproduce:
1. Start emacs -Q (no configuration)
2. Set some variables:

(setq org-pretty-entities t
      org-hide-emphasis-markers t
      fill-column 80)

3. Create an empty org-mode buffer with the following paragraph:
""
This is a /italic sentence for a buggy/ \alpha\beta\gamma\delta\nu\epsilon
paragraph in terms of filling, see, it's not filled yet and still the line
above
ends early.
""

So far so good, and with the previous config the emphasis markers are now
hidden and the greek letters appear as unicode characters.

4. Hit M-q (either fill-paragraph or org-fill-paragraph, both have the same
issue)
The first line wrongly wraps after "buggy" and leaves a big empty space
after that, as if the greek letters were not composed.

5. In the same buffer, compare the behaviour with this paragraph (very
similar, but note the ! character after "buggy"):
""
This is a /italic sentence for a buggy/ ! \alpha\beta\gamma\delta\nu\epsilon
paragraph in terms of filling, see, it's not filled yet and still the line
above
ends early.
""

When I fill this slightly different paragraph with M-q, it wraps the text
in the right way and the first line is completely filled.

So the issue happens when a hidden emphasis marker is immediately followed
(maybe after a space) by a composed character like \alpha. I am also facing
the same issue when the characters are composed by prettify-symbols-mode
instead of org-pretty-entities, and also with other types of hidden
emphasis markers. If org-hide-emphasis-markers is disabled then the issue
doesn't happen.

Thanks in advance!

Emacs  : GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.29, cairo version 1.17.4)
 of 2021-07-09
Package: Org mode version 9.5 (9.5-b83ae59 @
/home/lucas/.emacs.d/.local/straight/build-28.0.50/org/)

[-- Attachment #2: Type: text/html, Size: 2255 bytes --]

Re: [BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[Tim Cross]

I was able to reproduce this using Emacs 28.0.50 and emacs -Q.

It is important to ensure both of the below paragraphs are completely
'unwrapped' i.e. all on one line and then wrap them with M-q. 

The result seems counter-intuitive to me - the second paragraph with an
extra character is wrapped correctly. The first paragraph has the first
line broken too early.

This is likely going to be a very difficult issue to fix. I'm not sure
how fill column mode is aware of add on features like prettify symbols
and hidden emphasis markers. Obviously it is aware at some level because
it does wrap the second paragraph correctly. My wild guess would be that
the second one works because the additional character breaks up the
'special' handling and allows fill column to correctly calculate the
'virtual' line length.

Lucas Viana <lucasvianareis@gmail.com> writes:

> Dear list, 
>
> I'm not sure if this is a general issue with fill-paragraph or with the way org hides emphasis markers, but I'm facing a problem with bare emacs +
> org.
>
> Steps to reproduce:
> 1. Start emacs -Q (no configuration)
> 2. Set some variables:
>
> (setq org-pretty-entities t
>       org-hide-emphasis-markers t
>       fill-column 80)
>
> 3. Create an empty org-mode buffer with the following paragraph:
> ""
> This is a /italic sentence for a buggy/ \alpha\beta\gamma\delta\nu\epsilon
> paragraph in terms of filling, see, it's not filled yet and still the line above
> ends early.
> ""
>
> So far so good, and with the previous config the emphasis markers are now hidden and the greek letters appear as unicode characters.
>
> 4. Hit M-q (either fill-paragraph or org-fill-paragraph, both have the same issue)
> The first line wrongly wraps after "buggy" and leaves a big empty space after that, as if the greek letters were not composed.
>
> 5. In the same buffer, compare the behaviour with this paragraph (very similar, but note the ! character after "buggy"):
> ""
> This is a /italic sentence for a buggy/ ! \alpha\beta\gamma\delta\nu\epsilon
> paragraph in terms of filling, see, it's not filled yet and still the line above
> ends early.
> ""
>
> When I fill this slightly different paragraph with M-q, it wraps the text in the right way and the first line is completely filled.
>
> So the issue happens when a hidden emphasis marker is immediately followed (maybe after a space) by a composed character like \alpha. I am also
> facing the same issue when the characters are composed by prettify-symbols-mode instead of org-pretty-entities, and also with other types of
> hidden emphasis markers. If org-hide-emphasis-markers is disabled then the issue doesn't happen.
>
> Thanks in advance!
>
> Emacs  : GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.17.4)
>  of 2021-07-09
> Package: Org mode version 9.5 (9.5-b83ae59 @ /home/lucas/.emacs.d/.local/straight/build-28.0.50/org/)

Re: [BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[Rudolf Adamkovič]

On a related note, I recently disabled 
“org-hide-emphasis-markers,” only to discover that the columns 
in all my tables that use =mono= text do not line up. Ouch!

R+

Lucas Viana <lucasvianareis@gmail.com> writes:

> Dear list, 
> 
> I'm not sure if this is a general issue with fill-paragraph or 
> with the way org hides emphasis markers, but I'm facing a 
> problem with bare emacs + org. 
> 
> Steps to reproduce: 1. Start emacs -Q (no configuration) 2. Set 
> some variables: 
> 
> (setq org-pretty-entities t 
>       org-hide-emphasis-markers t fill-column 80) 
> 
> 3. Create an empty org-mode buffer with the following paragraph: 
> "" This is a /italic sentence for a buggy/ 
> \alpha\beta\gamma\delta\nu\epsilon paragraph in terms of 
> filling, see, it's not filled yet and still the line above ends 
> early.  "" 
> 
> So far so good, and with the previous config the emphasis 
> markers are now hidden and the greek letters appear as unicode 
> characters. 
> 
> 4. Hit M-q (either fill-paragraph or org-fill-paragraph, both 
> have the same issue) The first line wrongly wraps after "buggy" 
> and leaves a big empty space after that, as if the greek letters 
> were not composed. 
> 
> 5. In the same buffer, compare the behaviour with this paragraph 
> (very similar, but note the ! character after "buggy"): "" This 
> is a /italic sentence for a buggy/ ! 
> \alpha\beta\gamma\delta\nu\epsilon paragraph in terms of 
> filling, see, it's not filled yet and still the line above ends 
> early.  "" 
> 
> When I fill this slightly different paragraph with M-q, it wraps 
> the text in the right way and the first line is completely 
> filled. 
> 
> So the issue happens when a hidden emphasis marker is 
> immediately followed (maybe after a space) by a composed 
> character like \alpha. I am also facing the same issue when the 
> characters are composed by prettify-symbols-mode instead of 
> org-pretty-entities, and also with other types of hidden 
> emphasis markers. If org-hide-emphasis-markers is disabled then 
> the issue doesn't happen. 
> 
> Thanks in advance! 
> 
> Emacs  : GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ 
> Version 3.24.29, cairo version 1.17.4) 
>  of 2021-07-09 
> Package: Org mode version 9.5 (9.5-b83ae59 @ 
> /home/lucas/.emacs.d/.local/straight/build-28.0.50/org/) 

-- 
"Contrariwise," continued Tweedledee, "if it was so, it might be; 
and if it were so, it would be; but as it isn't, it ain't. That's 
logic." -- Lewis Carroll, Through the Looking Glass  Rudolf 
Adamkovič <salutis@me.com> Studenohorská 25 84103 Bratislava 
Slovakia  [he/him]

Re: [BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[Ihor Radchenko]

Lucas Viana <lucasvianareis@gmail.com> writes:

> Steps to reproduce:
> 1. Start emacs -Q (no configuration)
> 2. Set some variables:
>
> (setq org-pretty-entities t
>       org-hide-emphasis-markers t
>       fill-column 80)
>
> 3. Create an empty org-mode buffer with the following paragraph:
> ""
> This is a /italic sentence for a buggy/ \alpha\beta\gamma\delta\nu\epsilon
> paragraph in terms of filling, see, it's not filled yet and still the line
> above
> ends early.
> ""
>
> So far so good, and with the previous config the emphasis markers are now
> hidden and the greek letters appear as unicode characters.
>
> 4. Hit M-q (either fill-paragraph or org-fill-paragraph, both have the same
> issue)
> The first line wrongly wraps after "buggy" and leaves a big empty space
> after that, as if the greek letters were not composed.

Confirmed.
This appears to be Emacs bug though.
I reported it in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58791
Let's see what Emacs devs reply.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Fill-paragraph issue with hidden emphasis markers and composed characters

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> 4. Hit M-q (either fill-paragraph or org-fill-paragraph, both have the same
>> issue)
>> The first line wrongly wraps after "buggy" and leaves a big empty space
>> after that, as if the greek letters were not composed.
>
> Confirmed.
> This appears to be Emacs bug though.
> I reported it in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58791
> Let's see what Emacs devs reply.

The bug has been fixed on Emacs side (on master).
As a result of the fix, filling works as expected in Org as well.

So, this issue will disappear when using Org with next Emacs 29 release.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>