[BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)]

[BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)]

[Matt Beshara]

‘org-fill-paragraph’ does not correctly handle ^L characters 
(a.k.a. form feed, C-q C-l).  It should treat them as paragraph 
separating whitespace, but instead treats them as any other 
character which would appear in normal text.  Here is an example 
to demonstrate the current behaviour:

abc def
^L
ghi jkl

In org-mode, with point at the beginning or the end of the first 
or last line, doing ‘org-fill-paragraph’ (M-q) should do nothing, 
because the lines are already shorter than the fill-column.  What 
really happens is that I end up with one line which looks like:

abc def ^L def ghi

In text-mode, ‘fill-paragraph’ does handle ^L characters 
correctly, and pressing M-q anywhere in the first example results 
in no change being made to the buffer.

Emacs  : GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, X 
toolkit, cairo version 1.17.6)
 of 2022-08-27
Package: Org mode version 9.5.4 (release_9.5.4-19-g4dff42 @ 
/home/matt/Code/emacs/lisp/org/)

Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)])

[Ihor Radchenko]

Matt Beshara <m@mfa.pw> writes:

> ‘org-fill-paragraph’ does not correctly handle ^L characters 
> (a.k.a. form feed, C-q C-l).  It should treat them as paragraph 
> separating whitespace, but instead treats them as any other 
> character which would appear in normal text.  Here is an example 
> to demonstrate the current behaviour:
>
> abc def
> ^L
> ghi jkl

This may or may not be a correct expectation.

Please remember that Org mode is a markup language.

^L is not a part of Org specification for paragraph separators.
According to https://orgmode.org/worg/dev/org-syntax.html#Paragraphs,
paragraph boundaries should either be empty lines, or beginning of other
Org elements.

> In org-mode, with point at the beginning or the end of the first 
> or last line, doing ‘org-fill-paragraph’ (M-q) should do nothing, 
> because the lines are already shorter than the fill-column.  What 
> really happens is that I end up with one line which looks like:
>
> abc def ^L def ghi

Which is correct wrt current Org syntax specification, although I do
agree that it is counter-intuitive.

Dear all,
Should we add page break (^L) to our syntax? It sounds like a reasonable
addition for a text-based markup.

> In text-mode, ‘fill-paragraph’ does handle ^L characters 
> correctly, and pressing M-q anywhere in the first example results 
> in no change being made to the buffer.

Indeed. That's because text mode syntax is treated ^L specially.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)])

[Matt Beshara]

> ^L is not a part of Org specification for paragraph separators.
> According to 
> https://orgmode.org/worg/dev/org-syntax.html#Paragraphs,
> paragraph boundaries should either be empty lines, or beginning 
> of other
> Org elements.

Fair enough.  If the consensus from others is that ^L should be 
recognised as a paragraph separator and the code is eventually 
written to make that work, that would be nice, but as it stands I 
can just start adding newlines after ^L.

Thanks for your work on Org mode!
Matt

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)])

[Max Nikulin]

On 26/09/2022 18:39, Ihor Radchenko wrote:
> 
> Should we add page break (^L) to our syntax? It sounds like a reasonable
> addition for a text-based markup.

I do not have strong opinion. It is a control character while I prefer 
explicit markup and printable characters or commands in such case. On 
the other hand the character is widely used in .el files.

Users may expect \newline when ^L is exported to LaTeX.

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)])

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 26/09/2022 18:39, Ihor Radchenko wrote:
>> Should we add page break (^L) to our syntax? It sounds like a reasonable
>> addition for a text-based markup.
>
> I do not have strong opinion. It is a control character while I prefer explicit markup and
> printable characters or commands in such case. On the other hand the character is widely
> used in .el files.
>
> Users may expect \newline when ^L is exported to LaTeX.

I think I'm with you as well. I don't have a strong opinion and while
personally, I don't like control characters used as part of the syntax,
^L is often used in elisp files, so ......

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly [9.5.4 (release_9.5.4-19-g4dff42 @ /home/matt/Code/emacs/lisp/org/)])

[Robert Klein]

On Tue, 27 Sep 2022 01:10:49 +1000
Matt Beshara <m@mfa.pw> wrote:

> > ^L is not a part of Org specification for paragraph separators.
> > According to 
> > https://orgmode.org/worg/dev/org-syntax.html#Paragraphs,
> > paragraph boundaries should either be empty lines, or beginning 
> > of other
> > Org elements.  
> 
> Fair enough.  If the consensus from others is that ^L should be 
> recognised as a paragraph separator and the code is eventually 
> written to make that work, that would be nice, but as it stands I 
> can just start adding newlines after ^L.
> 
> Thanks for your work on Org mode!
> Matt
> 

Actually ^L is a *page* separator (to be exact “FORM FEED”).

Best regards
Robert

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly)

[Max Nikulin]

On 27/09/2022 03:13, Tim Cross wrote:
> Max Nikulin writes:
>> On 26/09/2022 18:39, Ihor Radchenko wrote:
>>> Should we add page break (^L) to our syntax? It sounds like a reasonable
>>> addition for a text-based markup.
>>
>> I do not have strong opinion. It is a control character while I prefer explicit markup and
>> printable characters or commands in such case. On the other hand the character is widely
>> used in .el files.
>>
>> Users may expect \newline when ^L is exported to LaTeX.
> 
> I think I'm with you as well. I don't have a strong opinion and while
> personally, I don't like control characters used as part of the syntax,
> ^L is often used in elisp files, so ......

Matt, could you, please, provide some details concerning your use case 
for ^L in Org files? Even if nothing will change this time, it may 
affect later decisions.

On 27/09/2022 13:21, Robert Klein wrote:
 > On Tue, 27 Sep 2022 01:10:49 +1000
 > Matt Beshara wrote:

 >> Fair enough.  If the consensus from others is that ^L should be
 >> recognised as a paragraph separator and the code is eventually
 >> written to make that work, that would be nice, but as it stands I
 >> can just start adding newlines after ^L.
 >
 > Actually ^L is a *page* separator (to be exact “FORM FEED”).

I have realized that page separator for plain text is not necessary 
paragraph separator. The same paragraph may continue on the next page. 
It makes things more tricky. ^L should be kept on its own line while 
text around should be wrapped.

However from a discussion happened a year ago I concluded that it may be 
not so trivial to tune `org-fill-element' for special cases.

Timothy. Re: [PATCH] Don't fill displayed equations. Sun, 03 Oct 2021 
16:50:54 +0800. https://list.orgmode.org/875yueij6r.fsf@gmail.com

Re: Should page break (^L) work as paragraph element separator and be included into Org syntax? (was: [BUG] org-fill-paragraph doesn't handle ^L correctly)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>  > Actually ^L is a *page* separator (to be exact “FORM FEED”).
>
> I have realized that page separator for plain text is not necessary 
> paragraph separator. The same paragraph may continue on the next page. 
> It makes things more tricky. ^L should be kept on its own line while 
> text around should be wrapped.

Good point.

> However from a discussion happened a year ago I concluded that it may be 
> not so trivial to tune `org-fill-element' for special cases.
>
> Timothy. Re: [PATCH] Don't fill displayed equations. Sun, 03 Oct 2021 
> 16:50:54 +0800. https://list.orgmode.org/875yueij6r.fsf@gmail.com

Actually, it is not that much hard. At least, it is not that hard for
paragraphs.

The linked discussion had other caveats involving whether
special filling is even reasonable for the discussed elements
(AFAIR).

However, org-fill-element actually does have a special treatment for
line-break objects inside paragraphs. There is nothing preventing us
from special treatment of other types of objects in similar fashion.

In fact, we may even utilize Emacs built-ins for page break symbols
specifically. Just use fill-region instead of fill-region-as-paragraph.
Similar to https://orgmode.org/list/875yhiyxnb.fsf@localhost
It's just that I am not sure about other possible side effects of such
change.

P.S. Does LaTeX understand ^L?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92