Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines

Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines

[Alain.Cochard]

Hello.  When I visit an org buffer containing only:

  foo[fn:1]

  [fn:1] bar
  #+begin_example


  x
  #+end_example

then, hitting <TAB> when the cursor is on either '#' does nothing
(i.e., no folding).

Tested with 'emacs -Q' and the following combinations:

GNU Emacs 27.2 (build 1, x86_64-redhat-linux-gnu, GTK+ Version
3.24.30, cairo version 1.17.4) of 2021-08-07 Org mode version 9.4.4
(release_9.4.4 @ /usr/share/emacs/27.2/lisp/org/)

GNU Emacs 28.1 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,
cairo version 1.17.4) of 2022-08-16 Org mode version 9.5.2
(release_9.5.2-25-gaf6f12 @
/home/cochard/usr/local/emacs-28.1/lisp/org/)

GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,
cairo version 1.17.4) of 2022-08-22 Org mode version 9.5.4
(release_9.5.4-17-g6e991f @
/home/cochard/usr/local/emacs-29/lisp/org/)

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines

[Ihor Radchenko]

Alain.Cochard@unistra.fr writes:

> Hello.  When I visit an org buffer containing only:
>
>   foo[fn:1]
>
>   [fn:1] bar
>   #+begin_example
>
>
>   x
>   #+end_example
>
> then, hitting <TAB> when the cursor is on either '#' does nothing
> (i.e., no folding).

This is not a bug. Just unintuitive syntax:

https://orgmode.org/manual/Creating-Footnotes.html#Creating-Footnotes
>> A footnote is started by a footnote marker in square brackets in
>> column 0, no indentation allowed. It ends at the next footnote
>> definition, headline, or after two consecutive empty lines. The
>> footnote reference is simply the marker in square brackets, inside
>> text. Markers always start with ‘fn:’.

The important part is:
It ends at the next footnote definition, headline, or after two consecutive empty lines.

So, your example is

foo[fn:1]

<begin footnote-definition>[fn:1] bar
#+begin_example ;; <- this is not considered an example block because no
major org elements can be inside a footnote definition.


<end footnote-definition after two empty lines>
<begin paragraph>x
#+end_example
<end paragraph>

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines

[Alain.Cochard]

Ihor Radchenko writes on Wed 31 Aug 2022 09:57:

 > This is not a bug. Just unintuitive syntax:
 > [...]
 > The important part is: It ends at the next footnote definition,
 > headline, or after two consecutive empty lines.
 > 
 > So, your example is
 > 
 > foo[fn:1]
 > 
 > <begin footnote-definition>[fn:1] bar
 > #+begin_example ;; <- this is not considered an example block because no
 > major org elements can be inside a footnote definition.
 > 
 > 
 > <end footnote-definition after two empty lines>
 > <begin paragraph>x
 > #+end_example
 > <end paragraph>

Thanks a lot for answering.

Is there an official definition for "major org element" ? (I could not
find anything relevant in the manual.)

I guess a headline would qualify (and it would not have occurred to me
to use a headline within a footnote; the manual is clear in that
respect anyway).

But then a list item would probably qualify as well?  But, as far as I
can see, lists and footnotes work perfectly well together... (because
2 consecutive empty lines also mark the end of a list item?)

Furthermore, as long as there is not more than one empty line in a
row, blocks also appear to me to work just fine within footnotes,
including upon latex export, and I use things like the following all
the time:

   #+LATEX_HEADER: \usepackage{fancyvrb}
   #+LATEX_HEADER: \VerbatimFootnotes
   foot[fn:1]

   [fn:1] note 
   #+begin_src fortran
	 print*,"foo"
   #+end_src

I simply never had to use 2 consecutive empty lines in blocks up to
now...

So I guess I'll turn my "suspected bug" into a feature request: that
#+BEGIN ... #+END blocks behave the same way whether they are within a
footnote or not.

Regards

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Syntax of footnote-definition: can we allow double empty lines inside? (was: Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines)

[Ihor Radchenko]

Alain.Cochard@unistra.fr writes:

> Ihor Radchenko writes on Wed 31 Aug 2022 09:57:
>
>  > This is not a bug. Just unintuitive syntax:
>  > [...]
>  > The important part is: It ends at the next footnote definition,
>  > headline, or after two consecutive empty lines.
>  > 
>  > So, your example is
>  > 
>  > foo[fn:1]
>  > 
>  > <begin footnote-definition>[fn:1] bar
>  > #+begin_example ;; <- this is not considered an example block because no
>  > major org elements can be inside a footnote definition.
>  > 
>  > 
>  > <end footnote-definition after two empty lines>
>  > <begin paragraph>x
>  > #+end_example
>  > <end paragraph>
>
> Thanks a lot for answering.
>
> Is there an official definition for "major org element" ? (I could not
> find anything relevant in the manual.)

Sorry, I actually confused things in attempt to make things more clear.
The official Org syntax defines 2 types of syntax structures:
elements and objects. Elements can be: (1) headings, (2) sections, (3)
greater elements, (4) lesser elements.

Each Org document is a sequence of elements with no gaps in-between.
Objects (e.g. markup) may be separated by plain text and must always be
inside an existing element.

Headings, sections, and greater elements can contain other elements
inside, while lesser elements cannot.

By "major element" I was actually referring to lesser element
(non-object). Not only I was wrong (footnote-definition is a greater
element), but also used a confusing terminology.

You can refer to https://orgmode.org/worg/dev/org-syntax.html for more
detailed info.

> I guess a headline would qualify (and it would not have occurred to me
> to use a headline within a footnote; the manual is clear in that
> respect anyway).
>
> But then a list item would probably qualify as well?  But, as far as I
> can see, lists and footnotes work perfectly well together... (because
> 2 consecutive empty lines also mark the end of a list item?)
>
> Furthermore, as long as there is not more than one empty line in a
> row, blocks also appear to me to work just fine within footnotes,
> including upon latex export, and I use things like the following all
> the time:
>
>    #+LATEX_HEADER: \usepackage{fancyvrb}
>    #+LATEX_HEADER: \VerbatimFootnotes
>    foot[fn:1]
>
>    [fn:1] note 
>    #+begin_src fortran
> 	 print*,"foo"
>    #+end_src
>
> I simply never had to use 2 consecutive empty lines in blocks up to
> now...
>
> So I guess I'll turn my "suspected bug" into a feature request: that
> #+BEGIN ... #+END blocks behave the same way whether they are within a
> footnote or not.

You are right, there is currently no way to have double blank lines
_inside_ footnote definition. This is because Org syntax prohibits
elements inside a greater element to influence the parent boundaries.
Once Org parser encounters ^[fn:...] right after previously parsed
element, it immediately looks forward for (1) next footnote definition
^[fn:...]; (2) headline ^\*+ ; (3) double blank lines; and sets the
footnote definition boundaries accordingly. Unconditionally. No
backtracking is allowed for the performance reasons.

The consequence is inability to have source blocks containing double
blank lines inside a footnote definition.

This is not an easy problem because we need to ensure backwards
compatibility and avoid doing major changes to Org syntax. The best I
can think of is allowing to escape blank lines like #+ and * lines
inside the code blocks with ",". See 12.6 Literal Examples section of
the manual, for example:

#+BEGIN_EXAMPLE
,* I am no real headline
#+END_EXAMPLE

We may add the same for double blank lines

#+BEGIN_EXAMPLE

,* I am no real headline

,
#+END_EXAMPLE

The downside is overly verbose ordinary code blocks. Double blank lines
may not be uncommon in the wild; sprinkling commas around many source
blocks will be ugly.

Alternative ideas are welcome.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92