Syntax of footnote-definition: can we allow double empty lines inside? (was: Suspected bug: example block in a footnote doesn't fold if starts with 2 blank lines)

[Ihor Radchenko <yantar92@gmail.com>]

Alain.Cochard@unistra.fr writes:

> Ihor Radchenko writes on Wed 31 Aug 2022 09:57:
>
>  > This is not a bug. Just unintuitive syntax:
>  > [...]
>  > The important part is: It ends at the next footnote definition,
>  > headline, or after two consecutive empty lines.
>  > 
>  > So, your example is
>  > 
>  > foo[fn:1]
>  > 
>  > <begin footnote-definition>[fn:1] bar
>  > #+begin_example ;; <- this is not considered an example block because no
>  > major org elements can be inside a footnote definition.
>  > 
>  > 
>  > <end footnote-definition after two empty lines>
>  > <begin paragraph>x
>  > #+end_example
>  > <end paragraph>
>
> Thanks a lot for answering.
>
> Is there an official definition for "major org element" ? (I could not
> find anything relevant in the manual.)

Sorry, I actually confused things in attempt to make things more clear.
The official Org syntax defines 2 types of syntax structures:
elements and objects. Elements can be: (1) headings, (2) sections, (3)
greater elements, (4) lesser elements.

Each Org document is a sequence of elements with no gaps in-between.
Objects (e.g. markup) may be separated by plain text and must always be
inside an existing element.

Headings, sections, and greater elements can contain other elements
inside, while lesser elements cannot.

By "major element" I was actually referring to lesser element
(non-object). Not only I was wrong (footnote-definition is a greater
element), but also used a confusing terminology.

You can refer to https://orgmode.org/worg/dev/org-syntax.html for more
detailed info.

> I guess a headline would qualify (and it would not have occurred to me
> to use a headline within a footnote; the manual is clear in that
> respect anyway).
>
> But then a list item would probably qualify as well?  But, as far as I
> can see, lists and footnotes work perfectly well together... (because
> 2 consecutive empty lines also mark the end of a list item?)
>
> Furthermore, as long as there is not more than one empty line in a
> row, blocks also appear to me to work just fine within footnotes,
> including upon latex export, and I use things like the following all
> the time:
>
>    #+LATEX_HEADER: \usepackage{fancyvrb}
>    #+LATEX_HEADER: \VerbatimFootnotes
>    foot[fn:1]
>
>    [fn:1] note 
>    #+begin_src fortran
> 	 print*,"foo"
>    #+end_src
>
> I simply never had to use 2 consecutive empty lines in blocks up to
> now...
>
> So I guess I'll turn my "suspected bug" into a feature request: that
> #+BEGIN ... #+END blocks behave the same way whether they are within a
> footnote or not.

You are right, there is currently no way to have double blank lines
_inside_ footnote definition. This is because Org syntax prohibits
elements inside a greater element to influence the parent boundaries.
Once Org parser encounters ^[fn:...] right after previously parsed
element, it immediately looks forward for (1) next footnote definition
^[fn:...]; (2) headline ^\*+ ; (3) double blank lines; and sets the
footnote definition boundaries accordingly. Unconditionally. No
backtracking is allowed for the performance reasons.

The consequence is inability to have source blocks containing double
blank lines inside a footnote definition.

This is not an easy problem because we need to ensure backwards
compatibility and avoid doing major changes to Org syntax. The best I
can think of is allowing to escape blank lines like #+ and * lines
inside the code blocks with ",". See 12.6 Literal Examples section of
the manual, for example:

#+BEGIN_EXAMPLE
,* I am no real headline
#+END_EXAMPLE

We may add the same for double blank lines

#+BEGIN_EXAMPLE

,* I am no real headline

,
#+END_EXAMPLE

The downside is overly verbose ordinary code blocks. Double blank lines
may not be uncommon in the wild; sprinkling commas around many source
blocks will be ugly.

Alternative ideas are welcome.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92