[possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Hi,

I recently discovered that if the last line of a LaTeX 'verse'
environment ends in '\\[0pt]', the space after the environment is
drastically altered. The space after a 'verse' environment (which is a
modified list environment) must be equivalent to
\partopsep+\topsep+\baselineskip. I have done the test with a document,
defining a rectangle of that height with tikz, where you can see the
difference in vertical space with and without '\\[0pt]':

https://i.imgur.com/khbboDZ.png

Naturally, this implies an issue when exporting the verse block to
LaTeX, since the presence of the '\\[0pt]' string alters the expected
result in the compiled PDF. My suggestion is that the last line of the
verse block is not exported with either the string '[0pt]' or the line
break mark '\\', since it makes no sense for that mark to be there: it
is the end of a paragraph and the very end of the environment.

Can anyone think of a possible scenario where removing the '\\[0pt]' in
the last line would cause a problem? If not, I might send a patch in the
next few days. (I'm afraid that going back to abusing
replace-regexp-in-string. I can't think of any other more elegant
solution...).

Best regards,

Juan Manuel 


-- 
Juan Manuel Macías -- Composición tipográfica, tratamiento de datos, diseño editorial y ortotipografía

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Can anyone think of a possible scenario where removing the '\\[0pt]' in
> the last line would cause a problem? If not, I might send a patch in the
> next few days. (I'm afraid that going back to abusing
> replace-regexp-in-string. I can't think of any other more elegant
> solution...).

In such scenario, we may technically violate what users ask us to do:

#+begin_verse
I really want newline here\\
#+end_verse

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

>> Can anyone think of a possible scenario where removing the '\\[0pt]' in
>> the last line would cause a problem? If not, I might send a patch in the
>> next few days. (I'm afraid that going back to abusing
>> replace-regexp-in-string. I can't think of any other more elegant
>> solution...).
>
> In such scenario, we may technically violate what users ask us to do:
>
> #+begin_verse
> I really want newline here\\
> #+end_verse

I'm not sure if users asked for that specifically.

You want a new line when there is a new line.

This makes sense:

 #+begin_verse
 I really want newline here\\
 blah...
 #+end_verse

This does not:

 #+begin_verse
 I really want newline here\\
 #+end_verse

For LaTeX it is the same as this:

 #+begin_verse
 I really want newline here
 #+end_verse

But this specifically alters the expected result in LaTeX, modifying the
space after the environment:

 #+begin_verse
 I really want newline here\\[0pt]
 #+end_verse

What is the cause? I don't know. But it happens.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> In such scenario, we may technically violate what users ask us to do:
>>
>> #+begin_verse
>> I really want newline here\\
>> #+end_verse
>
> I'm not sure if users asked for that specifically.
> ...
> This does not:
>
>  #+begin_verse
>  I really want newline here\\
>  #+end_verse

I am not sure.
What about

#+begin_verse
I really want newline here\\
#+end_verse
This must not be indented.

(Just trying to come up with valid uses of newline at the end of
LaTeX environment)

Although, the above causes Org export to fail...
\begin{verse}
This is test\\[0pt]\\[0pt]
\end{verse}
Paragraph.


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Juan Manuel Macías <maciaschain@posteo.net> writes:
>
>>> In such scenario, we may technically violate what users ask us to do:
>>>
>>> #+begin_verse
>>> I really want newline here\\
>>> #+end_verse
>>
>> I'm not sure if users asked for that specifically.
>> ...
>> This does not:
>>
>>  #+begin_verse
>>  I really want newline here\\
>>  #+end_verse
>
> I am not sure.
> What about
>
> #+begin_verse
> I really want newline here\\
> #+end_verse
>
> This must not be indented.

If I'm not mistaken, in TeX '\\' can only be used in horizontal mode:
that is, within a paragraph. At the end of an environment like verse
(\end{verse}) you are forced to enter vertical mode, and a new paragraph
starts. Doing something like this, for example, adds a new paragraph
after the environment:

\documentclass{article}
\usepackage{lipsum}
\begin{document}
\begin{verse}
  Lorem ipsum dolor\\
\end{verse}
\lipsum[1]
\end{document}

Even this one produces the same result:

\documentclass{article}
\usepackage{lipsum}
\begin{document}
\begin{verse}
  Lorem ipsum dolor\\
\end{verse}\lipsum[1]
\end{document}

If one wants, for example, that after a verse environment there is no
vertical space or indentation, the only way to do it is by modifying, globally or
locally (with another environment) the space after the environment and
adding a \noindent to the following paragraph:

\documentclass{article}
\usepackage{lipsum}
\setlength\partopsep{-\topsep}\addtolength\partopsep{0pt}
\begin{document}
\begin{verse}
  Lorem ipsum dolor\\
\end{verse}
\noindent\lipsum[1]
\end{document}

(taking into account that within the verse environment there is also a
left indentation that would also have to be modified).

Therefore, I think that whether or not a verse environment ends with \\ is
irrelevant to LaTeX. The problem is that when it ends with \\[0pt], for
some reason that escapes me, the space after the environment is altered.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> If I'm not mistaken, in TeX '\\' can only be used in horizontal mode:
> that is, within a paragraph. At the end of an environment like verse
> (\end{verse}) you are forced to enter vertical mode, and a new paragraph
> starts.

Is it true for any environment? Or just some?

> Therefore, I think that whether or not a verse environment ends with \\ is
> irrelevant to LaTeX. The problem is that when it ends with \\[0pt], for
> some reason that escapes me, the space after the environment is altered.

I see. Although, I am not 100% if it is something we have to deal with
in Org mode. May it be better to report this as a bug to LaTeX devs?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Juan Manuel Macías <maciaschain@posteo.net> writes:
>
>> If I'm not mistaken, in TeX '\\' can only be used in horizontal mode:
>> that is, within a paragraph. At the end of an environment like verse
>> (\end{verse}) you are forced to enter vertical mode, and a new paragraph
>> starts.
>
> Is it true for any environment? Or just some?

In principle, nothing prevents one from defining an environment for use
in horizontal mode (within the paragraph). E.g.:

\newenvironment{foo}{\itshape}{}

Lorem \begin{foo}ipsum\end{foo} dolor.

But the usual thing is that the beginning/end of an environment changes to
vertical mode, with \par or something more elaborate.


>> Therefore, I think that whether or not a verse environment ends with \\ is
>> irrelevant to LaTeX. The problem is that when it ends with \\[0pt], for
>> some reason that escapes me, the space after the environment is altered.
>
> I see. Although, I am not 100% if it is something we have to deal with
> in Org mode. May it be better to report this as a bug to LaTeX devs?

hmm... I don't know if this should be considered a bug. We may think
that \\[0pt] should never do anything, but we must keep in mind that the
end of the verse environment is the end of a paragraph, and it changes
to vertical mode. And the end of a paragraph is an illogical place to
put that command. But it seems that it also alters things when it is at
that point. Check out this reply from David Carlisle
(https://tex.stackexchange.com/a/82666):

#+begin_quote
\\ at the end of a paragraph causes bad output with an empty, maximally under-full, box,
and so you get a warning about badness 10000, the visual effect looks a bit like extra
vertical space but it is not: it is an extra spurious line at the end of the paragraph,
and for example it is not dropped at a page break and will break widow/club line
calculations.
#+end_quote

If there are environments that redefine \\, such as verse or tabular,
probably putting the optional argument of \\ with a value of 0pt at the
end of verse alters the calculation of the normal space after the
environment, making it shorter. I think that here we would not have a
LaTeX bug, because the syntax of the verse environment itself says that
the last line should not carry any \\ mark. 

At the end of the tables I have not noticed any side effects. But in the
export of the verse block, I would be in favor of somehow eliminating
that last \\[0pt].

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>>> If I'm not mistaken, in TeX '\\' can only be used in horizontal mode:
>>> that is, within a paragraph. At the end of an environment like verse
>>> (\end{verse}) you are forced to enter vertical mode, and a new paragraph
>>> starts.
>>
>> Is it true for any environment? Or just some?
>
> In principle, nothing prevents one from defining an environment for use
> in horizontal mode (within the paragraph). E.g.:
> ...
> But the usual thing is that the beginning/end of an environment changes to
> vertical mode, with \par or something more elaborate.

This is annoying.
ox-latex has some transcoders that unconditionally add \\ at the
end: clocks and planning lines. If they happen to be at the end of an
environment, it can be problematic.

> hmm... I don't know if this should be considered a bug. We may think
> that \\[0pt] should never do anything, but we must keep in mind that the
> end of the verse environment is the end of a paragraph, and it changes
> to vertical mode. And the end of a paragraph is an illogical place to
> put that command. But it seems that it also alters things when it is at
> that point. Check out this reply from David Carlisle
> (https://tex.stackexchange.com/a/82666):

The very reason we use \\[0pt] is because it supposed to prevent
interpreting [...] at the new line/transcoded element as argument.

You demonstrated that it is yet not always safe enough.

May it be better to use something like

\newcommand\nothing{}
\newcommand{\safenewline}{\\\nothing}

And then use \safenewline instead of \\[0pt]

In my tests,

\begin{center}
\begin{tabular}{ll}
[t] & s\safenewline
[I] & A\safenewline
[m] & kg\safenewline
\end{tabular}
\end{center}

Does not suffer from misinterpreting new line as argument.

> If there are environments that redefine \\, such as verse or tabular,
> probably putting the optional argument of \\ with a value of 0pt at the
> end of verse alters the calculation of the normal space after the
> environment, making it shorter. I think that here we would not have a
> LaTeX bug, because the syntax of the verse environment itself says that
> the last line should not carry any \\ mark. 

AFAIU, \safenewline should still use re-defined version of \\ according
to the context.

> At the end of the tables I have not noticed any side effects. But in the
> export of the verse block, I would be in favor of somehow eliminating
> that last \\[0pt].

`org-latex-verse-block' already has a giant regexp replacement:

      ;; In a verse environment, add a line break to each newline
      ;; character and change each white space at beginning of a line
      ;; into a normal space, calculated with `\fontdimen2\font'.  One
      ;; or more blank lines between lines are exported as a single
      ;; blank line.  If the `:lines' attribute is used, the last
      ;; verse of each stanza ends with the string `\\!', according to
      ;; the syntax of the `verse' package. The separation between
      ;; stanzas can be controlled with the length `\stanzaskip', of
      ;; the aforementioned package.  If the `:literal' attribute is
      ;; used, all blank lines are preserved and exported as
      ;; `\vspace*{\baselineskip}', including the blank lines before
      ;; or after CONTENTS.

We may as well strip the trailing \\[0pt] there.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> The very reason we use \\[0pt] is because it supposed to prevent
> interpreting [...] at the new line/transcoded element as argument.
>
> You demonstrated that it is yet not always safe enough.
>
> May it be better to use something like
>
> \newcommand\nothing{}
> \newcommand{\safenewline}{\\\nothing}
>
> And then use \safenewline instead of \\[0pt]
>
> In my tests,
>
> \begin{center}
> \begin{tabular}{ll}
> [t] & s\safenewline
> [I] & A\safenewline
> [m] & kg\safenewline
> \end{tabular}
> \end{center}
>
> Does not suffer from misinterpreting new line as argument.

I remember the thread where these issues were discussed and the long
discussion where many alternatives were proposed. After all, the \\[0pt]
solution still seems the safest to me. It seems that the problem is
located only in the verse environment, probably due to some particular
redefinition of the \\ macro that makes that environment.

In any case, square brackets are a problematic character in LaTeX
(think, e.g., of some environment that takes an optional argument). I
think pandoc chooses to always export them as {[}{]}:

#+begin_src sh :results latex
str="[hello world] [foo] [bar]"
pandoc -f org -t latex <<< $str
#+end_src

#+RESULTS:
#+begin_export latex
{[}hello world{]} {[}foo{]} {[}bar{]}
#+end_export

We could do the same, but I'm afraid it's too late if
org-latex-line-break-safe already exists... I don't remember if
something similar was proposed in that discussion, and it was rejected
for some reason.

> `org-latex-verse-block' already has a giant regexp replacement:
>
>       ;; In a verse environment, add a line break to each newline
>       ;; character and change each white space at beginning of a line
>       ;; into a normal space, calculated with `\fontdimen2\font'.  One
>       ;; or more blank lines between lines are exported as a single
>       ;; blank line.  If the `:lines' attribute is used, the last
>       ;; verse of each stanza ends with the string `\\!', according to
>       ;; the syntax of the `verse' package. The separation between
>       ;; stanzas can be controlled with the length `\stanzaskip', of
>       ;; the aforementioned package.  If the `:literal' attribute is
>       ;; used, all blank lines are preserved and exported as
>       ;; `\vspace*{\baselineskip}', including the blank lines before
>       ;; or after CONTENTS.
>
> We may as well strip the trailing \\[0pt] there.

I think it would be best to remove the last \\[0pt] in the verse block.
I can prepare a patch, but I'm afraid that org-latex-verse-block is
becoming an homage to replace-regexp-in-string...

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> May it be better to use something like
>>
>> \newcommand\nothing{}
>> \newcommand{\safenewline}{\\\nothing}
>>
>> And then use \safenewline instead of \\[0pt]
>>
>> In my tests,
>>
>> \begin{center}
>> \begin{tabular}{ll}
>> [t] & s\safenewline
>> [I] & A\safenewline
>> [m] & kg\safenewline
>> \end{tabular}
>> \end{center}
>>
>> Does not suffer from misinterpreting new line as argument.
>
> I remember the thread where these issues were discussed and the long
> discussion where many alternatives were proposed. After all, the \\[0pt]
> solution still seems the safest to me. It seems that the problem is
> located only in the verse environment, probably due to some particular
> redefinition of the \\ macro that makes that environment.

We chose \\[0pt] simply because we did not find anything better.
In fact, Max expressed some concerns about \\[0pt] - in
https://list.orgmode.org/orgmode/tik0uf$td1$1@ciao.gmane.io/ and
https://list.orgmode.org/orgmode/tio0th$vn8$1@ciao.gmane.io/

If the idea with custom command does not have obvious downsides, it
might be a better option. In the previous thread, we only considered
redefining \\ itself - obviously a non-starter for environments that
re-define \\ by their own, like here.

> In any case, square brackets are a problematic character in LaTeX
> (think, e.g., of some environment that takes an optional argument). I
> think pandoc chooses to always export them as {[}{]}:
>
> #+begin_src sh :results latex
> str="[hello world] [foo] [bar]"
> pandoc -f org -t latex <<< $str
> #+end_src
>
> #+RESULTS:
> #+begin_export latex
> {[}hello world{]} {[}foo{]} {[}bar{]}
> #+end_export
>
> We could do the same, but I'm afraid it's too late if
> org-latex-line-break-safe already exists... I don't remember if
> something similar was proposed in that discussion, and it was rejected
> for some reason.

It is not too late.

AFAIR, we just decided not to dig deeper about pandoc's approach.

As for {[}{]}, it is a bit difficult to implement. Especially when we
also consider user filters and derived backends. If we have several
transcoders of consequent elements, there is always a risk that even
when a given filter/transcoder is generating a valid LaTeX code,
concatenating them may still cause issues like we have with \\.

I am wondering if there are other examples of commands with optional
arguments that may cause a similar problem with

\command
[unrelated text]

If there are, we may actually want to consider pandoc's approach
seriously.

>> We may as well strip the trailing \\[0pt] there.
>
> I think it would be best to remove the last \\[0pt] in the verse block.
> I can prepare a patch, but I'm afraid that org-latex-verse-block is
> becoming an homage to replace-regexp-in-string...

Not a big deal. Or, if you want, you can make changes via temporary
buffer; if that is cleaner.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 17/01/2024 20:00, Ihor Radchenko wrote:
>>> \newcommand\nothing{}
>>> \newcommand{\safenewline}{\\\nothing}
[...]
>>> \begin{center}
>>> \begin{tabular}{ll}
>>> [t] & s\safenewline
[...]
> If the idea with custom command does not have obvious downsides

It does for tabularray https://github.com/lvjr/tabularray/issues/328
due to a custom parser, but probably it is safe for verse.

I think, it is necessary to wait a couple of years till the fix will 
propagate to all stable Linux distributions.

Notice that an arbitrary custom command is not the same as \empty. 
\safenewline would require more tricks, even \nothing would need

\NewTableCommand\nothing{}
Max Nikulin. Re: Line breaks and brackets in LaTeX export.
Fri, 21 Oct 2022 23:38:29 +0700.
https://list.orgmode.org/tiuhu5$nlp$2@ciao.gmane.io

or

\begin{tblr}[expand=\nothing]{rl}
Max Nikulin. Re: Line breaks and brackets in LaTeX export.
Thu, 20 Oct 2022 00:07:54 +0700.
https://list.orgmode.org/65a1c205-cd18-4abe-b36d-a93763d7f5e8@gmail.com

As to verse, I believe, \\... should be added at the end, but if a user 
added it explicitly then it should not be stripped.

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:


> If the idea with custom command does not have obvious downsides, it
> might be a better option. In the previous thread, we only considered
> redefining \\ itself - obviously a non-starter for environments that
> re-define \\ by their own, like here.

I find several drawbacks to adding a new latex command like \nothing.
First, the standardization of the exported LaTeX code is lost. \\[0pt],
at least, always compiles. A new command obviously needs to be defined
first. Let's imagine that someone wants to simply share the LaTeX code
of a table... Then there is the problem of how to name the new command
so that it doesn't 'clash' with some user-defined command. In LaTeX it
is usually good practice to use the at sign character (@) in the name of
a command or macro that is not in user space, since this character can
only be used in a *.sty file. In a *.tex file, if you want to use the at
sign to define or redefine something, you have to enclose the code
between \makeatletter...\makeatother. And, in any case, I think that the
LaTeX code produced by org should be as 'universal' as possible (standard
LaTeX code + packages included in TeX live), and leave the definition of
new commands or environments to the user's discretion.

On the other hand, we are not sure that a command like \nothing does not
have some undesirable effect. I seem to remember that in the
aforementioned thread, adding \relax (the typical command that is used
to tell LaTeX do nothing) was also proposed as a solution, and it was
discarded for some reason.

>> In any case, square brackets are a problematic character in LaTeX
>> (think, e.g., of some environment that takes an optional argument). I
>> think pandoc chooses to always export them as {[}{]}:
>>
>> #+begin_src sh :results latex
>> str="[hello world] [foo] [bar]"
>> pandoc -f org -t latex <<< $str
>> #+end_src
>>
>> #+RESULTS:
>> #+begin_export latex
>> {[}hello world{]} {[}foo{]} {[}bar{]}
>> #+end_export
>>
>> We could do the same, but I'm afraid it's too late if
>> org-latex-line-break-safe already exists... I don't remember if
>> something similar was proposed in that discussion, and it was rejected
>> for some reason.
>
> It is not too late.
>
> AFAIR, we just decided not to dig deeper about pandoc's approach.
>
> As for {[}{]}, it is a bit difficult to implement. Especially when we
> also consider user filters and derived backends. If we have several
> transcoders of consequent elements, there is always a risk that even
> when a given filter/transcoder is generating a valid LaTeX code,
> concatenating them may still cause issues like we have with \\.

I see. I think pandoc's solution is what Leslie Lamport recommends
(naturally, Lamport doesn't say to enclose /all/ brackets in curly
braces).

> I am wondering if there are other examples of commands with optional
> arguments that may cause a similar problem with
>
> \command
> [unrelated text]
>
> If there are, we may actually want to consider pandoc's approach
> seriously.

In principle, any environment that takes an optional argument in a
"dangerous" position. Just do a simple test. Something like this:

#+begin_figure
[lorem] ipsum
#+end_figure

will throw an error like ''LaTeX Error: Unknown float option...''

Of course, putting an empty line after #+begin... usually solves it. But
the user may not know it.

There are also a number of commands with an optional argument. For
example \pagebreak. Something like this will give an error:

lorem @@latex:\pagebreak@@ [ipsum]

\item is another typical example, but in this case org adds \relax.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> As for {[}{]}, it is a bit difficult to implement. Especially when we
>> also consider user filters and derived backends. If we have several
>> transcoders of consequent elements, there is always a risk that even
>> when a given filter/transcoder is generating a valid LaTeX code,
>> concatenating them may still cause issues like we have with \\.
>
> I see. I think pandoc's solution is what Leslie Lamport recommends
> (naturally, Lamport doesn't say to enclose /all/ brackets in curly
> braces).

This turned out to be a lot easier than I thought.
See the attached patch.

>> \command
>> [unrelated text]
>>
>> If there are, we may actually want to consider pandoc's approach
>> seriously.
>
> In principle, any environment that takes an optional argument in a
> "dangerous" position. Just do a simple test. Something like this:
>
> #+begin_figure
> [lorem] ipsum
> #+end_figure
>
> will throw an error like ''LaTeX Error: Unknown float option...''
>
> Of course, putting an empty line after #+begin... usually solves it. But
> the user may not know it.
>
> There are also a number of commands with an optional argument. For
> example \pagebreak. Something like this will give an error:
>
> lorem @@latex:\pagebreak@@ [ipsum]
>
> \item is another typical example, but in this case org adds \relax.

With the patch, I am getting the following:

* This is test
lorem @@latex:\pagebreak@@ [ipsum]

#+begin_figure
[lorem] figure
#+end_figure

| [foo] | 2 |
| [bar] | 3 |

- [bax]
- [aur]

exports to

lorem \pagebreak {[}ipsum]

\begin{figure}
{[}lorem] figure
\end{figure}

\begin{center}
\begin{tabular}{lr}
{[}foo] & 2\\[0pt]
{[}bar] & 3\\[0pt]
\end{tabular}
\end{center}

\begin{itemize}
\item {[}bax]
\item {[}aur]
\end{itemize}


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-latex-Make-sure-that-text-is-not-misinterpreted-a.patch --]
[-- Type: text/x-patch, Size: 2511 bytes --]

From db3fa01d6f15b3d3f499f77e342a608a822029c8 Mon Sep 17 00:00:00 2001
Message-ID: <db3fa01d6f15b3d3f499f77e342a608a822029c8.1705583005.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 18 Jan 2024 14:01:32 +0100
Subject: [PATCH] ox-latex: Make sure that [text] is not misinterpreted as
 LaTeX argument

* lisp/ox-latex.el (org-latex-plain-text): Protect plain text starting
from [.  It might be misinterpreted as optional command argument if
previous exported fragment ends with a command accepting such.
*
testing/lisp/test-ox-latex.el (text-ox-latex/protect-square-brackets):
Add new test.

Link: https://orgmode.org/list/87o7dju9vn.fsf@posteo.net
---
 lisp/ox-latex.el              |  9 +++++++++
 testing/lisp/test-ox-latex.el | 23 +++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 432f09f4e..a3505c25f 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -3094,6 +3094,15 @@ (defun org-latex-plain-text (text info)
 		    "\\(?:[ \t]*\\\\\\\\\\)?[ \t]*\n"
                     (concat org-latex-line-break-safe "\n")
                     output nil t)))
+    ;; Protect [foo] at the beginning of lines / beginning of the
+    ;; plain-text object.  This prevents LaTeX from unexpectedly
+    ;; interpreting @@latex:\pagebreak@@ [foo] as a command with
+    ;; optional argument.
+    (setq output (replace-regexp-in-string
+                  (rx bol (0+ space) (group "["))
+                  "{[}"
+                  output
+                  nil nil 1))
     ;; Return value.
     output))
 
diff --git a/testing/lisp/test-ox-latex.el b/testing/lisp/test-ox-latex.el
index 41df1b823..237ad97ec 100644
--- a/testing/lisp/test-ox-latex.el
+++ b/testing/lisp/test-ox-latex.el
@@ -29,6 +29,29 @@ (unless (featurep 'ox-latex)
 
 \f
 
+(ert-deftest text-ox-latex/protect-square-brackets ()
+  "Test [foo] being interpreted as plain text even after LaTeX commands."
+  (org-test-with-exported-text
+      'latex
+      "* This is test
+lorem @@latex:\\pagebreak@@ [ipsum]
+
+#+begin_figure
+[lorem] figure
+#+end_figure
+
+| [foo] | 2 |
+| [bar] | 3 |
+
+- [bax]
+- [aur]
+"
+    (goto-char (point-min))
+    (should (search-forward "lorem \\pagebreak {[}ipsum]"))
+    (should (search-forward "{[}lorem] figure"))
+    (should (search-forward "{[}foo]"))
+    (should (search-forward "\\item {[}bax]"))))
+
 (ert-deftest test-ox-latex/verse ()
   "Test verse blocks."
   (org-test-with-exported-text
-- 
2.43.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> This turned out to be a lot easier than I thought.
> See the attached patch.
>
>>> \command
>>> [unrelated text]
>>>
>>> If there are, we may actually want to consider pandoc's approach
>>> seriously.
>>
>> In principle, any environment that takes an optional argument in a
>> "dangerous" position. Just do a simple test. Something like this:
>>
>> #+begin_figure
>> [lorem] ipsum
>> #+end_figure
>>
>> will throw an error like ''LaTeX Error: Unknown float option...''
>>
>> Of course, putting an empty line after #+begin... usually solves it. But
>> the user may not know it.
>>
>> There are also a number of commands with an optional argument. For
>> example \pagebreak. Something like this will give an error:
>>
>> lorem @@latex:\pagebreak@@ [ipsum]
>>
>> \item is another typical example, but in this case org adds \relax.
>
> With the patch, I am getting the following:
>
> * This is test
> lorem @@latex:\pagebreak@@ [ipsum]
>
> #+begin_figure
> [lorem] figure
> #+end_figure
>
> | [foo] | 2 |
> | [bar] | 3 |
>
> - [bax]
> - [aur]
>
> exports to
>
> lorem \pagebreak {[}ipsum]
>
> \begin{figure}
> {[}lorem] figure
> \end{figure}
>
> \begin{center}
> \begin{tabular}{lr}
> {[}foo] & 2\\[0pt]
> {[}bar] & 3\\[0pt]
> \end{tabular}
> \end{center}
>
> \begin{itemize}
> \item {[}bax]
> \item {[}aur]
> \end{itemize}

Great! Simple and effective. And more surgical than pandoc's global
solution. But now a question arises... Your code clearly solves the
problem that led to the declaration of org-latex-line-break-safe,
without foreseeing any unwanted effects, since it is the solution that
is usually recommended from LaTeX. So, if this code is included in Org,
what is the future of org-latex-line-break-safe?

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 18/01/2024 20:05, Ihor Radchenko wrote:
> With the patch, I am getting the following:[...]
> \begin{center}
> \begin{tabular}{lr}
> {[}foo] & 2\\[0pt]
> {[}bar] & 3\\[0pt]
> \end{tabular}
> \end{center}

It means that [0pt] is not necessary any more. However users will have 
adjust their filters once more. I have no idea if many of them will 
complain concerning {[} where it is not really required. (/[omitted]/).

To have an additional excuse, it is better to add a note that it is 
inspired by pandoc strategy.

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 13/01/2024 22:08, Ihor Radchenko wrote:
> Juan Manuel Macías writes:
> 
>> Can anyone think of a possible scenario where removing the '\\[0pt]' in
>> the last line would cause a problem?

I would suggest to strip leading and trailing newlines before processing 
of the block content.

> In such scenario, we may technically violate what users ask us to do:
> 
> #+begin_verse
> I really want newline here\\
> #+end_verse

I really want newline here@@latex:\\%@@

P.S. Juan Manuel, I have sent another message to this thread without 
your address in Cc.

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> On 18/01/2024 20:05, Ihor Radchenko wrote:
>> With the patch, I am getting the following:[...]
>> \begin{center}
>> \begin{tabular}{lr}
>> {[}foo] & 2\\[0pt]
>> {[}bar] & 3\\[0pt]
>> \end{tabular}
>> \end{center}
>
> It means that [0pt] is not necessary any more. However users will have
> adjust their filters once more. I have no idea if many of them will
> complain concerning {[} where it is not really required.
> (/[omitted]/).
>
> To have an additional excuse, it is better to add a note that it is
> inspired by pandoc strategy.

I agree with the note. Perhaps both strategies could coexist: by
default, Ihor's code; and org-latex-line-break-safe with value "\\\\".
And, after some time, the latter could become obsolete, unless a new
''raison d'etre'' is found for it...

In any case, I would no longer see it necessary to modify
org-latex-verse-block anymore.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> \begin{itemize}
>> \item {[}bax]
>> \item {[}aur]
>> \end{itemize}
>
> Great! Simple and effective. And more surgical than pandoc's global
> solution. But now a question arises... Your code clearly solves the
> problem that led to the declaration of org-latex-line-break-safe,
> without foreseeing any unwanted effects, since it is the solution that
> is usually recommended from LaTeX. So, if this code is included in Org,
> what is the future of org-latex-line-break-safe?

It is still useful in situations like

Paragraph\\
#+begin_export latex
[foo]
#+end_export

or

Paragraph\\
@@latex:[foo]@@

My patch does not do anything about verbatim environments.

We may remove the use of `org-latex-line-break-safe' from some places,
but not all.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> Can anyone think of a possible scenario where removing the '\\[0pt]' in
>>> the last line would cause a problem?
>
> I would suggest to strip leading and trailing newlines before processing 
> of the block content.

May you elaborate?

>> In such scenario, we may technically violate what users ask us to do:
>> 
>> #+begin_verse
>> I really want newline here\\
>> #+end_verse
>
> I really want newline here@@latex:\\%@@

This implies that Org mode does not follow its own markup. I disagree
with such suggestion.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> It means that [0pt] is not necessary any more. However users will have 
> adjust their filters once more. I have no idea if many of them will 
> complain concerning {[} where it is not really required. (/[omitted]/).

We do not promise the exact way the exported LaTeX file looks like. User
filters are too flexible - we cannot reasonably guarantee that they will
keep working if they rely on the way Org mode transcodes to LaTeX in
specific Org version.

> To have an additional excuse, it is better to add a note that it is 
> inspired by pandoc strategy.

I can add it to the code comment, but for different reason. We do not
give "excuses" to make breaking changes - https://bzg.fr/en/the-software-maintainers-pledge/
But we do keep the flexibility to make changes where Org mode behavior
is undefined. Like in this case.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Juan Manuel Macías <maciaschain@posteo.net> writes:
>
>>> \begin{itemize}
>>> \item {[}bax]
>>> \item {[}aur]
>>> \end{itemize}
>>
>> Great! Simple and effective. And more surgical than pandoc's global
>> solution. But now a question arises... Your code clearly solves the
>> problem that led to the declaration of org-latex-line-break-safe,
>> without foreseeing any unwanted effects, since it is the solution that
>> is usually recommended from LaTeX. So, if this code is included in Org,
>> what is the future of org-latex-line-break-safe?
>
> It is still useful in situations like
>
> Paragraph\\
>
> #+begin_export latex
> [foo]
> #+end_export
>
> or
>
> Paragraph\\
> @@latex:[foo]@@

But since in both cases it is literal LaTeX code, the correct thing
would be for the user to be in charge of adding the curly braces to the
square brackets. I mean in such scenario it is LaTeX code, not Org.

Anyway, in both examples it does not make sense for LaTeX to end a
paragraph with the \\ macro, which is why we commented in previous
messages in the thread. The \\ macro is only used in horizontal mode;
this macro does not add a new paragraph but rather a forced line break
within the paragraph.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> Paragraph\\
>> @@latex:[foo]@@
>
> But since in both cases it is literal LaTeX code, the correct thing
> would be for the user to be in charge of adding the curly braces to the
> square brackets. I mean in such scenario it is LaTeX code, not Org.

Not really. Because we do not give guarantees about the details of LaTeX
export, we should try hard to not break things that users may put into
literal export snippets.

> Anyway, in both examples it does not make sense for LaTeX to end a
> paragraph with the \\ macro, which is why we commented in previous
> messages in the thread. The \\ macro is only used in horizontal mode;
> this macro does not add a new paragraph but rather a forced line break
> within the paragraph.

In the example with @@latex:[foo]@@, \\ is not at the end of a paragraph
- it is inside paragraph.

Another alternative could be modifying how inline latex snippet is
exported, but that's worse IMHO - we should try hard not to touch what
users tell Org to do explicitly.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Juan Manuel Macías <maciaschain@posteo.net> writes:
>
>>> Paragraph\\
>>> @@latex:[foo]@@
>>
>> But since in both cases it is literal LaTeX code, the correct thing
>> would be for the user to be in charge of adding the curly braces to the
>> square brackets. I mean in such scenario it is LaTeX code, not Org.
>
> Not really. Because we do not give guarantees about the details of LaTeX
> export, we should try hard to not break things that users may put into
> literal export snippets.
>
>> Anyway, in both examples it does not make sense for LaTeX to end a
>> paragraph with the \\ macro, which is why we commented in previous
>> messages in the thread. The \\ macro is only used in horizontal mode;
>> this macro does not add a new paragraph but rather a forced line break
>> within the paragraph.
>
> In the example with @@latex:[foo]@@, \\ is not at the end of a paragraph
> - it is inside paragraph.

What I mean is that literal LaTeX code is LaTeX code, despite the
redundancy. IMHO, Org's only duty in that case is to export such literal
code as valid LaTeX code, but Org "does not know" what that literal code
consists of. The user who enters the literal LaTeX code is the one who
has the duty to add the necessary elements to that code so that it is
compiled correctly by LaTeX. Let's look at this as a territorial
question:

Scenario A:

@@LaTeX:\libebreak@@ [ipsum] ==> the problem is in Org territory

Scenario B:

lorem\\ @@latex:[ipsum]@@ ==> the problem is in the user's territory

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Scenario B:
>
> lorem\\ @@latex:[ipsum]@@ ==> the problem is in the user's territory

I see your point.
Although I am still a bit hesitant to remove
`org-latex-line-break-safe'.
What would be the benefit of removing it? For now, I mostly just see
that it will make the life harder for users in Scenario B.

For verse blocks, AFAIU, even \\ is problematic. (Correct me if I am wrong)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> I see your point.
> Although I am still a bit hesitant to remove
> `org-latex-line-break-safe'.
> What would be the benefit of removing it? For now, I mostly just see
> that it will make the life harder for users in Scenario B.

It's a complicated situation, because we now have two solutions to the
same problem... It certainly sounds a bit abrupt to remove
org-latex-line-break-safe (at least for now). I see no problem in both
solutions coexisting. After all, the user can always give an "\\\\"
value to org-latex-line-break-safe. The other possibility is that
org-latex-line-break-safe is selectively deleted, as you mentioned in a
previous email. In tables and verse blocks, unless I'm missing
something, I think adding [0pt] would be unnecessary, with the new
solution.

> For verse blocks, AFAIU, even \\ is problematic. (Correct me if I am wrong)

From the tests I have done, the problem of the vertical space being
altered after the verse environment only appears when the last line ends
in \\[0pt]. If it ends in \\ the problem does not appear. It is not
recommended that a verse environment ends in \\, but it does not seem to
influence the output. In fact, that was how it was in Org in the days
pre org-latex-line-break-safe and there were never any problems.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 20/01/2024 19:35, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>>> Can anyone think of a possible scenario where removing the '\\[0pt]' in
>>>> the last line would cause a problem?
>>
>> I would suggest to strip leading and trailing newlines before processing
>> of the block content.
> 
> May you elaborate?

I expect that

#+BEGIN_VERSE


 From fairest creatures we desire increase,
That thereby beauty’s rose might never die,



#+END_VERSE

is exported to

\begin{verse}
 From fairest creatures we desire increase,\\
That thereby beauty’s rose might never die,
\end{verse}

and internally at some point it is represented as
"From fairest creatures we desire increase,\nThat thereby beauty’s rose 
might never die,"
without leading and trailing newlines.

>>> In such scenario, we may technically violate what users ask us to do:
>>>
>>> #+begin_verse
>>> I really want newline here\\
>>> #+end_verse
>>
>> I really want newline here@@latex:\\%@@
> 
> This implies that Org mode does not follow its own markup. I disagree
> with such suggestion.

I do not see any use case for "\\" at the last line. Perhaps it might 
even cause an org-lint warning. From my point of view, it is preferable 
to require more verbose markup in special cases to make intentions explicit.

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 20/01/2024 19:41, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> To have an additional excuse, it is better to add a note that it is
>> inspired by pandoc strategy.
> 
> I can add it to the code comment, but for different reason. We do not
> give "excuses" to make breaking changes - https://bzg.fr/en/the-software-maintainers-pledge/
> But we do keep the flexibility to make changes where Org mode behavior
> is undefined. Like in this case.

De-facto \\[0pt] was a breaking change despite formal contracts. I do 
not consider it as a wrong decision though. It is the case when where 
was no good choice, instead just ones that may cause some troubles 
(including leaving everything as is).

I recall some threads (besides ones started by Juan Manuel) where users 
considered [0pt] as an undesired consequence of some other changes. That 
is why I wrote "excuse".

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Max Nikulin]

On 21/01/2024 01:47, Ihor Radchenko wrote:
> Juan Manuel Macías writes:
>>
>> lorem\\ @@latex:[ipsum]@@ ==> the problem is in the user's territory

I agree with Juan Manuel.

> I see your point.
> Although I am still a bit hesitant to remove
> `org-latex-line-break-safe'.
> What would be the benefit of removing it? For now, I mostly just see
> that it will make the life harder for users in Scenario B.

An option might be disabling it by default without removing the code 
completely. Depending on user feedback it may be removed later.

Users, who need to submit LaTeX code, consider [0pt] as undesired noise. 
I see their point of view.

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> Although I am still a bit hesitant to remove
>> `org-latex-line-break-safe'.
>> What would be the benefit of removing it? For now, I mostly just see
>> that it will make the life harder for users in Scenario B.
>
> It's a complicated situation, because we now have two solutions to the
> same problem... It certainly sounds a bit abrupt to remove
> org-latex-line-break-safe (at least for now). I see no problem in both
> solutions coexisting. After all, the user can always give an "\\\\"
> value to org-latex-line-break-safe. The other possibility is that
> org-latex-line-break-safe is selectively deleted, as you mentioned in a
> previous email. In tables and verse blocks, unless I'm missing
> something, I think adding [0pt] would be unnecessary, with the new
> solution.

Upon looking closer into selective removal, it turned out to be more
tricky than I thought. I'm afraid that using \\[0pt] only in some places
may become a bit of headache to maintain - we may accidentally break
certain regexp replacements in `org-latex-verse-block'. In particular,
when verse contents is not straightforward and uses \\[0pt].

Given that `org-latex-line-break-safe' also introduced the problem with
verse blocks, I decided that it is better to remove it at the end.

See the attached tentative patchset.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-ox-latex-Make-sure-that-text-is-not-misinterprete.patch --]
[-- Type: text/x-patch, Size: 2518 bytes --]

From d2e74cc2734eaf8d782f5acf66b11956d6ffa47e Mon Sep 17 00:00:00 2001
Message-ID: <d2e74cc2734eaf8d782f5acf66b11956d6ffa47e.1705844367.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 18 Jan 2024 14:01:32 +0100
Subject: [PATCH v2 1/2] ox-latex: Make sure that [text] is not misinterpreted
 as LaTeX argument

* lisp/ox-latex.el (org-latex-plain-text): Protect plain text starting
from [.  It might be misinterpreted as optional command argument if
previous exported fragment ends with a command accepting such.
*
testing/lisp/test-ox-latex.el (text-ox-latex/protect-square-brackets):
Add new test.

Link: https://orgmode.org/list/87o7dju9vn.fsf@posteo.net
---
 lisp/ox-latex.el              |  9 +++++++++
 testing/lisp/test-ox-latex.el | 23 +++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index df20345f8..a64dd5a87 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -3095,6 +3095,15 @@ (defun org-latex-plain-text (text info)
 		    "\\(?:[ \t]*\\\\\\\\\\)?[ \t]*\n"
                     (concat org-latex-line-break-safe "\n")
                     output nil t)))
+    ;; Protect [foo] at the beginning of lines / beginning of the
+    ;; plain-text object.  This prevents LaTeX from unexpectedly
+    ;; interpreting @@latex:\pagebreak@@ [foo] as a command with
+    ;; optional argument.
+    (setq output (replace-regexp-in-string
+                  (rx bol (0+ space) (group "["))
+                  "{[}"
+                  output
+                  nil nil 1))
     ;; Return value.
     output))
 
diff --git a/testing/lisp/test-ox-latex.el b/testing/lisp/test-ox-latex.el
index 41df1b823..237ad97ec 100644
--- a/testing/lisp/test-ox-latex.el
+++ b/testing/lisp/test-ox-latex.el
@@ -29,6 +29,29 @@ (unless (featurep 'ox-latex)
 
 \f
 
+(ert-deftest text-ox-latex/protect-square-brackets ()
+  "Test [foo] being interpreted as plain text even after LaTeX commands."
+  (org-test-with-exported-text
+      'latex
+      "* This is test
+lorem @@latex:\\pagebreak@@ [ipsum]
+
+#+begin_figure
+[lorem] figure
+#+end_figure
+
+| [foo] | 2 |
+| [bar] | 3 |
+
+- [bax]
+- [aur]
+"
+    (goto-char (point-min))
+    (should (search-forward "lorem \\pagebreak {[}ipsum]"))
+    (should (search-forward "{[}lorem] figure"))
+    (should (search-forward "{[}foo]"))
+    (should (search-forward "\\item {[}bax]"))))
+
 (ert-deftest test-ox-latex/verse ()
   "Test verse blocks."
   (org-test-with-exported-text
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v2-0002-ox-latex-Remove-org-latex-line-break-safe.patch --]
[-- Type: text/x-patch, Size: 11526 bytes --]

From dd4618b31dc5070d6802e484fd58cf50f5d3606d Mon Sep 17 00:00:00 2001
Message-ID: <dd4618b31dc5070d6802e484fd58cf50f5d3606d.1705844367.git.yantar92@posteo.net>
In-Reply-To: <d2e74cc2734eaf8d782f5acf66b11956d6ffa47e.1705844367.git.yantar92@posteo.net>
References: <d2e74cc2734eaf8d782f5acf66b11956d6ffa47e.1705844367.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 21 Jan 2024 14:21:33 +0100
Subject: [PATCH v2 2/2] ox-latex: Remove org-latex-line-break-safe

This reverts commit 3f60acff7717e472d06833e9cf9fff6ca3d59337 and
subsequent relevant comments.

* lisp/ox-latex.el (org-latex-line-break-safe): Remove constant.  The
\\[0pt] is actually not safe to use in some scenarios. We use a
different approach to avoid plain text [...] being interpreted as
LaTeX optional argument - we escape [ like {[}; that's what pandoc
does.
(org-latex-clock):
(org-latex-line-break):
(org-latex-plain-text):
(org-latex-planning):
(org-latex--org-table):
(org-latex--math-table):
(org-latex-table-row):
(org-latex-verse-block):
* testing/lisp/test-org-table.el (test-org-table/to-latex):
* testing/lisp/test-ox-latex.el (test-ox-latex/verse):
(test-ox-latex/longtable): Remove references to
`org-latex-line-break-safe'.
* etc/ORG-NEWS (=ox-latex=: ~org-latex-line-break-safe~ is removed):
Announce removal.
* lisp/org-compat.el (org-latex-line-break-safe): Make obsolete.

Link: https://orgmode.org/list/878r4jg37s.fsf@posteo.net
---
 etc/ORG-NEWS                   | 10 +++++++
 lisp/org-compat.el             | 18 +++++++++++++
 lisp/ox-latex.el               | 49 ++++++++++++----------------------
 testing/lisp/test-org-table.el |  6 ++---
 testing/lisp/test-ox-latex.el  | 16 +++++------
 5 files changed, 56 insertions(+), 43 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 847ddf614..46f1e66a7 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -13,6 +13,16 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
 ** Important announcements and breaking changes
+*** =ox-latex=: ~org-latex-line-break-safe~ is removed
+
+~org-latex-line-break-safe~ constant was previously introduced to deal
+with edge cases when LaTeX interprets [...] as LaTeX command
+argument.  However, it caused a number of other issues and proved
+itself not to be as "safe" as it supposed to be.
+
+We now use a Pandoc's approach to deal with the same problem,
+utilizing ={[}= to escape =[...]= instances where needed.
+
 *** ~org-agenda-search-headline-for-time~ now ignores all the timestamp in headings
 
 Previously, ~org-agenda-search-headline-for-time~ made Org agenda
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 17f07db9d..27d094b07 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -629,6 +629,24 @@ (define-obsolete-variable-alias 'org-reveal-start-hook
 (define-obsolete-function-alias 'org-file-url-p 'org-url-p "9.6")
 (define-obsolete-variable-alias 'org-plantuml-executable-args 'org-plantuml-args
   "Org 9.6")
+
+(defconst org-latex-line-break-safe "\\\\[0pt]"
+  "Linebreak protecting the following [...].
+
+Without \"[0pt]\" it would be interpreted as an optional argument to
+the \\\\.
+
+This constant, for example, makes the below code not err:
+
+\\begin{tabular}{c|c}
+    [t] & s\\\\[0pt]
+    [I] & A\\\\[0pt]
+    [m] & kg
+\\end{tabular}")
+(make-obsolete 'org-latex-line-break-safe
+               "should not be used - it is not safe in all the scenarios."
+               "9.7")
+
 (defun org-in-fixed-width-region-p ()
   "Non-nil if point in a fixed-width region."
   (save-match-data
diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index a64dd5a87..d6e74cb45 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -298,23 +298,10 @@ (defconst org-latex-language-alist
 - `:script-tag' the script otf tag.")
 
 
-(defconst org-latex-line-break-safe "\\\\[0pt]"
-  "Linebreak protecting the following [...].
 
-Without \"[0pt]\" it would be interpreted as an optional argument to
-the \\\\.
-
-This constant, for example, makes the below code not err:
-
-\\begin{tabular}{c|c}
-    [t] & s\\\\[0pt]
-    [I] & A\\\\[0pt]
-    [m] & kg
-\\end{tabular}")
-
-(defconst org-latex-table-matrix-macros `(("bordermatrix" . "\\cr")
+(defconst org-latex-table-matrix-macros '(("bordermatrix" . "\\cr")
 					  ("qbordermatrix" . "\\cr")
-					  ("kbordermatrix" . ,org-latex-line-break-safe))
+					  ("kbordermatrix" . "\\\\"))
   "Alist between matrix macros and their row ending.")
 
 (defconst org-latex-math-environments-re
@@ -2124,7 +2111,7 @@ (defun org-latex-clock (clock _contents info)
 	   (concat (org-timestamp-translate (org-element-property :value clock))
 		   (let ((time (org-element-property :duration clock)))
 		     (and time (format " (%s)" time)))))
-   org-latex-line-break-safe))
+   "\\\\"))
 
 
 ;;;; Code
@@ -2722,7 +2709,7 @@ ;;;; Line Break
 (defun org-latex-line-break (_line-break _contents _info)
   "Transcode a LINE-BREAK object from Org to LaTeX.
 CONTENTS is nil.  INFO is a plist holding contextual information."
-  (concat org-latex-line-break-safe "\n"))
+  "\\\\\n")
 
 
 ;;;; Link
@@ -3092,9 +3079,7 @@ (defun org-latex-plain-text (text info)
     ;; Handle break preservation if required.
     (when (plist-get info :preserve-breaks)
       (setq output (replace-regexp-in-string
-		    "\\(?:[ \t]*\\\\\\\\\\)?[ \t]*\n"
-                    (concat org-latex-line-break-safe "\n")
-                    output nil t)))
+		    "\\(?:[ \t]*\\\\\\\\\\)?[ \t]*\n" "\\\\\n" output nil t)))
     ;; Protect [foo] at the beginning of lines / beginning of the
     ;; plain-text object.  This prevents LaTeX from unexpectedly
     ;; interpreting @@latex:\pagebreak@@ [foo] as a command with
@@ -3139,7 +3124,7 @@ (defun org-latex-planning (planning _contents info)
 		(format (plist-get info :latex-active-timestamp-format)
 			(org-timestamp-translate scheduled)))))))
     " ")
-   org-latex-line-break-safe))
+   "\\\\"))
 
 
 ;;;; Property Drawer
@@ -3919,11 +3904,11 @@ (defun org-latex--org-table (table contents info)
 		(format "\\begin{%s}%s{%s}\n" table-env width alignment)
 		(and above?
 		     (org-string-nw-p caption)
-		     (concat caption org-latex-line-break-safe "\n"))
+		     (concat caption "\\\\\n"))
 		contents
 		(and (not above?)
 		     (org-string-nw-p caption)
-		     (concat caption org-latex-line-break-safe "\n"))
+		     (concat caption "\\\\\n"))
 		(format "\\end{%s}" table-env)
 		(and fontsize "}"))))
      (t
@@ -4008,7 +3993,7 @@ (defun org-latex--math-table (table info)
 		 (lambda (cell)
 		   (substring (org-element-interpret-data cell) 0 -1))
 		 (org-element-map row 'table-cell #'identity info) "&")
-		(or (cdr (assoc env org-latex-table-matrix-macros)) org-latex-line-break-safe)
+		(or (cdr (assoc env org-latex-table-matrix-macros)) "\\\\")
 		"\n")))
 	   (org-element-map table 'table-row #'identity info) "")))
     (concat
@@ -4083,7 +4068,7 @@ (defun org-latex-table-row (table-row contents info)
             (setq table-head-cache (make-hash-table :test #'eq))
             (plist-put info :org-latex-table-head-cache table-head-cache))
           (if-let ((head-contents (gethash (org-element-parent table-row) table-head-cache)))
-              (puthash (org-element-parent table-row) (concat head-contents org-latex-line-break-safe "\n" contents)
+              (puthash (org-element-parent table-row) (concat head-contents "\\\\\n" contents)
                        table-head-cache)
             (puthash (org-element-parent table-row) contents table-head-cache))))
       ;; Return LaTeX string as the transcoder.
@@ -4092,7 +4077,7 @@ (defun org-latex-table-row (table-row contents info)
        ;; hline was specifically marked.
        (and booktabsp (not (org-export-get-previous-element table-row info))
 	    "\\toprule\n")
-       contents org-latex-line-break-safe "\n"
+       contents "\\\\\n"
        (cond
 	;; Special case for long tables.  Define header and footers.
 	((and longtablep (org-export-table-row-ends-header-p table-row info))
@@ -4100,9 +4085,9 @@ (defun org-latex-table-row (table-row contents info)
 			      (org-element-lineage table-row 'table) info))))
 	   (format "%s
 \\endfirsthead
-\\multicolumn{%d}{l}{%s} \\\\[0pt]
+\\multicolumn{%d}{l}{%s} \\\\
 %s
-%s \\\\[0pt]\n
+%s \\\\\n
 %s
 \\endhead
 %s\\multicolumn{%d}{r}{%s} \\\\
@@ -4211,15 +4196,15 @@ (defun org-latex-verse-block (verse-block contents info)
 	       (replace-regexp-in-string
                 (if (not lit)
 		    (rx-to-string
-                     `(seq (group ,org-latex-line-break-safe "\n")
-		           (1+ (group line-start (0+ space) ,org-latex-line-break-safe "\n"))))
-		  (concat "^[ \t]*" (regexp-quote org-latex-line-break-safe) "$"))
+                     `(seq (group "\\\\\n")
+		           (1+ (group line-start (0+ space) "\\\\\n"))))
+		  "^[ \t]*\\\\$")
 	        (if (not lit)
 		    (if lin "\\\\!\n\n" "\n\n")
 		  "\\vspace*{\\baselineskip}")
 	        (replace-regexp-in-string
 	         "\\([ \t]*\\\\\\\\\\)?[ \t]*\n"
-                 (concat org-latex-line-break-safe "\n")
+                 "\\\\\n"
 	         (if (not lit)
 		     (concat (org-trim contents t) "\n")
 		   contents)
diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el
index 92ccd2a05..6ee790894 100644
--- a/testing/lisp/test-org-table.el
+++ b/testing/lisp/test-org-table.el
@@ -1633,11 +1633,11 @@ (ert-deftest test-org-table/to-generic ()
 (ert-deftest test-org-table/to-latex ()
   "Test `orgtbl-to-latex' specifications."
   (should
-   (equal "\\begin{tabular}{l}\na\\\\[0pt]\n\\end{tabular}"
+   (equal "\\begin{tabular}{l}\na\\\\\n\\end{tabular}"
 	  (orgtbl-to-latex (org-table-to-lisp "| a |") nil)))
   ;; Test :environment parameter.
   (should
-   (equal "\\begin{tabularx}{l}\na\\\\[0pt]\n\\end{tabularx}"
+   (equal "\\begin{tabularx}{l}\na\\\\\n\\end{tabularx}"
 	  (orgtbl-to-latex (org-table-to-lisp "| a |")
 			   '(:environment "tabularx"))))
   ;; Test :booktabs parameter.
@@ -1646,7 +1646,7 @@ (ert-deftest test-org-table/to-latex ()
     "\\toprule" (orgtbl-to-latex (org-table-to-lisp "| a |") '(:booktabs t))))
   ;; Handle LaTeX snippets.
   (should
-   (equal "\\begin{tabular}{l}\n\\(x\\)\\\\[0pt]\n\\end{tabular}"
+   (equal "\\begin{tabular}{l}\n\\(x\\)\\\\\n\\end{tabular}"
 	  (orgtbl-to-latex (org-table-to-lisp "| $x$ |") nil)))
   ;; Test pseudo objects and :raw parameter.
   (should
diff --git a/testing/lisp/test-ox-latex.el b/testing/lisp/test-ox-latex.el
index 237ad97ec..d0be4e5a4 100644
--- a/testing/lisp/test-ox-latex.el
+++ b/testing/lisp/test-ox-latex.el
@@ -71,14 +71,14 @@ (ert-deftest test-ox-latex/verse ()
     (should
      (search-forward
       "\\begin{verse}
-lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\
 lorem ipsum dolor
 
-lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\
 lorem ipsum dolor
 
-lorem ipsum dolor\\\\[0pt]
-lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\
+lorem ipsum dolor\\\\
 \\end{verse}"))))
 
 (ert-deftest test-ox-latex/longtable ()
@@ -98,15 +98,15 @@ (ert-deftest test-ox-latex/longtable ()
     (should
      (search-forward
       "\\begin{longtable}{lr}
-First & Second\\\\[0pt]
-Column & Column\\\\[0pt]
+First & Second\\\\
+Column & Column\\\\
 \\hline
 \\endfirsthead"))
     (goto-char (point-min))
     (should
      (search-forward
-      "First & Second\\\\[0pt]
-Column & Column \\\\[0pt]
+      "First & Second\\\\
+Column & Column \\\\
 
 \\hline
 \\endhead"))
-- 
2.43.0


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Upon looking closer into selective removal, it turned out to be more
> tricky than I thought. I'm afraid that using \\[0pt] only in some places
> may become a bit of headache to maintain - we may accidentally break
> certain regexp replacements in `org-latex-verse-block'. In particular,
> when verse contents is not straightforward and uses \\[0pt].
>
> Given that `org-latex-line-break-safe' also introduced the problem with
> verse blocks, I decided that it is better to remove it at the end.

LaTeX code generated without org-latex-line-break-safe is much cleaner.
The decision to make this variable obsolete seems correct to me, if
there is already a better and more discreet solution. I don't know if it
will cause many problems with custom settings and filters written in
relation to this variable, as Maxim commented in a previous email... I
hope not. I have a couple of filters written already, but I don't think
it will take much work for me to readjust them.

Best regards,

Juan Manuel 

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Given that `org-latex-line-break-safe' also introduced the problem with
> verse blocks, I decided that it is better to remove it at the end.
>
> See the attached tentative patchset.

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=03b383df8

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [possible patch] Remove the '\\[0pt]' string from the last line of a verse block in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> I would suggest to strip leading and trailing newlines before processing
>>> of the block content.
>> 
>> May you elaborate?
>
> I expect that
>
> #+BEGIN_VERSE
>
>
>  From fairest creatures we desire increase,
> That thereby beauty’s rose might never die,
>
>
>
> #+END_VERSE
>
> is exported to
>
> \begin{verse}
>  From fairest creatures we desire increase,\\
> That thereby beauty’s rose might never die,
> \end{verse}
>
> and internally at some point it is represented as
> "From fairest creatures we desire increase,\nThat thereby beauty’s rose 
> might never die,"
> without leading and trailing newlines.

This has nothing to do with what happens when exporting verse blocks.

If you try a simple example like

#+options: toc:nil
* Heading
First.

Foo
#+BEGIN_VERSE

 From fairest creatures we desire increase,
That thereby beauty’s rose might never die,

#+END_VERSE
Bar

Foo
#+BEGIN_VERSE
 From fairest creatures we desire increase,
That thereby beauty’s rose might never die,
#+END_VERSE
Bar

you will see that lading and trailing spaces are insignificant:

\section{Heading}
\label{sec:orgc54e19b}
First.

Foo
\begin{verse}
\hspace*{1\fontdimen2\font}From fairest creatures we desire increase,\\
That thereby beauty’s rose might never die,\\
\end{verse}
Bar

Foo
\begin{verse}
\hspace*{1\fontdimen2\font}From fairest creatures we desire increase,\\
That thereby beauty’s rose might never die,\\
\end{verse}
Bar

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>