Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Thibaut Verron]

Hello,

I'm forwarding this question asked on stackexchange: 
http://tex.stackexchange.com/questions/186605/with-orgtbl-how-to-ensure-
that-braces-and-dollars-are-not-escaped 

After some investigation, it seems that the behavior is hidden deep in the 
export routines, and I was wisely suggested to ask the question on this list 
instead.

I have given some tex-related details in the linked question, including some 
motivations and an example, the tl;dr is that in some conditions, the 
orgtbl-to-latex exporter will perform arbitrary escape of some characters in 
the cells, or other kind of transformations:

 $\text{test}$ 
is exported verbatim (OK).

But
 \pbox{test}    
becomes
 \pbox\{test\}

 {test}
becomes
 \{test\}

 {$test$}
becomes
 \{\$test\$\}

And the exporter seems to be trying to be smart, because it will still 
ensure that the result is correct:

 {$\infty$}
becomes
 \{\$$\infty$\$\}

The weirdest of all might be this one:
 \pbox{Foo: \\${bar= (2^{3},1)}$, ${baz= (8^{4})}$}
becomes 
 \pbox\{Foo: \\${bar= (2^{3},1)}$, \$\{baz= (8$^{\text{4}}$)\}\$\}

(Note how the two mathematical expressions recieve different treatment, and 
the decision to insert "\text" around the exponent!)

The option `:no-export`, as expected, has no effect, since it only controls 
whether `#_^&%` are escaped or not. 

Is this a known feature, or a bug? And is there a known workaround?

Thanks,

Thibaut Verron

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Nicolas Goaziou]

Hello,

Thibaut Verron <thibaut.verron@gmail.com> writes:

> I'm forwarding this question asked on stackexchange: 
> http://tex.stackexchange.com/questions/186605/with-orgtbl-how-to-ensure-
> that-braces-and-dollars-are-not-escaped 
>
> After some investigation, it seems that the behavior is hidden deep in the 
> export routines, and I was wisely suggested to ask the question on this list 
> instead.
>
> I have given some tex-related details in the linked question, including some 
> motivations and an example, the tl;dr is that in some conditions, the 
> orgtbl-to-latex exporter will perform arbitrary escape of some characters in 
> the cells, or other kind of transformations:
>
>  $\text{test}$ 
> is exported verbatim (OK).
>
> But
>  \pbox{test}    
> becomes
>  \pbox\{test\}

You should upgrade to Org 8.

>  {test}
> becomes
>  \{test\}

This is intended.

>  {$test$}
> becomes
>  \{\$test\$\}

This is to be expected as matching single dollar math snippets is
fragile. I suggest to use \(...\) instead: {\(test\)}

> And the exporter seems to be trying to be smart, because it will still 
> ensure that the result is correct:
>
>  {$\infty$}
> becomes
>  \{\$$\infty$\$\}

Ditto. Use \(...\). Or, better, {\infty} as \infty is an Org entity
which will properly translated into LaTeX code.

> The weirdest of all might be this one:
>  \pbox{Foo: \\${bar= (2^{3},1)}$, ${baz= (8^{4})}$}
> becomes 
>  \pbox\{Foo: \\${bar= (2^{3},1)}$, \$\{baz= (8$^{\text{4}}$)\}\$\}

You cannot write raw LaTeX macros with complicated arguments (e.g.,
containing braces) in an Org buffer. In this case, you have to tell the
exporter it is LaTeX code and don't expect it to find it out:

  @@latex:\pbox{Foo: \\${bar= (2^{3},1)}$, ${baz= (8^{4})}$}@@

This will be ignored in any exporter but latex (and beamer).

> The option `:no-export`, as expected, has no effect, since it only controls 
> whether `#_^&%` are escaped or not. 

This option doesn't exist anymore in Org 8.

> Is this a known feature, or a bug? And is there a known workaround?

Actually, this is consistent if you understand the limitations of Org.
As a rule of thumb, avoid using $..$ constructs and macros with
convoluted arguments (but "\hfill{}" and "\vspace{1cm}" are fine) when
writing raw LaTeX in an Org buffer.

For anything more complicated, use @@latex:...@@
or #+BEGIN_LATEX...#+END_LATEX (inline and non-inline version).

Again, all this assumes you are using Org 8.

HTH,


Regards,

-- 
Nicolas Goaziou

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Thibaut Verron]

[-- Attachment #1: Type: text/plain, Size: 3943 bytes --]

2014-06-26 12:38 GMT+02:00 Nicolas Goaziou <mail@nicolasgoaziou.fr>:

> Hello,
>
> Thibaut Verron <thibaut.verron@gmail.com> writes:
>
> > I'm forwarding this question asked on stackexchange:
> > http://tex.stackexchange.com/questions/186605/with-orgtbl-how-to-ensure-
> > that-braces-and-dollars-are-not-escaped
> >
> > After some investigation, it seems that the behavior is hidden deep in
> the
> > export routines, and I was wisely suggested to ask the question on this
> list
> > instead.
> >
> > I have given some tex-related details in the linked question, including
> some
> > motivations and an example, the tl;dr is that in some conditions, the
> > orgtbl-to-latex exporter will perform arbitrary escape of some
> characters in
> > the cells, or other kind of transformations:
> >
> >  $\text{test}$
> > is exported verbatim (OK).
> >
> > But
> >  \pbox{test}
> > becomes
> >  \pbox\{test\}
>
> You should upgrade to Org 8.
>
> >  {test}
> > becomes
> >  \{test\}
>
> This is intended.
>
> >  {$test$}
> > becomes
> >  \{\$test\$\}
>
> This is to be expected as matching single dollar math snippets is
> fragile. I suggest to use \(...\) instead: {\(test\)}
>
> > And the exporter seems to be trying to be smart, because it will still
> > ensure that the result is correct:
> >
> >  {$\infty$}
> > becomes
> >  \{\$$\infty$\$\}
>
> Ditto. Use \(...\). Or, better, {\infty} as \infty is an Org entity
> which will properly translated into LaTeX code.
>
> > The weirdest of all might be this one:
> >  \pbox{Foo: \\${bar= (2^{3},1)}$, ${baz= (8^{4})}$}
> > becomes
> >  \pbox\{Foo: \\${bar= (2^{3},1)}$, \$\{baz= (8$^{\text{4}}$)\}\$\}
>
> You cannot write raw LaTeX macros with complicated arguments (e.g.,
> containing braces) in an Org buffer. In this case, you have to tell the
> exporter it is LaTeX code and don't expect it to find it out:
>
>   @@latex:\pbox{Foo: \\${bar= (2^{3},1)}$, ${baz= (8^{4})}$}@@
>
> This will be ignored in any exporter but latex (and beamer).
>
> > The option `:no-export`, as expected, has no effect, since it only
> controls
> > whether `#_^&%` are escaped or not.
>
> This option doesn't exist anymore in Org 8.
>
> > Is this a known feature, or a bug? And is there a known workaround?
>
> Actually, this is consistent if you understand the limitations of Org.
> As a rule of thumb, avoid using $..$ constructs and macros with
> convoluted arguments (but "\hfill{}" and "\vspace{1cm}" are fine) when
> writing raw LaTeX in an Org buffer.
>
> For anything more complicated, use @@latex:...@@
> or #+BEGIN_LATEX...#+END_LATEX (inline and non-inline version).
>
> Again, all this assumes you are using Org 8.
>
> HTH,
>
>
> Regards,
>
> --
> Nicolas Goaziou
>

Thank you for your answer,

I forgot to repeat the information about my environment I gave on
stackexchange. In particular, this is Org 8.2.6.

Also, this is not an org buffer, but a latex buffer with the orgtbl minor
mode. Indeed I should have mentioned it in the body of the question, it's
easily missed in the title.

However, I understand that the underlying mechanism is the same as the one
used to export an org buffer to latex, and that it probably suffers from
the same limitations.
Now, are these "limitations of Org" really preventing it from exporting a
string verbatim? That would seem like the most logical default in this
situation, wouldn't it? (Disclaimer: I don't "understand the limitations of
Org", so these last questions may be ridiculous to someone who does)

Thanks,

Thibaut Verron

PS. Indeed \(..\) does work for the dollars, thank you for the tip. I
should have pointed out that \bgroup ... \egroup works as a replacement for
outermost pairs of braces too. However, I do not have any solution for the
\pbox{...} thing. And I would prefer a more robust solution which would not
require me to change the way I write the tables, because otherwise, I'd
still risk facing new unexportable constructs at random times.

[-- Attachment #2: Type: text/html, Size: 5390 bytes --]

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Thibaut Verron]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

>
> Now, are these "limitations of Org" really preventing it from exporting a
> string verbatim? That would seem like the most logical default in this
> situation, wouldn't it? (Disclaimer: I don't "understand the limitations of
> Org", so these last questions may be ridiculous to someone who does)
>
>
>
Apparently not, the following quick attempt seems to be doing the job fine
enough:

  (defun tv/orgtbl-to-latex-verbatim (table params)
    (flet ((org-export-string-as
            (string backend &optional b e)
            string))
     (orgtbl-to-latex table params)))

However, it is extra dirty, and ignoring so many parameters in a function
is probably not safe. :-)
Is there really no similar mechanism built-in org? Should I polish this
function and submit a patch? Or am I running into a wall?

Thanks,

Thibaut Verron

[-- Attachment #2: Type: text/html, Size: 1938 bytes --]

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Nicolas Goaziou]

Thibaut Verron <thibaut.verron@gmail.com> writes:

>> Now, are these "limitations of Org" really preventing it from exporting a
>> string verbatim? That would seem like the most logical default in this
>> situation, wouldn't it?

I disagree in the general case. The most logical default for Org is to
treat contents as plain text and ensure that the export conforms to what
appears in the buffer. As a bonus, it can leave LaTeX code as-is when it
recognizes some, which depends on Org's understanding of LaTeX syntax
(hence the limitations I'm talking about).

Now, in the context of a LaTeX buffer using orgtbl minor mode, it could
make sense in some situations to treat cell contents verbatim. I don't
think it should be the default, but there could be an option for that.
Anyway, there's a solution, see below.

> Apparently not, the following quick attempt seems to be doing the job fine
> enough:
>
>   (defun tv/orgtbl-to-latex-verbatim (table params)
>     (flet ((org-export-string-as
>             (string backend &optional b e)
>             string))
>      (orgtbl-to-latex table params)))
>
> However, it is extra dirty, and ignoring so many parameters in a function
> is probably not safe. :-)

I think defining your own translator function is the way to go. For
example, the following (untested) could work:

  (defun my-orgtbl-to-latex-verbatim (table params)
    (let* ((alignment (mapconcat (lambda (x) (if x "r" "l"))
  			       org-table-last-alignment ""))
  	 (params2
  	  (list
  	   :tstart (concat "\\begin{tabular}{" alignment "}")
  	   :tend "\\end{tabular}"
  	   :lstart "" :lend " \\\\" :sep " & "
  	   :efmt "%s\\,(%s)" :hline "\\hline")))
      (orgtbl-to-generic table (org-combine-plists params2 params))))


Regards,

-- 
Nicolas Goaziou

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Thibaut Verron]

[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]

2014-06-26 15:17 GMT+02:00 Nicolas Goaziou <mail@nicolasgoaziou.fr>:

> Thibaut Verron <thibaut.verron@gmail.com> writes:
>
> >> Now, are these "limitations of Org" really preventing it from exporting
> a
> >> string verbatim? That would seem like the most logical default in this
> >> situation, wouldn't it?
>
> I disagree in the general case. The most logical default for Org is to
> treat contents as plain text and ensure that the export conforms to what
> appears in the buffer. As a bonus, it can leave LaTeX code as-is when it
> recognizes some, which depends on Org's understanding of LaTeX syntax
> (hence the limitations I'm talking about).
>

Of course, I did not mean it for org-mode buffers! ;)


>
> Now, in the context of a LaTeX buffer using orgtbl minor mode, it could
> make sense in some situations to treat cell contents verbatim. I don't
> think it should be the default, but there could be an option for that.
> Anyway, there's a solution, see below.
>
> > Apparently not, the following quick attempt seems to be doing the job
> fine
> > enough:
> >
> >   (defun tv/orgtbl-to-latex-verbatim (table params)
> >     (flet ((org-export-string-as
> >             (string backend &optional b e)
> >             string))
> >      (orgtbl-to-latex table params)))
> >
> > However, it is extra dirty, and ignoring so many parameters in a function
> > is probably not safe. :-)
>
> I think defining your own translator function is the way to go. For
> example, the following (untested) could work:
>
>   (defun my-orgtbl-to-latex-verbatim (table params)
>     (let* ((alignment (mapconcat (lambda (x) (if x "r" "l"))
>                                org-table-last-alignment ""))
>          (params2
>           (list
>            :tstart (concat "\\begin{tabular}{" alignment "}")
>            :tend "\\end{tabular}"
>            :lstart "" :lend " \\\\" :sep " & "
>            :efmt "%s\\,(%s)" :hline "\\hline")))
>       (orgtbl-to-generic table (org-combine-plists params2 params))))
>
>
Indeed it does work.

But, unless I am mistaken, this is exactly the definition given here:
http://orgmode.org/manual/Translator-functions.html#Translator-functions
and so I was not wrong, this used to work as I expected.

I suspect that this change (regression?) will cause problems to a lot of
other users when they will upgrade their org to the current version.

Would changing the last lines of `orgtbl-to-latex` to something like this
work as a long-term solution?

    (require 'ox-latex)
    (let* ((*orgtbl-verbatim* (plist-get params :verbatim))
           (backend (if *orgtbl-verbatim* nil 'latex)))
      (orgtbl-to-generic table (org-combine-plists params2 params)
backend))))

Thanks for your time,

Thibaut verron

[-- Attachment #2: Type: text/html, Size: 4471 bytes --]

Re: Orgtbl-mode in latex: escaped braces and dollars, and other arbitrary transformations

[Nicolas Goaziou]

Thibaut Verron <thibaut.verron@gmail.com> writes:

> Would changing the last lines of `orgtbl-to-latex` to something like this
> work as a long-term solution?
>
>     (require 'ox-latex)
>     (let* ((*orgtbl-verbatim* (plist-get params :verbatim))
>            (backend (if *orgtbl-verbatim* nil 'latex)))
>       (orgtbl-to-generic table (org-combine-plists params2 params)
> backend))))

The check should happen in `orgtbl-to-generic', which is responsible for
translating cell contents.

Or, better, backend could become a parameter, e.g., :contents, and not
an optional argument anymore. Then one could override it with specific
params.

Also, docstrings and documentation should be updated accordingly.


Regards,

-- 
Nicolas Goaziou