Minor problems with dvipng latex image preview

Minor problems with dvipng latex image preview

[Nick Dokos]

The main problem is that the latex->dvi invocation is hard-wired in
org-create-formula-image-with-dvipng and in addition, when the latex
file is created, the value of org-latex-packages-alist is spliced in.

That in itself is fine, except in the case when (for normal latex
processing) I choose minted for code prettification. Following the
docstring of or-export-latex-listings, I set:

(setq org-export-latex-listings 'minted)
(add-to-list 'org-latex-packages-alist '("" "minted"))

in which case, I end up with a \usepackage{minted} in the preview
latex file. But minted requires that latex be invoked with
--shell-escape, which cannot be done because the latex->dvi invocation
is hardwired.

A customizable variable to hold such options could be used, but there
might be cleaner ways. For now, I've used the even dirtier approach of
hard-wiring the option into the latex->dvi call in the above function:

--8<---------------cut here---------------start------------->8---
            ...
	    (call-process "latex" nil nil nil "--shell-escape" texfile))
            ...
--8<---------------cut here---------------end--------------->8---

If a customizable variable is added, then the docstring for
org-export-latex-listings will need to be modified to mention it.

And finally, the same docstring also refers to org-latex.el and
org-latex-to-pdf-process, rather than using the 8.x names.

If you want to experiment, here is a small file:

--8<---------------cut here---------------start------------->8---
* Euler

\[
e^{i\pi} = -1
\]
--8<---------------cut here---------------end--------------->8---

Hit C-c C-x C-l for the preview, C-c C-c to go back.

Org-mode version 8.0.2 (release_8.0.2-116-gd54209)
GNU Emacs 24.3.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.10) of 2013-04-09

-- 
Nick

Re: Minor problems with dvipng latex image preview

[Nicolas Goaziou]

Hello,

Nick Dokos <ndokos@gmail.com> writes:

> The main problem is that the latex->dvi invocation is hard-wired in
> org-create-formula-image-with-dvipng and in addition, when the latex
> file is created, the value of org-latex-packages-alist is spliced in.
>
> That in itself is fine, except in the case when (for normal latex
> processing) I choose minted for code prettification. Following the
> docstring of or-export-latex-listings, I set:
>
> (setq org-export-latex-listings 'minted)

You mean `org-latex-listings'.  `org-export-latex-listings' belongs to
the old export framework (like almost all variables with
"org-export-BACKEND-" prefix).

> (add-to-list 'org-latex-packages-alist '("" "minted"))

> in which case, I end up with a \usepackage{minted} in the preview
> latex file. 

Use:

  (add-to-list 'org-latex-packages-alist '("" "minted" nil))

to tell Org not to include the package for previewing snippets.


Regards,

-- 
Nicolas Goaziou

Re: Minor problems with dvipng latex image preview

[Nick Dokos]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

>> ... I set:
>>
>> (setq org-export-latex-listings 'minted)
>
> You mean `org-latex-listings'.  `org-export-latex-listings' belongs to
> the old export framework (like almost all variables with
> "org-export-BACKEND-" prefix).
>

Yes, indeed.

>> (add-to-list 'org-latex-packages-alist '("" "minted"))
>
>> in which case, I end up with a \usepackage{minted} in the preview
>> latex file. 
>
> Use:
>
>   (add-to-list 'org-latex-packages-alist '("" "minted" nil))
>
> to tell Org not to include the package for previewing snippets.
>

OK, that works - I didn't know about the three-element list
form. Thanks!

Perhaps the docstring for org-latex-listings should include
the three-element list form, with a pointer to the
org-latex-packages-alist doc for more details.

There is also a (perhaps unlikely) scenario where this is not enough:
previewing typeset code where I *want* to use minted:

--8<---------------cut here---------------start------------->8---
* Code

\begin{minted}{c}
  printf("Hello world\n");
\end{minted}
--8<---------------cut here---------------end--------------->8---

-- 
Nick

Re: Minor problems with dvipng latex image preview

[Nicolas Goaziou]

Nick Dokos <ndokos@gmail.com> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>>> ... I set:
>>>
>>> (setq org-export-latex-listings 'minted)
>>
>> You mean `org-latex-listings'.  `org-export-latex-listings' belongs to
>> the old export framework (like almost all variables with
>> "org-export-BACKEND-" prefix).
>>
>
> Yes, indeed.
>
>>> (add-to-list 'org-latex-packages-alist '("" "minted"))
>>
>>> in which case, I end up with a \usepackage{minted} in the preview
>>> latex file. 
>>
>> Use:
>>
>>   (add-to-list 'org-latex-packages-alist '("" "minted" nil))
>>
>> to tell Org not to include the package for previewing snippets.
>>
>
> OK, that works - I didn't know about the three-element list
> form. Thanks!

The surprising part of that third element is that it is assumed to be
non-nil when missing (see `org-latex-packages-to-string').

> Perhaps the docstring for org-latex-listings should include
> the three-element list form, with a pointer to the
> org-latex-packages-alist doc for more details.

The docstring already contains two references to
`org-latex-packages-alist'.  Wouldn't suggesting to insert

  (add-to-list 'org-latex-packages-alist '("" "minted" nil))

be confusing, since we don't provide a third element for "listings" and
"color" packages? Well, unless we provide the element for the three of
them (t for the first two, and nil for the last).

> There is also a (perhaps unlikely) scenario where this is not enough:
> previewing typeset code where I *want* to use minted:
>
> * Code
>
> \begin{minted}{c} printf("Hello world\n"); \end{minted}

In that case, I suggest to use `imagemagick' for the conversion, since
it relies on `org-latex-pdf-process' value (and is therefore
customizable).


Regards,

-- 
Nicolas Goaziou

Re: Minor problems with dvipng latex image preview

[Nick Dokos]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

>> OK, that works - I didn't know about the three-element list
>> form. Thanks!
>
> The surprising part of that third element is that it is assumed to be
> non-nil when missing (see `org-latex-packages-to-string').
>

Yes, presumably in the name of backward compatibility and "least
surprise": if one uses the two-element form, one gets the package
included in both export and previews, which is probably what is
wanted in general (although minted is something of an exception).


>> Perhaps the docstring for org-latex-listings should include
>> the three-element list form, with a pointer to the
>> org-latex-packages-alist doc for more details.
>
> The docstring already contains two references to
> `org-latex-packages-alist'.  Wouldn't suggesting to insert
>
>   (add-to-list 'org-latex-packages-alist '("" "minted" nil))
>
> be confusing, since we don't provide a third element for "listings" and
> "color" packages? Well, unless we provide the element for the three of
> them (t for the first two, and nil for the last).
>

Yes, it's not particularly easy to explain. But if one copies the code
from the docstring verbatim, one can slam into the problem and it is not
easy to debug.

>> There is also a (perhaps unlikely) scenario where this is not enough:
>> previewing typeset code where I *want* to use minted:
>>
>> * Code
>>
>> \begin{minted}{c} printf("Hello world\n"); \end{minted}
>
> In that case, I suggest to use `imagemagick' for the conversion, since
> it relies on `org-latex-pdf-process' value (and is therefore
> customizable).
>

I learnt quite a bit from this discussion (thank you!), but I'm still a
bit puzzled about your reluctance that custom options be added to the
latex call. Why is that? Too many customizations? dvipng should be
deprecated?  Too many twisty passages to explain? 

BTW, I found myself wishing for some debugging aid along the following
lines: an option to keep the .tex file produced (it *is* kept in case of
error, but sometimes it would be nice to look at it even if there is no
error), and a message in *Messages* with the complete command that
call-process is executing: that way, one can easily execute the command
by hand. One can always use the debugger for this, but that feels like
the proverbial elephant gun in search of a fly.

-- 
Nick

Re: Minor problems with dvipng latex image preview

[Nicolas Goaziou]

Nick Dokos <ndokos@gmail.com> writes:

> I learnt quite a bit from this discussion (thank you!), but I'm still a
> bit puzzled about your reluctance that custom options be added to the
> latex call. Why is that? Too many customizations? dvipng should be
> deprecated?  Too many twisty passages to explain?

Huh? I'm not reluctant to anything with regards to this discussion. I'm
just suggesting solutions to your problem.

Anyway, in a nutshell, your proposal is to:

  - add a custom variable, e.g., `org-latex-dvi-process-options' (which
    library should it belong to?)
  - modify `org-latex-listings' docstring (in particular, add third
    elements and new custom variable)
  - add a custom variable, e.g., `org-latex-dvi-process-debug', which,
    when non-nil asks to leave produced "tex" file.

Is that right?


Regards,

-- 
Nicolas Goaziou

Re: Minor problems with dvipng latex image preview

[Nick Dokos]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Nick Dokos <ndokos@gmail.com> writes:
>
>> I learnt quite a bit from this discussion (thank you!), but I'm still a
>> bit puzzled about your reluctance that custom options be added to the
>> latex call. Why is that? Too many customizations? dvipng should be
>> deprecated?  Too many twisty passages to explain?
>
> Huh? I'm not reluctant to anything with regards to this discussion. I'm
> just suggesting solutions to your problem.
>

Ah, OK - sorry I misunderstood.

> Anyway, in a nutshell, your proposal is to:
>
>   - add a custom variable, e.g., `org-latex-dvi-process-options' (which
>     library should it belong to?)

Unless it would make sense to toss the whole dvipng thing overboard and
just keep imagemagick.

>   - modify `org-latex-listings' docstring (in particular, add third
>     elements and new custom variable)

I'm not sure any more that it can all be explained clearly in the
docstring (at least I've been trying different mental gyrations and I
have not come up with anything satisfactory). So maybe the thing to do
is add a page to worg and a pointer to it in the docstring. If that's
acceptable, I volunteer to write the worg page (at least the initial
version).

As a separate issue, I proposed some debugging aids:

>   - add a custom variable, e.g., `org-latex-dvi-process-debug', which,
>     when non-nil asks to leave produced "tex" file.
>

... and a call-process-log function that logs the command in *Messages*
before executing it with call-process (or something more or less
equivalent). It would be used wherever call-process is used now.

I would actually propose that the debug variable inhibit the deletion
of intermediate files everywhere, not just in latex preview.

-- 
Nick

Re: Minor problems with dvipng latex image preview

[Nicolas Goaziou]

Nick Dokos <ndokos@gmail.com> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:

>> Anyway, in a nutshell, your proposal is to:
>>
>>   - add a custom variable, e.g., `org-latex-dvi-process-options' (which
>>     library should it belong to?)
>
> Unless it would make sense to toss the whole dvipng thing overboard and
> just keep imagemagick.

I'm not sure there is a really good reason to drop it. Is it inferior in
some way?

Also, imagemagick is not optimal either. Since it uses
`org-latex-pdf-process', "pdflatex" is called three times by default,
which is unnecessary for a short snippet.

So, both dvipng and imagemagick should have a variable to set the
program to call, along with its arguments.

>>   - modify `org-latex-listings' docstring (in particular, add third
>>     elements and new custom variable)
>
> I'm not sure any more that it can all be explained clearly in the
> docstring (at least I've been trying different mental gyrations and I
> have not come up with anything satisfactory). So maybe the thing to do
> is add a page to worg and a pointer to it in the docstring. If that's
> acceptable, I volunteer to write the worg page (at least the initial
> version).

A worg page can't hurt, but a URL in the docstring is not very handy
either. It also defeats the "self-documenting" part of Emacs.

> As a separate issue, I proposed some debugging aids:
>
>>   - add a custom variable, e.g., `org-latex-dvi-process-debug', which,
>>     when non-nil asks to leave produced "tex" file.
>>
>
> ... and a call-process-log function that logs the command in *Messages*
> before executing it with call-process (or something more or less
> equivalent). It would be used wherever call-process is used now.
>
> I would actually propose that the debug variable inhibit the deletion
> of intermediate files everywhere, not just in latex preview.

Some parts of Org already have their own debug variable (see
`org-export-async-debug'). A meta debug variable would not be optimal,
would it?


Regards,

-- 
Nicolas Goaziou

Re: Minor problems with dvipng latex image preview

[Nick Dokos]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Nick Dokos <ndokos@gmail.com> writes:
>
>> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>>> Anyway, in a nutshell, your proposal is to:
>>>
>>>   - add a custom variable, e.g., `org-latex-dvi-process-options' (which
>>>     library should it belong to?)
>>
>> Unless it would make sense to toss the whole dvipng thing overboard and
>> just keep imagemagick.
>
> I'm not sure there is a really good reason to drop it. Is it inferior in
> some way?
>

I wouldn't say inferior, although the dvipng implementation is slightly
more brittle than the imagemagick one (imo, of course). But it is two
implementations where one would suffice (Windows might present problems
however: I don't know the availability of dvipng/imagemagick on that
platform - I believe they are both available on MacOS, but I could be
wrong).

The reason I was using dvipng is that the dvipng preview method predated
the imagemagick method so that's where I started and that's where I
stayed, but it was trivial to switch to imagemagick for experimentation
and I may stick with it in the end. And the only reason I can see for
keeping both is so that nobody's workflow gets broken.

> Also, imagemagick is not optimal either. Since it uses
> `org-latex-pdf-process', "pdflatex" is called three times by default,
> which is unnecessary for a short snippet.
>

Agreed. I switched to texi2dvi a long time ago, even patching it when
its infamous bug was still around. By this time, it might even be
possible to actually make it the default (although Windows might present
problems again). But given the history, it doesn't give one the warm
fuzzies either.

> So, both dvipng and imagemagick should have a variable to set the
> program to call, along with its arguments.
>
>>>   - modify `org-latex-listings' docstring (in particular, add third
>>>     elements and new custom variable)
>>
>> I'm not sure any more that it can all be explained clearly in the
>> docstring (at least I've been trying different mental gyrations and I
>> have not come up with anything satisfactory). So maybe the thing to do
>> is add a page to worg and a pointer to it in the docstring. If that's
>> acceptable, I volunteer to write the worg page (at least the initial
>> version).
>
> A worg page can't hurt, but a URL in the docstring is not very handy
> either. It also defeats the "self-documenting" part of Emacs.
>

Agreed again (about the self-documenting part: not sure about the URL
not being very handy), but some things are just too fiddly to fit into a
few sentences. I have now written a document of 335 lines (still not
done and only covering the options available today, but I am trying to
provide enough context to make it stand on its own): I may be suffering
from Pascal's syndrome ("Forgive the length of this letter; I did not
have the time to make it shorter") and I do tend to be verbose in
explanations (as some people on this list can probably testify), but I'm
not sure I can shorten it by much, even if I suddenly turn into
Hemingway - yeah, I wish :-)

In any case, a note like "If you get into trouble or you want to know
more, see FOO for the gory details" does not seem too bad to me.

>> As a separate issue, I proposed some debugging aids:
>>
>>>   - add a custom variable, e.g., `org-latex-dvi-process-debug', which,
>>>     when non-nil asks to leave produced "tex" file.
>>>
>>
>> ... and a call-process-log function that logs the command in *Messages*
>> before executing it with call-process (or something more or less
>> equivalent). It would be used wherever call-process is used now.
>>
>> I would actually propose that the debug variable inhibit the deletion
>> of intermediate files everywhere, not just in latex preview.
>
> Some parts of Org already have their own debug variable (see
> `org-export-async-debug'). A meta debug variable would not be optimal,
> would it?
>

Optimal in what sense? Also, I'm not sure what you mean by a "meta
debug" variable. I was thinking of a more global debug variable (it
would e.g. subsume the role of org-export-async-debug), but you
are right that it's more complicated than that: e.g. there is the
question of what all the intermediate files are and where they are
located.

That's why I wanted a log message in *Messages*: I could then say "keep
all intermediate files" by turning on the variable, carry out the
operation and then look in *Messages* to find out where those files are
- no mucking around through the sources. What I do now is find the
function that produces the file(s), and either edebug it, break at (or
just after) the call-process call site and evaluate the variables to
figure out where everything is, then go look at them; or add a (debug)
call just after and otherwise proceed the same way.

It's not too bad, but I like systems that can tell me what they are
doing, so if the need arises, I can easily figure out what went wrong.

-- 
Nick

Re: Minor problems with dvipng latex image preview

[Nicolas Goaziou]

Hello,

Nick Dokos <ndokos@gmail.com> writes:

> I wouldn't say inferior, although the dvipng implementation is slightly
> more brittle than the imagemagick one (imo, of course). But it is two
> implementations where one would suffice (Windows might present problems
> however: I don't know the availability of dvipng/imagemagick on that
> platform - I believe they are both available on MacOS, but I could be
> wrong).

What about asking, in a fresh thread, the users about it, then? FWIW,
I'm all for anything related to spring cleaning.

>> Also, imagemagick is not optimal either. Since it uses
>> `org-latex-pdf-process', "pdflatex" is called three times by default,
>> which is unnecessary for a short snippet.

> Agreed again (about the self-documenting part: not sure about the URL
> not being very handy), but some things are just too fiddly to fit into a
> few sentences. I have now written a document of 335 lines (still not
> done and only covering the options available today, but I am trying to
> provide enough context to make it stand on its own): I may be suffering
> from Pascal's syndrome ("Forgive the length of this letter; I did not
> have the time to make it shorter") and I do tend to be verbose in
> explanations (as some people on this list can probably testify), but I'm
> not sure I can shorten it by much, even if I suddenly turn into
> Hemingway - yeah, I wish :-)
>
> In any case, a note like "If you get into trouble or you want to know
> more, see FOO for the gory details" does not seem too bad to me.

So be it. Tell me when the worg page is ready, I'll update the docstring
accordingly.

> Optimal in what sense? Also, I'm not sure what you mean by a "meta
> debug" variable. I was thinking of a more global debug variable (it
> would e.g. subsume the role of org-export-async-debug), but you
> are right that it's more complicated than that: e.g. there is the
> question of what all the intermediate files are and where they are
> located.

"debug" can cover many different cases, and a "meta", i.e. global,
variable would have to explain all of them in its docstring. In the end,
the net gain for adding such a variable is not clear (besides
discoverability, but is it important for this kind of variable?).

> That's why I wanted a log message in *Messages*: I could then say "keep
> all intermediate files" by turning on the variable, carry out the
> operation and then look in *Messages* to find out where those files are
> - no mucking around through the sources. What I do now is find the
> function that produces the file(s), and either edebug it, break at (or
> just after) the call-process call site and evaluate the variables to
> figure out where everything is, then go look at them; or add a (debug)
> call just after and otherwise proceed the same way.
>
> It's not too bad, but I like systems that can tell me what they are
> doing, so if the need arises, I can easily figure out what went wrong.

Improvements to the debug system need to be discussed thoroughly in
order to set up complete specifications (I don't think it's just about
latex snippets).

This cannot happen as a side note in a thread between you and me. If you
think it is important enough, please initiate the process in a new
thread.


Regards,

-- 
Nicolas Goaziou

Re: Minor problems with dvipng latex image preview

[Nick Dokos]

Nicolas Goaziou <n.goaziou@gmail.com> writes:


>> But it is two implementations where one would suffice (Windows might
>> present problems however: I don't know the availability of
>> dvipng/imagemagick on that platform - I believe they are both
>> available on MacOS, but I could be wrong).
>
> What about asking, in a fresh thread, the users about it, then? FWIW,
> I'm all for anything related to spring cleaning.
>

OK, will do.

> So be it. Tell me when the worg page is ready, I'll update the docstring
> accordingly.
>

OK, will do.

> Improvements to the debug system need to be discussed thoroughly in
> order to set up complete specifications (I don't think it's just about
> latex snippets).
>
> This cannot happen as a side note in a thread between you and me. If you
> think it is important enough, please initiate the process in a new
> thread.

OK, will do.

Thanks!
-- 
Nick (AKA "broken record")