babel latex headers and image generation commands

babel latex headers and image generation commands

[Matt Huszagh]

I spent some time today trying to get latex babel source blocks to work
for me and discovered that calling `org-babel-execute:latex` ignores the
:headers header if the output file is a png without setting imagemagick
to t. It's easy to see this in the source code: the conditions mentioned
above leads to calling org-create-formula-image without passing in the
headers. I think this is a bug, although maybe I missed it somewhere in
the documentation? Here's a MWE if you want it

#+header: :file "test.png"
#+header: :headers '("\\def\\hello{hello}")
#+begin_src latex :results output file link
\hello
#+end_src

doesn't work, but

#+header: :file "test.png"
#+header: :imagemagick t
#+header: :headers '("\\def\\hello{hello}")
#+begin_src latex :results output file link
\hello
#+end_src

does.

However, this got me thinking that I wish executing latex blocks behaved
a bit more like latex fragment previews. Particularly, the ability to
customize `image-converter`. So, I'm thinking about adding a
customization option that allows a user to use the image-converter
portion of an existing org-preview-latex-process-alist entry (I guess
the most obvious choice would be
org-preview-latex-default-process). Although I guess I could just add a
new variable like what we have with org-latex-pdf-process. Maybe that's
better since we'd only be using one part of the
latex-process-alist. What are people's thoughts on this?

Re: babel latex headers and image generation commands

[Matt Huszagh]

Ok, here's an implementation that seems to be working pretty well so
far.

`org-latex-img-process` is the new customization. Most of the execute
function is unaltered, but I've added the condition:
```
             ((and (not imagemagick)
                   (assoc extension org-latex-img-process))
```

Here's the change in full.

```
    (setq org-latex-pdf-process
          '("latexmk -f -interaction=nonstopmode -output-directory=%o %f"))

    (setq org-latex-img-process
          '(("svg" . ("dvisvgm %f -P -n -b min -o %O"))))

    (defun org-babel-execute:latex (body params)
      "Execute a block of Latex code with Babel.
This function is called by `org-babel-execute-src-block'."
      (setq body (org-babel-expand-body:latex body params))
      (if (cdr (assq :file params))
          (let* ((out-file (cdr (assq :file params)))
	         (extension (file-name-extension out-file))
	         (tex-file (org-babel-temp-file "latex-" ".tex"))
	         (border (cdr (assq :border params)))
	         (imagemagick (cdr (assq :imagemagick params)))
	         (im-in-options (cdr (assq :iminoptions params)))
	         (im-out-options (cdr (assq :imoutoptions params)))
	         (fit (or (cdr (assq :fit params)) border))
	         (height (and fit (cdr (assq :pdfheight params))))
	         (width (and fit (cdr (assq :pdfwidth params))))
	         (headers (cdr (assq :headers params)))
	         (in-buffer (not (string= "no" (cdr (assq :buffer params)))))
	         (org-latex-packages-alist
	          (append (cdr (assq :packages params)) org-latex-packages-alist)))
            (cond
             ((and (string-suffix-p ".png" out-file) (not imagemagick))
              (org-create-formula-image
               body out-file org-format-latex-options in-buffer))
             ((string-suffix-p ".tikz" out-file)
	      (when (file-exists-p out-file) (delete-file out-file))
	      (with-temp-file out-file
	        (insert body)))
             ((and (not imagemagick)
                   (assoc extension org-latex-img-process))
              (with-temp-file tex-file
                (insert (concat
                         (org-latex-make-preamble
	                  (org-export-get-environment (org-export-get-backend 'latex))
	                  org-format-latex-header
	                  'snippet)
                         (if headers
			     (concat "\n"
				     (if (listp headers)
				         (mapconcat #'identity headers "\n")
				       headers) "\n")
		           "")
                         "\\begin{document}"
		         body
		         "\\end{document}")))
              (let ((tmp-pdf (org-babel-latex-tex-to-pdf tex-file)))
                (when (file-exists-p out-file) (delete-file out-file))
                (let* ((log-buf (get-buffer-create "*Org Babel LaTeX Output*"))
                       (err-msg "fix")
                       (img-out (org-compile-file
	                         tmp-pdf
                                 (cdr (assoc "svg" org-latex-img-process))
                                 extension err-msg log-buf)))
                  (shell-command (format "mv %s %s" img-out out-file)))))
	     ((and (or (string= "svg" extension)
		       (string= "html" extension))
	           (executable-find org-babel-latex-htlatex))
	      ;; TODO: this is a very different way of generating the
	      ;; frame latex document than in the pdf case.  Ideally, both
	      ;; would be unified.  This would prevent bugs creeping in
	      ;; such as the one fixed on Aug 16 2014 whereby :headers was
	      ;; not included in the SVG/HTML case.
	      (with-temp-file tex-file
	        (insert (concat
		         "\\documentclass[preview]{standalone}
\\def\\pgfsysdriver{pgfsys-tex4ht.def}
"
		         (mapconcat (lambda (pkg)
				      (concat "\\usepackage" pkg))
				    org-babel-latex-htlatex-packages
				    "\n")
		         (if headers
			     (concat "\n"
				     (if (listp headers)
				         (mapconcat #'identity headers "\n")
				       headers) "\n")
		           "")
		         "\\begin{document}"
		         body
		         "\\end{document}")))
	      (when (file-exists-p out-file) (delete-file out-file))
	      (let ((default-directory (file-name-directory tex-file)))
	        (shell-command (format "%s %s" org-babel-latex-htlatex tex-file)))
	      (cond
	       ((file-exists-p (concat (file-name-sans-extension tex-file) "-1.svg"))
	        (if (string-suffix-p ".svg" out-file)
		    (progn
		      (shell-command "pwd")
		      (shell-command (format "mv %s %s"
					     (concat (file-name-sans-extension tex-file) "-1.svg")
					     out-file)))
	          (error "SVG file produced but HTML file requested")))
	       ((file-exists-p (concat (file-name-sans-extension tex-file) ".html"))
	        (if (string-suffix-p ".html" out-file)
		    (shell-command "mv %s %s"
			           (concat (file-name-sans-extension tex-file)
				           ".html")
			           out-file)
	          (error "HTML file produced but SVG file requested")))))
	     ((or (string= "pdf" extension) imagemagick)
	      (with-temp-file tex-file
	        (require 'ox-latex)
	        (insert
	         (org-latex-guess-inputenc
	          (org-splice-latex-header
	           org-format-latex-header
	           (delq
		    nil
		    (mapcar
		     (lambda (el)
		       (unless (and (listp el) (string= "hyperref" (cadr el)))
		         el))
		     org-latex-default-packages-alist))
	           org-latex-packages-alist
	           nil))
	         (if fit "\n\\usepackage[active, tightpage]{preview}\n" "")
	         (if border (format "\\setlength{\\PreviewBorder}{%s}" border) "")
	         (if height (concat "\n" (format "\\pdfpageheight %s" height)) "")
	         (if width  (concat "\n" (format "\\pdfpagewidth %s" width))   "")
	         (if headers
		     (concat "\n"
			     (if (listp headers)
			         (mapconcat #'identity headers "\n")
			       headers) "\n")
	           "")
	         (if fit
		     (concat "\n\\begin{document}\n\\begin{preview}\n" body
			     "\n\\end{preview}\n\\end{document}\n")
	           (concat "\n\\begin{document}\n" body "\n\\end{document}\n"))))
              (when (file-exists-p out-file) (delete-file out-file))
	      (let ((transient-pdf-file (org-babel-latex-tex-to-pdf tex-file)))
	        (cond
	         ((string= "pdf" extension)
	          (rename-file transient-pdf-file out-file))
	         (imagemagick
	          (org-babel-latex-convert-pdf
	           transient-pdf-file out-file im-in-options im-out-options)
	          (when (file-exists-p transient-pdf-file)
		    (delete-file transient-pdf-file)))
	         (t
	          (error "Can not create %s files, please specify a .png or .pdf file or try the :imagemagick header argument"
		         extension))))))
            nil) ;; signal that output has already been written to file
        body))
```

What do people think? This gives you much more control over how the
latex is compiled and reuses existing functionality for previews.

Re: babel latex headers and image generation commands

[Matt Huszagh]

I've thought about this more and the solution I presented above isn't
quite sufficient for me. I need something where I get complete control
over what goes in the latex source block on a block-by-block basis. In
other words, I don't want a user-configurable option like
org-format-latex-header plus a list of user-configurable packages in
every block.

I could add an option alongside the option I introduced above that tells
the execute function to only compose the tex file from the body (btw,
this doesn't have to be manual for the user, just define a class and use
snippets), but at some point I wonder how much it makes sense to keep
adding options to this function since it'll just make it harder to
maintain.

What do people think? If there's any interest, I'm more than happy to
put in the extra time and add this functionality to
latex-execute. Otherwise, I'll just advise the function for my own
specific needs.

I do think there's a real use-case here: namely, complete case-by-case
control over the source latex and its output, including export backend
dependent things like colors. But, maybe this is just me and the
existing functionality is fine with people.

Matt

Re: babel latex headers and image generation commands

[Bastien]

Hi Matt,

Matt Huszagh <huszaghmatt@gmail.com> writes:

> What do people think? If there's any interest, I'm more than happy to
> put in the extra time and add this functionality to
> latex-execute.

If you find the time to share your change as a _patch_ (not the whole
updated Elisp function), that will allow more readers on this list to
quickly understand what is at stake.

Thanks!

-- 
 Bastien

Re: babel latex headers and image generation commands

[Matt Huszagh]

[-- Attachment #1.1: Type: text/plain, Size: 9587 bytes --]

Bastien <bzg@gnu.org> writes:

> If you find the time to share your change as a _patch_ (not the whole
> updated Elisp function), that will allow more readers on this list to
> quickly understand what is at stake.

Ok, I've finally gotten around to taking a crack at this. The patch is
attached. Basically, it allows a lot more control when converting a
latex source block into an svg image file.

Specifically, this new method does not place any restrictions on the
latex contents (the old svg generation required the use of
\documentclass[preview]{standalone}), or on the specific generation
commands (it does require first compiling to pdf then to svg, though I
guess this could be generalized if necessary). Additionally, it allows
you to use a setup independent of the setup used for inline latex
snippets. I think this is important because snippets and latex source
blocks have different use cases. For instance, I use snippets for simple
inline math (e.g. \(x=6\)) and source blocks for complicated full-blown
latex pictures and sets of equations, etc. that are not supported by
latex fragments.

I'll present my use case as an example (see the patch to understand the
customizations).

;; this is particular to my use case, see `latex-preamble-by-backend'
(defun by-backend (blist)
  (let ((ret nil))
    (if org-export-current-backend
        (let* ((backend-name org-export-current-backend)
               (elem (assoc backend-name blist)))
          (if elem
              (setq ret (cdr elem))))
      (let ((elem (assoc t blist)))
        (setq ret (cdr elem))))
    (eval ret)))

;; custom function for preamble generation
(defun latex-preamble-by-backend (params)
  (concat "\\documentclass{"
          (cdr (assoc :class params))
          "}"
          "\\definecolor{fg}{rgb}{"
          (by-backend '((html . "0,0,0")
                        (t . (org-latex-color :foreground))))
          "}\n"
          "\\definecolor{bg}{rgb}{"
          (by-backend '((html . "1,1,1")
                        (t . (org-latex-color :background))))
          "}\n"
          "\\def\\pc{"
          (by-backend '((html . "100")
                        (t . "20")))
          "}\n"))

;; actual set customization
(setq org-babel-latex-preamble
      (lambda (params)
        (latex-preamble-by-backend params)))

Now if I define a source block:

#+header: :class math :results file link replace :file tmp/some-file.svg
#+begin_src latex :hidden
{\color{fg}
\begin{equation*}
|z| = \sqrt{x^2 + y^2}
\end{equation*}
}
#+end_src

Using my customization above, this will turn into the following latex

\documentclass{math}\definecolor{fg}{rgb}{0.819608,0.721569,0.592157}
\definecolor{bg}{rgb}{0.0235294,0.137255,0.160784}
\def\pc{20}
\begin{document}{\color{fg}
\begin{equation*}
|z| = \sqrt{x^2 + y^2}
\end{equation*}
}\end{document}

which is then turned into an svg with inkscape, though this could be
dvisvgm, or whatever you want.

Math is a custom document class I've defined. But my function also
allows the setting of other classes (which is useful because some
classes work better for tikz-like images and others work better for
math). In fact, since the preamble generation function can use any of
the source block parameters, I could make it do a bunch of other
conditional stuff. I've also configured it to set my color scheme in a
backend-dependent way. This allows me to get color schemes that match my
emacs theme when generating images for viewing within emacs and another
color scheme (in my case black foreground, white background) when
exporting to html. Note that none of this is enforced on other
people. My patch simply allows you to achieve this sort of flexibility.

I think this patch needs some discussing. I had some trouble deciding on
the best way to implement this functionality because its hard to get
anything to be thematically consistent with the current latex execute
command, which feels like a collection of loosely-related functionality
(for instance, the htlatex backend generates a completely different
latex source than the imagemagick backend, even though the outputs could
be quite similar -- svg and png). I think there's a lot of room for
improvement in the latex execute function that could make it more
general and flexible, but it's hard to do this in a way that doesn't
break existing workflows. My patch tries to be minimally invasive to
these workflows, though it will break workflows in an easily recoverable
way for anyone using htlatex for svg output.

Anyway, I'd be curious to hear thoughts and I'd be interested to discuss
options for further refactoring the latex execute function.

Matt

On Fri, Aug 28, 2020 at 11:10 PM Matt Huszagh <huszaghmatt@gmail.com> wrote:

> Bastien <bzg@gnu.org> writes:
>
> > If you find the time to share your change as a _patch_ (not the whole
> > updated Elisp function), that will allow more readers on this list to
> > quickly understand what is at stake.
>
> Ok, I've finally gotten around to taking a crack at this. The patch is
> attached. Basically, it allows a lot more control when converting a
> latex source block into an svg image file.
>
> Specifically, this new method does not place any restrictions on the
> latex contents (the old svg generation required the use of
> \documentclass[preview]{standalone}), or on the specific generation
> commands (it does require first compiling to pdf then to svg, though I
> guess this could be generalized if necessary). Additionally, it allows
> you to use a setup independent of the setup used for inline latex
> snippets. I think this is important because snippets and latex source
> blocks have different use cases. For instance, I use snippets for simple
> inline math (e.g. \(x=6\)) and source blocks for complicated full-blown
> latex pictures and sets of equations, etc. that are not supported by
> latex fragments.
>
> I'll present my use case as an example (see the patch to understand the
> customizations).
>
> ;; this is particular to my use case, see `latex-preamble-by-backend'
> (defun by-backend (blist)
>   (let ((ret nil))
>     (if org-export-current-backend
>         (let* ((backend-name org-export-current-backend)
>                (elem (assoc backend-name blist)))
>           (if elem
>               (setq ret (cdr elem))))
>       (let ((elem (assoc t blist)))
>         (setq ret (cdr elem))))
>     (eval ret)))
>
> ;; custom function for preamble generation
> (defun latex-preamble-by-backend (params)
>   (concat "\\documentclass{"
>           (cdr (assoc :class params))
>           "}"
>           "\\definecolor{fg}{rgb}{"
>           (by-backend '((html . "0,0,0")
>                         (t . (org-latex-color :foreground))))
>           "}\n"
>           "\\definecolor{bg}{rgb}{"
>           (by-backend '((html . "1,1,1")
>                         (t . (org-latex-color :background))))
>           "}\n"
>           "\\def\\pc{"
>           (by-backend '((html . "100")
>                         (t . "20")))
>           "}\n"))
>
> ;; actual set customization
> (setq org-babel-latex-preamble
>       (lambda (params)
>         (latex-preamble-by-backend params)))
>
> Now if I define a source block:
>
> #+header: :class math :results file link replace :file tmp/some-file.svg
> #+begin_src latex :hidden
> {\color{fg}
> \begin{equation*}
> |z| = \sqrt{x^2 + y^2}
> \end{equation*}
> }
> #+end_src
>
> Using my customization above, this will turn into the following latex
>
> \documentclass{math}\definecolor{fg}{rgb}{0.819608,0.721569,0.592157}
> \definecolor{bg}{rgb}{0.0235294,0.137255,0.160784}
> \def\pc{20}
> \begin{document}{\color{fg}
> \begin{equation*}
> |z| = \sqrt{x^2 + y^2}
> \end{equation*}
> }\end{document}
>
> which is then turned into an svg with inkscape, though this could be
> dvisvgm, or whatever you want.
>
> Math is a custom document class I've defined. But my function also
> allows the setting of other classes (which is useful because some
> classes work better for tikz-like images and others work better for
> math). In fact, since the preamble generation function can use any of
> the source block parameters, I could make it do a bunch of other
> conditional stuff. I've also configured it to set my color scheme in a
> backend-dependent way. This allows me to get color schemes that match my
> emacs theme when generating images for viewing within emacs and another
> color scheme (in my case black foreground, white background) when
> exporting to html. Note that none of this is enforced on other
> people. My patch simply allows you to achieve this sort of flexibility.
>
> I think this patch needs some discussing. I had some trouble deciding on
> the best way to implement this functionality because its hard to get
> anything to be thematically consistent with the current latex execute
> command, which feels like a collection of loosely-related functionality
> (for instance, the htlatex backend generates a completely different
> latex source than the imagemagick backend, even though the outputs could
> be quite similar -- svg and png). I think there's a lot of room for
> improvement in the latex execute function that could make it more
> general and flexible, but it's hard to do this in a way that doesn't
> break existing workflows. My patch tries to be minimally invasive to
> these workflows, though it will break workflows in an easily recoverable
> way for anyone using htlatex for svg output.
>
> Anyway, I'd be curious to hear thoughts and I'd be interested to discuss
> options for further refactoring the latex execute function.
>
> Matt
>
>

[-- Attachment #1.2: Type: text/html, Size: 11594 bytes --]

[-- Attachment #2: 0001-ob-latex.el-Make-latex-to-svg-compilation-very-custo.patch --]
[-- Type: application/octet-stream, Size: 2876 bytes --]

From 0834f59ac9485158c26c4b25ec40df56be2c15ca Mon Sep 17 00:00:00 2001
From: Matt Huszagh <huszaghmatt@gmail.com>
Date: Fri, 28 Aug 2020 22:26:05 -0700
Subject: [PATCH] ob-latex.el: Make latex to svg compilation very customizable

* lisp/ob-latex.el (org-babel-latex-preamble): Add latex preamble
customization.
(org-babel-latex-pdf-svg-process): Add customization for converting a
pdf to svg.
(org-babel-execute:latex): Add specific case for svg generation from
latex block.
---
 lisp/ob-latex.el | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-latex.el b/lisp/ob-latex.el
index 4b343dd14..ba7bb8277 100644
--- a/lisp/ob-latex.el
+++ b/lisp/ob-latex.el
@@ -70,6 +70,23 @@
   :group 'org-babel
   :type 'string)
 
+(defcustom org-babel-latex-preamble
+  (lambda (_)
+    "\\documentclass[preview]{standalone}
+\\def\\pgfsysdriver{pgfsys-tex4ht.def}
+")
+  "Closure which evaluates at runtime to the latex preamble.  It
+takes 1 argument which is the parameters of the source block."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-pdf-svg-process
+  "inkscape --pdf-poppler %f -T -l -o %O"
+  "Command used to convert a PDF file to an SVG file when
+executing a latex source block."
+  :group 'org-babel
+  :type 'string)
+
 (defcustom org-babel-latex-htlatex-packages
   '("[usenames]{color}" "{tikz}" "{color}" "{listings}" "{amsmath}")
   "Packages to use for htlatex export."
@@ -114,12 +131,26 @@ This function is called by `org-babel-execute-src-block'."
 			 (mapconcat #'identity headers "\n"))))
 	   (org-create-formula-image
             body out-file org-format-latex-options in-buffer)))
+	 ((string= "svg" extension)
+	  (with-temp-file tex-file
+		 (insert (concat (funcall org-babel-latex-preamble params)
+			 (mapconcat #'identity headers "\n")
+			 "\\begin{document}"
+			 body
+			 "\\end{document}")))
+	  (let ((tmp-pdf (org-babel-latex-tex-to-pdf tex-file)))
+                  (let* ((log-buf (get-buffer-create "*Org Babel LaTeX Output*"))
+                         (err-msg "org babel latex failed")
+                         (img-out (org-compile-file
+	                           tmp-pdf
+                                   (list org-babel-latex-pdf-svg-process)
+                                   extension err-msg log-buf)))
+                    (shell-command (format "mv %s %s" img-out out-file)))))
          ((string-suffix-p ".tikz" out-file)
 	  (when (file-exists-p out-file) (delete-file out-file))
 	  (with-temp-file out-file
 	    (insert body)))
-	 ((and (or (string= "svg" extension)
-		   (string= "html" extension))
+	 ((and (string= "html" extension)
 	       (executable-find org-babel-latex-htlatex))
 	  ;; TODO: this is a very different way of generating the
 	  ;; frame latex document than in the pdf case.  Ideally, both
-- 
2.28.0

Re: [PATCH] babel latex headers and image generation commands

[Matt Huszagh]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

Matt Huszagh <huszaghmatt@gmail.com> writes:

> Ok, I've finally gotten around to taking a crack at this. The patch is
> attached. Basically, it allows a lot more control when converting a
> latex source block into an svg image file.

I've added a few changes to the patch that additionally allow custom the
begin and end document environments. The purpose here is to allow latex
code within the document environment that is ignored by the body
export. For instance, I can set the page color with
{\color{some-color}...} and this doesn't mess up latex exports.

Please comment with any questions/concerns/thoughts.

Thanks!

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-latex.el-Make-latex-to-svg-compilation-very-custo.patch --]
[-- Type: text/x-patch, Size: 3827 bytes --]

From 6dbd5ae840f02333f83d40a9c27be06968279563 Mon Sep 17 00:00:00 2001
From: Matt Huszagh <huszaghmatt@gmail.com>
Date: Fri, 28 Aug 2020 22:26:05 -0700
Subject: [PATCH] ob-latex.el: Make latex to svg compilation very customizable

* lisp/ob-latex.el (org-babel-latex-preamble): Add latex preamble
customization.
(org-babel-latex-begin-env): Add latex document environment begin
customization.
(org-babel-latex-end-env): Add latex document environment end
customization.
(org-babel-latex-pdf-svg-process): Add customization for converting a
pdf to svg.
(org-babel-execute:latex): Add specific case for svg generation from
latex block.
---
 lisp/ob-latex.el | 57 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 55 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-latex.el b/lisp/ob-latex.el
index 4b343dd14..359179476 100644
--- a/lisp/ob-latex.el
+++ b/lisp/ob-latex.el
@@ -70,6 +70,45 @@
   :group 'org-babel
   :type 'string)
 
+(defcustom org-babel-latex-preamble
+  (lambda (_)
+    "\\documentclass[preview]{standalone}
+\\def\\pgfsysdriver{pgfsys-tex4ht.def}
+")
+  "Closure which evaluates at runtime to the latex preamble.  It
+takes 1 argument which is the parameters of the source block."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-begin-env
+  (lambda (_)
+    "\\begin{document}")
+  "Closure which evaluates at runtime to the begin part of the
+document environment.  It takes 1 argument which is the
+parameters of the source block.  This allows adding additional
+code that will be ignored when exporting the literal latex
+source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-end-env
+  (lambda (_)
+    "\\end{document}")
+  "Closure which evaluates at runtime to the end part of the
+document environment.  It takes 1 argument which is the
+parameters of the source block.  This allows adding additional
+code that will be ignored when exporting the literal latex
+source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-pdf-svg-process
+  "inkscape --pdf-poppler %f -T -l -o %O"
+  "Command used to convert a PDF file to an SVG file when
+executing a latex source block."
+  :group 'org-babel
+  :type 'string)
+
 (defcustom org-babel-latex-htlatex-packages
   '("[usenames]{color}" "{tikz}" "{color}" "{listings}" "{amsmath}")
   "Packages to use for htlatex export."
@@ -114,12 +153,26 @@ This function is called by `org-babel-execute-src-block'."
 			 (mapconcat #'identity headers "\n"))))
 	   (org-create-formula-image
             body out-file org-format-latex-options in-buffer)))
+	 ((string= "svg" extension)
+	  (with-temp-file tex-file
+		 (insert (concat (funcall org-babel-latex-preamble params)
+			 (mapconcat #'identity headers "\n")
+			 (funcall org-babel-latex-begin-env)
+			 body
+			 (funcall org-babel-latex-end-env))))
+	  (let ((tmp-pdf (org-babel-latex-tex-to-pdf tex-file)))
+                  (let* ((log-buf (get-buffer-create "*Org Babel LaTeX Output*"))
+                         (err-msg "org babel latex failed")
+                         (img-out (org-compile-file
+	                           tmp-pdf
+                                   (list org-babel-latex-pdf-svg-process)
+                                   extension err-msg log-buf)))
+                    (shell-command (format "mv %s %s" img-out out-file)))))
          ((string-suffix-p ".tikz" out-file)
 	  (when (file-exists-p out-file) (delete-file out-file))
 	  (with-temp-file out-file
 	    (insert body)))
-	 ((and (or (string= "svg" extension)
-		   (string= "html" extension))
+	 ((and (string= "html" extension)
 	       (executable-find org-babel-latex-htlatex))
 	  ;; TODO: this is a very different way of generating the
 	  ;; frame latex document than in the pdf case.  Ideally, both
-- 
2.28.0

Re: [PATCH] babel latex headers and image generation commands

[Matt Huszagh]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

Matt Huszagh <huszaghmatt@gmail.com> writes:

> I've added a few changes to the patch that additionally allow custom the
> begin and end document environments. The purpose here is to allow latex
> code within the document environment that is ignored by the body
> export. For instance, I can set the page color with
> {\color{some-color}...} and this doesn't mess up latex exports.

I've fixed a minor bug with the previous patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-latex.el-Make-latex-to-svg-compilation-very-custo.patch --]
[-- Type: text/x-patch, Size: 3841 bytes --]

From 1bcd1d28dde6625d0c648c92243260b46433e1eb Mon Sep 17 00:00:00 2001
From: Matt Huszagh <huszaghmatt@gmail.com>
Date: Fri, 28 Aug 2020 22:26:05 -0700
Subject: [PATCH] ob-latex.el: Make latex to svg compilation very customizable

* lisp/ob-latex.el (org-babel-latex-preamble): Add latex preamble
customization.
(org-babel-latex-begin-env): Add latex document environment begin
customization.
(org-babel-latex-end-env): Add latex document environment end
customization.
(org-babel-latex-pdf-svg-process): Add customization for converting a
pdf to svg.
(org-babel-execute:latex): Add specific case for svg generation from
latex block.
---
 lisp/ob-latex.el | 57 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 55 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-latex.el b/lisp/ob-latex.el
index 4b343dd14..991873e2e 100644
--- a/lisp/ob-latex.el
+++ b/lisp/ob-latex.el
@@ -70,6 +70,45 @@
   :group 'org-babel
   :type 'string)
 
+(defcustom org-babel-latex-preamble
+  (lambda (_)
+    "\\documentclass[preview]{standalone}
+\\def\\pgfsysdriver{pgfsys-tex4ht.def}
+")
+  "Closure which evaluates at runtime to the latex preamble.  It
+takes 1 argument which is the parameters of the source block."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-begin-env
+  (lambda (_)
+    "\\begin{document}")
+  "Closure which evaluates at runtime to the begin part of the
+document environment.  It takes 1 argument which is the
+parameters of the source block.  This allows adding additional
+code that will be ignored when exporting the literal latex
+source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-end-env
+  (lambda (_)
+    "\\end{document}")
+  "Closure which evaluates at runtime to the end part of the
+document environment.  It takes 1 argument which is the
+parameters of the source block.  This allows adding additional
+code that will be ignored when exporting the literal latex
+source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-pdf-svg-process
+  "inkscape --pdf-poppler %f -T -l -o %O"
+  "Command used to convert a PDF file to an SVG file when
+executing a latex source block."
+  :group 'org-babel
+  :type 'string)
+
 (defcustom org-babel-latex-htlatex-packages
   '("[usenames]{color}" "{tikz}" "{color}" "{listings}" "{amsmath}")
   "Packages to use for htlatex export."
@@ -114,12 +153,26 @@ This function is called by `org-babel-execute-src-block'."
 			 (mapconcat #'identity headers "\n"))))
 	   (org-create-formula-image
             body out-file org-format-latex-options in-buffer)))
+	 ((string= "svg" extension)
+	  (with-temp-file tex-file
+		 (insert (concat (funcall org-babel-latex-preamble params)
+			 (mapconcat #'identity headers "\n")
+			 (funcall org-babel-latex-begin-env params)
+			 body
+			 (funcall org-babel-latex-end-env params))))
+	  (let ((tmp-pdf (org-babel-latex-tex-to-pdf tex-file)))
+                  (let* ((log-buf (get-buffer-create "*Org Babel LaTeX Output*"))
+                         (err-msg "org babel latex failed")
+                         (img-out (org-compile-file
+	                           tmp-pdf
+                                   (list org-babel-latex-pdf-svg-process)
+                                   extension err-msg log-buf)))
+                    (shell-command (format "mv %s %s" img-out out-file)))))
          ((string-suffix-p ".tikz" out-file)
 	  (when (file-exists-p out-file) (delete-file out-file))
 	  (with-temp-file out-file
 	    (insert body)))
-	 ((and (or (string= "svg" extension)
-		   (string= "html" extension))
+	 ((and (string= "html" extension)
 	       (executable-find org-babel-latex-htlatex))
 	  ;; TODO: this is a very different way of generating the
 	  ;; frame latex document than in the pdf case.  Ideally, both
-- 
2.28.0

Re: [PATCH] babel latex headers and image generation commands

[Bastien]

Hi Matt,

Matt Huszagh <huszaghmatt@gmail.com> writes:

> I've added a few changes to the patch that additionally allow custom the
> begin and end document environments. The purpose here is to allow latex
> code within the document environment that is ignored by the body
> export. For instance, I can set the page color with
> {\color{some-color}...} and this doesn't mess up latex exports.
>
> Please comment with any questions/concerns/thoughts.

It would be good to hear feedback from other ob-latex.el users.

-- 
 Bastien

Re: [PATCH] babel latex headers and image generation commands

[Bastien]

Hi Matt,

Matt Huszagh <huszaghmatt@gmail.com> writes:

> Matt Huszagh <huszaghmatt@gmail.com> writes:
>
>> I've added a few changes to the patch that additionally allow custom the
>> begin and end document environments. The purpose here is to allow latex
>> code within the document environment that is ignored by the body
>> export. For instance, I can set the page color with
>> {\color{some-color}...} and this doesn't mess up latex exports.
>
> I've fixed a minor bug with the previous patch.

Thanks -- that'd be for after 9.4 of course.

> From 1bcd1d28dde6625d0c648c92243260b46433e1eb Mon Sep 17 00:00:00 2001
> From: Matt Huszagh <huszaghmatt@gmail.com>
> Date: Fri, 28 Aug 2020 22:26:05 -0700
> Subject: [PATCH] ob-latex.el: Make latex to svg compilation very customizable
>
> * lisp/ob-latex.el (org-babel-latex-preamble): Add latex preamble
> customization.
> (org-babel-latex-begin-env): Add latex document environment begin
> customization.
> (org-babel-latex-end-env): Add latex document environment end
> customization.
> (org-babel-latex-pdf-svg-process): Add customization for converting a
> pdf to svg.
> (org-babel-execute:latex): Add specific case for svg generation from
> latex block.

Prefer

  * lisp/ob-latex.el (org-babel-latex-preamble): New option for LaTeX
  preamble customization.

"New option" is quite standard, an "option" being a customizable
variable.  In this case, "New option" would probably be enough, given
the name of the option is quite self-explanatory.  Also, some find it
pedantic, but "LaTeX" is the correct spelling in a changelog I guess.

> +(defcustom org-babel-latex-preamble
> +  (lambda (_)
> +    "\\documentclass[preview]{standalone}
> +\\def\\pgfsysdriver{pgfsys-tex4ht.def}
> +")
> +  "Closure which evaluates at runtime to the latex preamble.  It
> +takes 1 argument which is the parameters of the source block."
> +  :group 'org-babel
> +  :type 'function)

The first line of the docstring should contain a sentence, so you'd
need to have a new paragraph after "runtime to the LaTeX preamble."

> +(defcustom org-babel-latex-pdf-svg-process
> +  "inkscape --pdf-poppler %f -T -l -o %O"
> +  "Command used to convert a PDF file to an SVG file when
> +executing a latex source block."
> +  :group 'org-babel
> +  :type 'string)

What for users who don't have inkscape?  

Thanks,

-- 
 Bastien

Re: [PATCH] babel latex headers and image generation commands

[Matt Huszagh]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

Bastien <bzg@gnu.org> writes:

> Prefer
>
>   * lisp/ob-latex.el (org-babel-latex-preamble): New option for LaTeX
>   preamble customization.
>
> "New option" is quite standard, an "option" being a customizable
> variable.  In this case, "New option" would probably be enough, given
> the name of the option is quite self-explanatory.  Also, some find it
> pedantic, but "LaTeX" is the correct spelling in a changelog I guess.

Fixed in new patch (attached).

> The first line of the docstring should contain a sentence, so you'd
> need to have a new paragraph after "runtime to the LaTeX preamble."

Also fixed. Making the first line a full sentence means that some lines
are a little longer than 80 characters. Is this acceptable?

> What for users who don't have inkscape?  

This is just a default, but I could use a dvisvgm command as the default
instead? Either way, converting a PDF to SVG will require an executable
outside Emacs, but I guess dvisvgm is more likely to be installed for
people using a texlive installation. My personal preference for inkscape
is because it should handle all PDF inputs, whereas there are some cases
where dvisvgm may fail (see
https://github.com/mgieseki/dvisvgm/issues/139) due to changes in
ghostscript. Still, dvisvgm generally does a very good job with PDF
inputs. Let me know your thoughts, I'd be happy to set the default to a
dvisvgm command instead.

Matt


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-latex.el-Make-latex-to-svg-compilation-very-custo.patch --]
[-- Type: text/x-patch, Size: 3938 bytes --]

From 1ff86f2267b653dff225837ccf13ebf417f7ed03 Mon Sep 17 00:00:00 2001
From: Matt Huszagh <huszaghmatt@gmail.com>
Date: Fri, 28 Aug 2020 22:26:05 -0700
Subject: [PATCH] ob-latex.el: Make latex to svg compilation very customizable

* lisp/ob-latex.el (org-babel-latex-preamble): New option for LaTeX
preamble customization.
(org-babel-latex-begin-env): New option for LaTeX document environment
begin customization.
(org-babel-latex-end-env): New option for LaTeX document environment
end customization.
(org-babel-latex-pdf-svg-process): New option for converting a pdf to
svg.
(org-babel-execute:latex): Add specific case for svg generation from
LaTeX block.
---
 lisp/ob-latex.el | 59 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 56 insertions(+), 3 deletions(-)

diff --git a/lisp/ob-latex.el b/lisp/ob-latex.el
index 4b343dd14..6a4f7a6ba 100644
--- a/lisp/ob-latex.el
+++ b/lisp/ob-latex.el
@@ -66,7 +66,46 @@
   "LaTeX-specific header arguments.")
 
 (defcustom org-babel-latex-htlatex "htlatex"
-  "The htlatex command to enable conversion of latex to SVG or HTML."
+  "The htlatex command to enable conversion of LaTeX to SVG or HTML."
+  :group 'org-babel
+  :type 'string)
+
+(defcustom org-babel-latex-preamble
+  (lambda (_)
+    "\\documentclass[preview]{standalone}
+\\def\\pgfsysdriver{pgfsys-tex4ht.def}
+")
+  "Closure which evaluates at runtime to the LaTeX preamble.
+
+It takes 1 argument which is the parameters of the source block."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-begin-env
+  (lambda (_)
+    "\\begin{document}")
+  "Closure which evaluates at runtime to the begin part of the document environment.
+
+It takes 1 argument which is the parameters of the source block.
+This allows adding additional code that will be ignored when
+exporting the literal LaTeX source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-end-env
+  (lambda (_)
+    "\\end{document}")
+  "Closure which evaluates at runtime to the end part of the document environment.
+
+It takes 1 argument which is the parameters of the source block.
+This allows adding additional code that will be ignored when
+exporting the literal LaTeX source."
+  :group 'org-babel
+  :type 'function)
+
+(defcustom org-babel-latex-pdf-svg-process
+  "inkscape --pdf-poppler %f -T -l -o %O"
+  "Command used to convert a PDF file to an SVG file when executing a latex source block."
   :group 'org-babel
   :type 'string)
 
@@ -114,12 +153,26 @@ This function is called by `org-babel-execute-src-block'."
 			 (mapconcat #'identity headers "\n"))))
 	   (org-create-formula-image
             body out-file org-format-latex-options in-buffer)))
+	 ((string= "svg" extension)
+	  (with-temp-file tex-file
+		 (insert (concat (funcall org-babel-latex-preamble params)
+			 (mapconcat #'identity headers "\n")
+			 (funcall org-babel-latex-begin-env params)
+			 body
+			 (funcall org-babel-latex-end-env params))))
+	  (let ((tmp-pdf (org-babel-latex-tex-to-pdf tex-file)))
+                  (let* ((log-buf (get-buffer-create "*Org Babel LaTeX Output*"))
+                         (err-msg "org babel latex failed")
+                         (img-out (org-compile-file
+	                           tmp-pdf
+                                   (list org-babel-latex-pdf-svg-process)
+                                   extension err-msg log-buf)))
+                    (shell-command (format "mv %s %s" img-out out-file)))))
          ((string-suffix-p ".tikz" out-file)
 	  (when (file-exists-p out-file) (delete-file out-file))
 	  (with-temp-file out-file
 	    (insert body)))
-	 ((and (or (string= "svg" extension)
-		   (string= "html" extension))
+	 ((and (string= "html" extension)
 	       (executable-find org-babel-latex-htlatex))
 	  ;; TODO: this is a very different way of generating the
 	  ;; frame latex document than in the pdf case.  Ideally, both
-- 
2.28.0

Re: [PATCH] babel latex headers and image generation commands

[Bastien]

Hi Matt,

sorry for the delayed answer.

Matt Huszagh <huszaghmatt@gmail.com> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Prefer
>>
>>   * lisp/ob-latex.el (org-babel-latex-preamble): New option for LaTeX
>>   preamble customization.
>>
>> "New option" is quite standard, an "option" being a customizable
>> variable.  In this case, "New option" would probably be enough, given
>> the name of the option is quite self-explanatory.  Also, some find it
>> pedantic, but "LaTeX" is the correct spelling in a changelog I guess.
>
> Fixed in new patch (attached).

Applied as ae35a3459, thanks!

>> The first line of the docstring should contain a sentence, so you'd
>> need to have a new paragraph after "runtime to the LaTeX preamble."
>
> Also fixed. Making the first line a full sentence means that some lines
> are a little longer than 80 characters. Is this acceptable?
>
>> What for users who don't have inkscape?  
>
> This is just a default, but I could use a dvisvgm command as the default
> instead? Either way, converting a PDF to SVG will require an executable
> outside Emacs, but I guess dvisvgm is more likely to be installed for
> people using a texlive installation. My personal preference for inkscape
> is because it should handle all PDF inputs, whereas there are some cases
> where dvisvgm may fail (see
> https://github.com/mgieseki/dvisvgm/issues/139) due to changes in
> ghostscript. Still, dvisvgm generally does a very good job with PDF
> inputs. Let me know your thoughts, I'd be happy to set the default to a
> dvisvgm command instead.

Let's see what people think when they try it after 9.5.

Can you provide a patch against etc/ORG-NEWS announce this?

Thanks,

-- 
 Bastien

Re: [PATCH] babel latex headers and image generation commands

[Matt Huszagh]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

Bastien <bzg@gnu.org> writes:

> sorry for the delayed answer.

No worries!

> Can you provide a patch against etc/ORG-NEWS announce this?

Attached. Let me know what you think.

Matt


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-etc-ORG-NEWS-Describe-new-behavior-of-babel-LaTeX-SV.patch --]
[-- Type: text/x-patch, Size: 1910 bytes --]

From 51fb3ef9843ae45884803142f150c5d2f4f4d4c9 Mon Sep 17 00:00:00 2001
From: Matt Huszagh <huszaghmatt@gmail.com>
Date: Sat, 24 Oct 2020 09:36:56 -0700
Subject: [PATCH] etc/ORG-NEWS: Describe new behavior of babel LaTeX SVG images

---
 etc/ORG-NEWS | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..b9b0c1271 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -35,6 +35,28 @@ omit a file description was to omit the header argument entirely,
 which made it difficult/impossible to provide a default value for
 =file-desc=.
 
+*** New behavior for babel LaTeX SVG image files
+
+Org babel now uses a two-stage process for converting latex source
+blocks to SVG image files (when the extension of the output file is
+~.svg~). The first stage in the process converts the latex block into
+a PDF file, which is then converted into an SVG file in the second
+stage. The TeX->PDF part uses the existing infrastructure for
+~org-babel-latex-tex-to-pdf~. The PDF->SVG part uses a command
+specified in a new customization,
+~org-babel-latex-pdf-svg-process~. By default, this uses inkscape for
+conversion, but since it is fully customizable, any other command can
+be used in its place. For instance, dvisvgm might be used here. This
+two-part processing replaces the previous use of htlatex to process
+LaTeX directly to SVG (htlatex is still used for HTML conversion).
+
+Conversion to SVG exposes a number of additional customizations that
+give the user full control over the contents of the latex source
+block. ~org-babel-latex-preamble~, ~org-babel-latex-begin-env~ and
+~org-babel-latex-end-env~ are new customization options added to allow
+the user to specify the preamble and code that preceedes and proceeds
+the contents of the source block.
+
 ** New features
 *** =ob-python= improvements to =:return= header argument 
 
-- 
2.28.0

Re: [PATCH] babel latex headers and image generation commands

[Bastien]

Matt Huszagh <huszaghmatt@gmail.com> writes:

>> Can you provide a patch against etc/ORG-NEWS announce this?
>
> Attached. Let me know what you think.

Applied (2af68d6a4) with a minor modification in the headline.

Thanks,

-- 
 Bastien