Code block syntax highlighting in async export

Code block syntax highlighting in async export

[David Lukeš]

Hi all,

I get different syntax highlighting in code blocks when using async
export: no colors, just bold, italics and underlines. Is this expected,
and if so, is there maybe an easy fix I just don’t know about?

I found a similar problem in [batch export], but the workaround seems
relatively complicated and brittle.

Alternatively, if this is an inherent limitation of async/batch export,
I’m thinking maybe [using an external colorizer like pygments] might be
the simpler option.

Thanks for any tips!

Best,

David


[batch export] <https://emacs.stackexchange.com/a/38515>

[using an external colorizer like pygments]
<https://linevi.ch/en/org-pygments.html>

Re: Code block syntax highlighting in async export

[Ihor Radchenko]

David Lukeš <dafydd.lukes@gmail.com> writes:

> I get different syntax highlighting in code blocks when using async
> export: no colors, just bold, italics and underlines. Is this expected,
> and if so, is there maybe an easy fix I just don’t know about?

Could you please provide an example?
See https://orgmode.org/manual/Feedback.html

> I found a similar problem in [batch export], but the workaround seems
> relatively complicated and brittle.

That is likely a different problem. In batch mode, Emacs disabled
font-lock-mode unconditionally, which makes fontification in some major
modes broken.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Code block syntax highlighting in async export

[Max Nikulin]

On 02/10/2022 14:24, Ihor Radchenko wrote:
> David Lukeš writes:
> 
>> I get different syntax highlighting in code blocks when using async
>> export: no colors, just bold, italics and underlines. Is this expected,
>> and if so, is there maybe an easy fix I just don’t know about?
> 
> Could you please provide an example?
> See https://orgmode.org/manual/Feedback.html
> 
>> I found a similar problem in [batch export], but the workaround seems
>> relatively complicated and brittle.
> 
> That is likely a different problem. In batch mode, Emacs disabled
> font-lock-mode unconditionally, which makes fontification in some major
> modes broken.

Certainly more details is necessary, even export format (backend) is not 
specified.

A problem that has some similar symptoms:
M. Pger. Re: no syntax highlighting for code blocks with org-publish. 
Mon, 18 Jul 2022 22:55:15 +0000. 
https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com

Re: Code block syntax highlighting in async export

[David Lukeš]

> Could you please provide an example?

> Certainly more details is necessary, even export format (backend) is not
> specified.

Sorry for that! I somehow thought that the issue would sound familiar
to more seasoned Org users, and someone would come back with either
"yeah, you need to do X" or "yeah, that's a known limitation".

The export backend is HTML. In case anyone wants to try reproducing
this, I made a minimal working example here:
https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting

> A problem that has some similar symptoms:
> M. Pger. Re: no syntax highlighting for code blocks with org-publish.
> Mon, 18 Jul 2022 22:55:15 +0000.
> https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com

This looks indeed very similar, thank you! I'll investigate if I can
leverage any of the workarounds mentioned.

Best,

David

Re: Code block syntax highlighting in async export

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

Hi David,

> Sorry for that! I somehow thought that the issue would sound familiar
> to more seasoned Org users, and someone would come back with either
> “yeah, you need to do X” or “yeah, that’s a known limitation”.

Each backend is its own beast, somewhat 🙂.

> The export backend is HTML. In case anyone wants to try reproducing
> this, I made a minimal working example here:
> <https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting>

> This looks indeed very similar, thank you! I’ll investigate if I can
> leverage any of the workarounds mentioned.

There’s a decent chance that `engrave-faces' (a package of mine that can now be
used for code blocks in LaTeX) will work in batch mode. Integrating it with
`ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list.

All the best,
Timothy

-- 
Timothy (`tecosaur'/`TEC'), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: Code block syntax highlighting in async export

[David Lukeš]

> I'll investigate if I can leverage any of the workarounds mentioned.

So, for batch processing, avoiding --batch or --script and just using
--load with a script that either ends by killing Emacs, or killing it
via an additional --eval, works, i.e. syntax highlighting uses colors.
This was suggested in [1]. As [2] points out, adding -nw removes the
colors, and so does adding --daemon (suggested in [3]), so don't do
either.

Enabling font-lock-mode manually and defining all the fontification
(see [4]) sounds like it would also work, but honestly, that's too
involved, so I didn't try it.

Using an external colorizer (as shown in [5]), seems like a good
alternative, if non-Emacs dependencies are acceptable.

These two last options can also be made to work with async export (my
original question), where you don't have control over how the batch
instance of Emacs is invoked.

> There’s a decent chance that `engrave-faces' (a package of mine that can now be
> used for code blocks in LaTeX) will work in batch mode. Integrating it with
> `ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list.

Sounds great! If it will work in regular batch mode, then it should
work with async as well :)

Best,

David

[1] https://list.orgmode.org/orgmode/vZh_aXsUhlwNcRMF3miS6hNUo6xxUCFuzm4dwfCNQHcm4Ib53t8sJVVJ-PmYD4sRfoCvWRjtEG7L5rfN4pRe5nDxLFWxWEG8ZqcZ5YvJyDs=@protonmail.com/
[2] https://list.orgmode.org/orgmode/YmBR481ZMgZpA6svOfEMt3nYa8UPEKgvFUY5s8SpRWu2YdJz0pb0gPz8ajcTeKFbF7SD1fD9XibICa8Z-JdTiIqgjOlD1y4yg9uHCB176ZA=@protonmail.com/
[3] https://list.orgmode.org/orgmode/87a6923dbw.fsf@localhost/
[4] https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/
[5] https://linevi.ch/en/org-pygments.html

Re: Code block syntax highlighting in async export

[Ihor Radchenko]

David Lukeš <dafydd.lukes@gmail.com> writes:

>> I'll investigate if I can leverage any of the workarounds mentioned.
>
> So, for batch processing, avoiding --batch or --script and just using
> --load with a script that either ends by killing Emacs, or killing it
> via an additional --eval, works, i.e. syntax highlighting uses colors.
> This was suggested in [1]. As [2] points out, adding -nw removes the
> colors, and so does adding --daemon (suggested in [3]), so don't do
> either.
>
> Enabling font-lock-mode manually and defining all the fontification
> (see [4]) sounds like it would also work, but honestly, that's too
> involved, so I didn't try it.

To clarify, the latest Org version should not care about font-lock being
enabled. `font-lock-ensure' usually works in batch mode as well even
though font-lock-mode is technically disabled.

Except when it does not. Some major modes do fontification awkwardly,
which can be considered a bug in those major modes.
You can refer to https://yhetil.org/emacs-bugs/874jzdhh5m.fsf@localhost/

> Using an external colorizer (as shown in [5]), seems like a good
> alternative, if non-Emacs dependencies are acceptable.
>
> These two last options can also be made to work with async export (my
> original question), where you don't have control over how the batch
> instance of Emacs is invoked.
>
>> There’s a decent chance that `engrave-faces' (a package of mine that can now be
>> used for code blocks in LaTeX) will work in batch mode. Integrating it with
>> `ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list.
>
> Sounds great! If it will work in regular batch mode, then it should
> work with async as well :)

I would not be so sure. If the issue is major-mode not doing
fontification in batch mode, engrave-faces cannot do much.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Code block syntax highlighting in async export

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Hi Ihor,

> I would not be so sure. If the issue is major-mode not doing
> fontification in batch mode, engrave-faces cannot do much.

Try it 😉

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: Code block syntax highlighting in async export

[Max Nikulin]

On 03/10/2022 19:08, David Lukeš wrote:
>> Could you please provide an example?
> 
>> A problem that has some similar symptoms:
>> M. Pger. Re: no syntax highlighting for code blocks with org-publish.
>> Mon, 18 Jul 2022 22:55:15 +0000.
>> https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com
> 
> This looks indeed very similar, thank you! I'll investigate if I can
> leverage any of the workarounds mentioned.

I have realized that 
https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-elisp.html 
has colorized source blocks. The file is created using 
https://git.sr.ht/~bzg/worg/blob/master/publish.sh and the script is 
executed inside a container with no available DISPLAY.

Re: Code block syntax highlighting in async export

[David Lukeš]

> I have realized that
> https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-elisp.html
> has colorized source blocks. The file is created using
> https://git.sr.ht/~bzg/worg/blob/master/publish.sh and the script is
> executed inside a container with no available DISPLAY.

Well spotted! I think the trick there is setting
org-html-htmlize-output-type to 'css. See the variable's
documentation:

  Choices are css to export the CSS selectors only,inline-css
  to export the CSS attribute values inline in the HTML or nil to
  export plain text.  We use as default inline-css, in order to
  make the resulting HTML self-containing.

  However, this will fail when using Emacs in batch mode for export, because
  then no rich font definitions are in place.  It will also not be good if
  people with different Emacs setup contribute HTML files to a website,
  because the fonts will represent the individual setups.  In these cases,
  it is much better to let Org/Htmlize assign classes only, and to use
  a style file to define the look of these classes.
  To get a start for your css file, start Emacs session and make sure that
  all the faces you are interested in are defined, for example by loading files
  in all modes you want.  Then, use the command
  M-x org-html-htmlize-generate-css to extract class definitions.

So, still a workaround, technically, but a nice and clean one that
definitely deserves to be mentioned in this context!

Best,

David

Re: Code block syntax highlighting in async export

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

David Lukeš <dafydd.lukes@gmail.com> writes:

>> Could you please provide an example?
>
>> Certainly more details is necessary, even export format (backend) is not
>> specified.
>
> Sorry for that! I somehow thought that the issue would sound familiar
> to more seasoned Org users, and someone would come back with either
> "yeah, you need to do X" or "yeah, that's a known limitation".
>
> The export backend is HTML. In case anyone wants to try reproducing
> this, I made a minimal working example here:
> https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting

Thanks for the easy reproducer!
I played around with the export there, and I think I have found the
cause.

As expected, the problem originates from batch mode of Emacs
operation.

ox-html relies upon htmlize-region -> htmlize-buffer-1 ->
htmlize-make-face-map -> htmlize-face-to-fstruct ->
htmlize-face-to-fstruct-1.

In synchronous export `htmlize-face-to-fstruct-1' returns:

font-lock-comment-face -> #s(htmlize-fstruct "#b22222" nil nil nil nil nil nil nil "org-comment")
font-lock-comment-delimiter-face -> #s(htmlize-fstruct "#b22222" nil nil nil nil nil nil nil "org-comment-delimiter")
font-lock-doc-face -> #s(htmlize-fstruct "#8b2252" nil nil nil nil nil nil nil "org-doc")
font-lock-keyword-face -> #s(htmlize-fstruct "#a020f0" nil nil nil nil nil nil nil "org-keyword")
font-lock-function-name-face -> #s(htmlize-fstruct "#0000ff" nil nil nil nil nil nil nil "org-function-name")

In asynchronous export:

default -> #s(htmlize-fstruct "#000000" "#ffffff" 1 nil nil nil nil nil "org-default")
font-lock-comment-face -> #s(htmlize-fstruct nil nil nil t t nil nil nil "org-comment")
font-lock-comment-delimiter-face -> #s(htmlize-fstruct nil nil nil t t nil nil nil "org-comment-delimiter")
font-lock-doc-face -> #s(htmlize-fstruct nil nil nil nil t nil nil nil "org-doc")
font-lock-keyword-face -> #s(htmlize-fstruct nil nil nil t nil nil nil nil "org-keyword")
font-lock-function-name-face -> #s(htmlize-fstruct nil nil nil t nil nil nil nil "org-function-name")

-----------------------------

What can we do on the Org side to fix the issue?

One way could be avoiding batch execution altogether, like in the
attached patch. However, it will break our process sentinel watching for
the export results. Moreover, I am not sure how it will work when export
is done from terminal Emacs. Also, I know no way to _not_ create an
extra Emacs frame during such export - I imagine that a new frame (even
minimized) might be annoying for the users.

Maybe we should request Emacs to add --graphical-batch mode feature that
will preserve graphical features while not interfering with the WM?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-Do-not-disable-graphical-Emacs-capabilities-durin.patch --]
[-- Type: text/x-patch, Size: 1581 bytes --]

From 8bc0a66275ca6ffdf142e8e2c2b38d79923531d8 Mon Sep 17 00:00:00 2001
Message-Id: <8bc0a66275ca6ffdf142e8e2c2b38d79923531d8.1664855997.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Tue, 4 Oct 2022 11:58:45 +0800
Subject: [PATCH] ox: Do not disable graphical Emacs capabilities during async
 export

* lisp/ox.el (org-export-async-start): Run Emacs in possibly iconized
window instead of batch mode.  This way, font colors can be processed
by htmlize during async export.
---
 lisp/ox.el | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 42204ffde..a31459632 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -6432,7 +6432,9 @@ (defun org-export-async-start  (fun body)
                     (funcall ',copy-fun)
                     (restore-buffer-modified-p nil)
                     ;; Sexp to evaluate in the buffer.
-                    (print ,body)))
+                    (print ,body)
+                    ;; Kill Emacs at the end.
+                    (kill-emacs 0)))
          nil temp-file nil 'silent))
       ;; Start external process.
       (let* ((process-connection-type nil)
@@ -6444,7 +6446,9 @@ (defun org-export-async-start  (fun body)
 		(list "org-export-process"
 		      proc-buffer
 		      (expand-file-name invocation-name invocation-directory)
-		      "--batch")
+		      ;; "--batch"
+                      "--iconic"
+                      )
 		(if org-export-async-init-file
 		    (list "-Q" "-l" org-export-async-init-file)
 		  (list "-l" user-init-file))
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Code block syntax highlighting in async export

[Ihor Radchenko]

Timothy <orgmode@tec.tecosaur.net> writes:

> Hi Ihor,
>
>> I would not be so sure. If the issue is major-mode not doing
>> fontification in batch mode, engrave-faces cannot do much.
>
> Try it 😉

You are right. With default preset, engrave-faces does not need to query
Emacs about face colors and attributes and should not depend on the
`display-graphic-p'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Code block syntax highlighting in async export

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

Hi Ihor,

>> Try it 😉
>
> You are right. With default preset, engrave-faces does not need to query
> Emacs about face colors and attributes and should not depend on the
> `display-graphic-p’.

Yep, it works because it’s basically implementing its own face-resolving layer.
This was originally done to allow exporting in a theme different to the one
currently being used as seamlessly as possible, it working with batch mode was a
bonus.

For the information of anyone else following the thread, you can easily add new
presets by creating an elisp setup file for batch mode and running `C-u M-x
engrave-faces-use-theme' in it — code defining a new preset will then be
generated and inserted at point.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.