* Code block syntax highlighting in async export @ 2022-10-02 3:24 David Lukeš 2022-10-02 7:24 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: David Lukeš @ 2022-10-02 3:24 UTC (permalink / raw) To: emacs-orgmode Hi all, I get different syntax highlighting in code blocks when using async export: no colors, just bold, italics and underlines. Is this expected, and if so, is there maybe an easy fix I just don’t know about? I found a similar problem in [batch export], but the workaround seems relatively complicated and brittle. Alternatively, if this is an inherent limitation of async/batch export, I’m thinking maybe [using an external colorizer like pygments] might be the simpler option. Thanks for any tips! Best, David [batch export] <https://emacs.stackexchange.com/a/38515> [using an external colorizer like pygments] <https://linevi.ch/en/org-pygments.html> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-02 3:24 Code block syntax highlighting in async export David Lukeš @ 2022-10-02 7:24 ` Ihor Radchenko 2022-10-02 12:13 ` Max Nikulin 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-10-02 7:24 UTC (permalink / raw) To: David Lukeš; +Cc: emacs-orgmode David Lukeš <dafydd.lukes@gmail.com> writes: > I get different syntax highlighting in code blocks when using async > export: no colors, just bold, italics and underlines. Is this expected, > and if so, is there maybe an easy fix I just don’t know about? Could you please provide an example? See https://orgmode.org/manual/Feedback.html > I found a similar problem in [batch export], but the workaround seems > relatively complicated and brittle. That is likely a different problem. In batch mode, Emacs disabled font-lock-mode unconditionally, which makes fontification in some major modes broken. -- Ihor Radchenko, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-02 7:24 ` Ihor Radchenko @ 2022-10-02 12:13 ` Max Nikulin 2022-10-03 12:08 ` David Lukeš 0 siblings, 1 reply; 13+ messages in thread From: Max Nikulin @ 2022-10-02 12:13 UTC (permalink / raw) To: David Lukeš; +Cc: emacs-orgmode On 02/10/2022 14:24, Ihor Radchenko wrote: > David Lukeš writes: > >> I get different syntax highlighting in code blocks when using async >> export: no colors, just bold, italics and underlines. Is this expected, >> and if so, is there maybe an easy fix I just don’t know about? > > Could you please provide an example? > See https://orgmode.org/manual/Feedback.html > >> I found a similar problem in [batch export], but the workaround seems >> relatively complicated and brittle. > > That is likely a different problem. In batch mode, Emacs disabled > font-lock-mode unconditionally, which makes fontification in some major > modes broken. Certainly more details is necessary, even export format (backend) is not specified. A problem that has some similar symptoms: M. Pger. Re: no syntax highlighting for code blocks with org-publish. Mon, 18 Jul 2022 22:55:15 +0000. https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-02 12:13 ` Max Nikulin @ 2022-10-03 12:08 ` David Lukeš 2022-10-03 12:42 ` Timothy ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: David Lukeš @ 2022-10-03 12:08 UTC (permalink / raw) To: Max Nikulin; +Cc: emacs-orgmode > Could you please provide an example? > Certainly more details is necessary, even export format (backend) is not > specified. Sorry for that! I somehow thought that the issue would sound familiar to more seasoned Org users, and someone would come back with either "yeah, you need to do X" or "yeah, that's a known limitation". The export backend is HTML. In case anyone wants to try reproducing this, I made a minimal working example here: https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting > A problem that has some similar symptoms: > M. Pger. Re: no syntax highlighting for code blocks with org-publish. > Mon, 18 Jul 2022 22:55:15 +0000. > https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com This looks indeed very similar, thank you! I'll investigate if I can leverage any of the workarounds mentioned. Best, David ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 12:08 ` David Lukeš @ 2022-10-03 12:42 ` Timothy 2022-10-03 13:22 ` David Lukeš 2022-10-03 15:57 ` Max Nikulin 2022-10-04 4:04 ` Ihor Radchenko 2 siblings, 1 reply; 13+ messages in thread From: Timothy @ 2022-10-03 12:42 UTC (permalink / raw) To: David Lukeš; +Cc: Max Nikulin, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1087 bytes --] Hi David, > Sorry for that! I somehow thought that the issue would sound familiar > to more seasoned Org users, and someone would come back with either > “yeah, you need to do X” or “yeah, that’s a known limitation”. Each backend is its own beast, somewhat 🙂. > The export backend is HTML. In case anyone wants to try reproducing > this, I made a minimal working example here: > <https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting> > This looks indeed very similar, thank you! I’ll investigate if I can > leverage any of the workarounds mentioned. There’s a decent chance that `engrave-faces' (a package of mine that can now be used for code blocks in LaTeX) will work in batch mode. Integrating it with `ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list. All the best, Timothy -- Timothy (`tecosaur'/`TEC'), Org mode contributor. Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/tec>. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 12:42 ` Timothy @ 2022-10-03 13:22 ` David Lukeš 2022-10-03 13:58 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: David Lukeš @ 2022-10-03 13:22 UTC (permalink / raw) To: Timothy; +Cc: Max Nikulin, emacs-orgmode > I'll investigate if I can leverage any of the workarounds mentioned. So, for batch processing, avoiding --batch or --script and just using --load with a script that either ends by killing Emacs, or killing it via an additional --eval, works, i.e. syntax highlighting uses colors. This was suggested in [1]. As [2] points out, adding -nw removes the colors, and so does adding --daemon (suggested in [3]), so don't do either. Enabling font-lock-mode manually and defining all the fontification (see [4]) sounds like it would also work, but honestly, that's too involved, so I didn't try it. Using an external colorizer (as shown in [5]), seems like a good alternative, if non-Emacs dependencies are acceptable. These two last options can also be made to work with async export (my original question), where you don't have control over how the batch instance of Emacs is invoked. > There’s a decent chance that `engrave-faces' (a package of mine that can now be > used for code blocks in LaTeX) will work in batch mode. Integrating it with > `ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list. Sounds great! If it will work in regular batch mode, then it should work with async as well :) Best, David [1] https://list.orgmode.org/orgmode/vZh_aXsUhlwNcRMF3miS6hNUo6xxUCFuzm4dwfCNQHcm4Ib53t8sJVVJ-PmYD4sRfoCvWRjtEG7L5rfN4pRe5nDxLFWxWEG8ZqcZ5YvJyDs=@protonmail.com/ [2] https://list.orgmode.org/orgmode/YmBR481ZMgZpA6svOfEMt3nYa8UPEKgvFUY5s8SpRWu2YdJz0pb0gPz8ajcTeKFbF7SD1fD9XibICa8Z-JdTiIqgjOlD1y4yg9uHCB176ZA=@protonmail.com/ [3] https://list.orgmode.org/orgmode/87a6923dbw.fsf@localhost/ [4] https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/ [5] https://linevi.ch/en/org-pygments.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 13:22 ` David Lukeš @ 2022-10-03 13:58 ` Ihor Radchenko 2022-10-03 14:40 ` Timothy 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-10-03 13:58 UTC (permalink / raw) To: David Lukeš; +Cc: Timothy, Max Nikulin, emacs-orgmode David Lukeš <dafydd.lukes@gmail.com> writes: >> I'll investigate if I can leverage any of the workarounds mentioned. > > So, for batch processing, avoiding --batch or --script and just using > --load with a script that either ends by killing Emacs, or killing it > via an additional --eval, works, i.e. syntax highlighting uses colors. > This was suggested in [1]. As [2] points out, adding -nw removes the > colors, and so does adding --daemon (suggested in [3]), so don't do > either. > > Enabling font-lock-mode manually and defining all the fontification > (see [4]) sounds like it would also work, but honestly, that's too > involved, so I didn't try it. To clarify, the latest Org version should not care about font-lock being enabled. `font-lock-ensure' usually works in batch mode as well even though font-lock-mode is technically disabled. Except when it does not. Some major modes do fontification awkwardly, which can be considered a bug in those major modes. You can refer to https://yhetil.org/emacs-bugs/874jzdhh5m.fsf@localhost/ > Using an external colorizer (as shown in [5]), seems like a good > alternative, if non-Emacs dependencies are acceptable. > > These two last options can also be made to work with async export (my > original question), where you don't have control over how the batch > instance of Emacs is invoked. > >> There’s a decent chance that `engrave-faces' (a package of mine that can now be >> used for code blocks in LaTeX) will work in batch mode. Integrating it with >> `ox-html', `ox-ansii', and `ox-odt' are all (distantly) on my todo list. > > Sounds great! If it will work in regular batch mode, then it should > work with async as well :) I would not be so sure. If the issue is major-mode not doing fontification in batch mode, engrave-faces cannot do much. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 13:58 ` Ihor Radchenko @ 2022-10-03 14:40 ` Timothy 2022-10-04 4:10 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: Timothy @ 2022-10-03 14:40 UTC (permalink / raw) To: Ihor Radchenko; +Cc: David Lukeš, Timothy, Max Nikulin, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 413 bytes --] Hi Ihor, > I would not be so sure. If the issue is major-mode not doing > fontification in batch mode, engrave-faces cannot do much. Try it 😉 All the best, Timothy -- Timothy (‘tecosaur’/‘TEC’), Org mode contributor. Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/tec>. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 14:40 ` Timothy @ 2022-10-04 4:10 ` Ihor Radchenko 2022-10-04 4:14 ` Timothy 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-10-04 4:10 UTC (permalink / raw) To: Timothy; +Cc: David Lukeš, Max Nikulin, emacs-orgmode Timothy <orgmode@tec.tecosaur.net> writes: > Hi Ihor, > >> I would not be so sure. If the issue is major-mode not doing >> fontification in batch mode, engrave-faces cannot do much. > > Try it 😉 You are right. With default preset, engrave-faces does not need to query Emacs about face colors and attributes and should not depend on the `display-graphic-p'. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-04 4:10 ` Ihor Radchenko @ 2022-10-04 4:14 ` Timothy 0 siblings, 0 replies; 13+ messages in thread From: Timothy @ 2022-10-04 4:14 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Timothy, David Lukeš, Max Nikulin, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 986 bytes --] Hi Ihor, >> Try it 😉 > > You are right. With default preset, engrave-faces does not need to query > Emacs about face colors and attributes and should not depend on the > `display-graphic-p’. Yep, it works because it’s basically implementing its own face-resolving layer. This was originally done to allow exporting in a theme different to the one currently being used as seamlessly as possible, it working with batch mode was a bonus. For the information of anyone else following the thread, you can easily add new presets by creating an elisp setup file for batch mode and running `C-u M-x engrave-faces-use-theme' in it — code defining a new preset will then be generated and inserted at point. All the best, Timothy -- Timothy (‘tecosaur’/‘TEC’), Org mode contributor. Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/tec>. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 12:08 ` David Lukeš 2022-10-03 12:42 ` Timothy @ 2022-10-03 15:57 ` Max Nikulin 2022-10-03 19:32 ` David Lukeš 2022-10-04 4:04 ` Ihor Radchenko 2 siblings, 1 reply; 13+ messages in thread From: Max Nikulin @ 2022-10-03 15:57 UTC (permalink / raw) To: David Lukeš; +Cc: emacs-orgmode On 03/10/2022 19:08, David Lukeš wrote: >> Could you please provide an example? > >> A problem that has some similar symptoms: >> M. Pger. Re: no syntax highlighting for code blocks with org-publish. >> Mon, 18 Jul 2022 22:55:15 +0000. >> https://list.orgmode.org/inTtQOKWAhnFydjBE9fQOOdo_oV6kIPG5_E0d8DsfFiYMIJ4RNLRk68Id6dyrzy1qIDK8YJu0Wty7objv22vR31PNrh-qB9vKxMgHPxNAu8=@protonmail.com > > This looks indeed very similar, thank you! I'll investigate if I can > leverage any of the workarounds mentioned. I have realized that https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-elisp.html has colorized source blocks. The file is created using https://git.sr.ht/~bzg/worg/blob/master/publish.sh and the script is executed inside a container with no available DISPLAY. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 15:57 ` Max Nikulin @ 2022-10-03 19:32 ` David Lukeš 0 siblings, 0 replies; 13+ messages in thread From: David Lukeš @ 2022-10-03 19:32 UTC (permalink / raw) To: Max Nikulin; +Cc: emacs-orgmode > I have realized that > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-elisp.html > has colorized source blocks. The file is created using > https://git.sr.ht/~bzg/worg/blob/master/publish.sh and the script is > executed inside a container with no available DISPLAY. Well spotted! I think the trick there is setting org-html-htmlize-output-type to 'css. See the variable's documentation: Choices are css to export the CSS selectors only,inline-css to export the CSS attribute values inline in the HTML or nil to export plain text. We use as default inline-css, in order to make the resulting HTML self-containing. However, this will fail when using Emacs in batch mode for export, because then no rich font definitions are in place. It will also not be good if people with different Emacs setup contribute HTML files to a website, because the fonts will represent the individual setups. In these cases, it is much better to let Org/Htmlize assign classes only, and to use a style file to define the look of these classes. To get a start for your css file, start Emacs session and make sure that all the faces you are interested in are defined, for example by loading files in all modes you want. Then, use the command M-x org-html-htmlize-generate-css to extract class definitions. So, still a workaround, technically, but a nice and clean one that definitely deserves to be mentioned in this context! Best, David ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Code block syntax highlighting in async export 2022-10-03 12:08 ` David Lukeš 2022-10-03 12:42 ` Timothy 2022-10-03 15:57 ` Max Nikulin @ 2022-10-04 4:04 ` Ihor Radchenko 2 siblings, 0 replies; 13+ messages in thread From: Ihor Radchenko @ 2022-10-04 4:04 UTC (permalink / raw) To: David Lukeš; +Cc: Max Nikulin, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 2733 bytes --] David Lukeš <dafydd.lukes@gmail.com> writes: >> Could you please provide an example? > >> Certainly more details is necessary, even export format (backend) is not >> specified. > > Sorry for that! I somehow thought that the issue would sound familiar > to more seasoned Org users, and someone would come back with either > "yeah, you need to do X" or "yeah, that's a known limitation". > > The export backend is HTML. In case anyone wants to try reproducing > this, I made a minimal working example here: > https://github.com/dlukes/org-html-a-sync-export-syntax-highlighting Thanks for the easy reproducer! I played around with the export there, and I think I have found the cause. As expected, the problem originates from batch mode of Emacs operation. ox-html relies upon htmlize-region -> htmlize-buffer-1 -> htmlize-make-face-map -> htmlize-face-to-fstruct -> htmlize-face-to-fstruct-1. In synchronous export `htmlize-face-to-fstruct-1' returns: font-lock-comment-face -> #s(htmlize-fstruct "#b22222" nil nil nil nil nil nil nil "org-comment") font-lock-comment-delimiter-face -> #s(htmlize-fstruct "#b22222" nil nil nil nil nil nil nil "org-comment-delimiter") font-lock-doc-face -> #s(htmlize-fstruct "#8b2252" nil nil nil nil nil nil nil "org-doc") font-lock-keyword-face -> #s(htmlize-fstruct "#a020f0" nil nil nil nil nil nil nil "org-keyword") font-lock-function-name-face -> #s(htmlize-fstruct "#0000ff" nil nil nil nil nil nil nil "org-function-name") In asynchronous export: default -> #s(htmlize-fstruct "#000000" "#ffffff" 1 nil nil nil nil nil "org-default") font-lock-comment-face -> #s(htmlize-fstruct nil nil nil t t nil nil nil "org-comment") font-lock-comment-delimiter-face -> #s(htmlize-fstruct nil nil nil t t nil nil nil "org-comment-delimiter") font-lock-doc-face -> #s(htmlize-fstruct nil nil nil nil t nil nil nil "org-doc") font-lock-keyword-face -> #s(htmlize-fstruct nil nil nil t nil nil nil nil "org-keyword") font-lock-function-name-face -> #s(htmlize-fstruct nil nil nil t nil nil nil nil "org-function-name") ----------------------------- What can we do on the Org side to fix the issue? One way could be avoiding batch execution altogether, like in the attached patch. However, it will break our process sentinel watching for the export results. Moreover, I am not sure how it will work when export is done from terminal Emacs. Also, I know no way to _not_ create an extra Emacs frame during such export - I imagine that a new frame (even minimized) might be annoying for the users. Maybe we should request Emacs to add --graphical-batch mode feature that will preserve graphical features while not interfering with the WM? [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ox-Do-not-disable-graphical-Emacs-capabilities-durin.patch --] [-- Type: text/x-patch, Size: 1581 bytes --] From 8bc0a66275ca6ffdf142e8e2c2b38d79923531d8 Mon Sep 17 00:00:00 2001 Message-Id: <8bc0a66275ca6ffdf142e8e2c2b38d79923531d8.1664855997.git.yantar92@gmail.com> From: Ihor Radchenko <yantar92@gmail.com> Date: Tue, 4 Oct 2022 11:58:45 +0800 Subject: [PATCH] ox: Do not disable graphical Emacs capabilities during async export * lisp/ox.el (org-export-async-start): Run Emacs in possibly iconized window instead of batch mode. This way, font colors can be processed by htmlize during async export. --- lisp/ox.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/ox.el b/lisp/ox.el index 42204ffde..a31459632 100644 --- a/lisp/ox.el +++ b/lisp/ox.el @@ -6432,7 +6432,9 @@ (defun org-export-async-start (fun body) (funcall ',copy-fun) (restore-buffer-modified-p nil) ;; Sexp to evaluate in the buffer. - (print ,body))) + (print ,body) + ;; Kill Emacs at the end. + (kill-emacs 0))) nil temp-file nil 'silent)) ;; Start external process. (let* ((process-connection-type nil) @@ -6444,7 +6446,9 @@ (defun org-export-async-start (fun body) (list "org-export-process" proc-buffer (expand-file-name invocation-name invocation-directory) - "--batch") + ;; "--batch" + "--iconic" + ) (if org-export-async-init-file (list "-Q" "-l" org-export-async-init-file) (list "-l" user-init-file)) -- 2.35.1 [-- Attachment #3: Type: text/plain, Size: 225 bytes --] -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply related [flat|nested] 13+ messages in thread
end of thread, other threads:[~2022-10-04 4:20 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-10-02 3:24 Code block syntax highlighting in async export David Lukeš 2022-10-02 7:24 ` Ihor Radchenko 2022-10-02 12:13 ` Max Nikulin 2022-10-03 12:08 ` David Lukeš 2022-10-03 12:42 ` Timothy 2022-10-03 13:22 ` David Lukeš 2022-10-03 13:58 ` Ihor Radchenko 2022-10-03 14:40 ` Timothy 2022-10-04 4:10 ` Ihor Radchenko 2022-10-04 4:14 ` Timothy 2022-10-03 15:57 ` Max Nikulin 2022-10-03 19:32 ` David Lukeš 2022-10-04 4:04 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).