Re: org-mode: example blocks are no longer syntax highlighted in emacs

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Bruno Barbier]

Hi Rudi,

Rudi C <rudiwillalwaysloveyou@gmail.com> writes:

> After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
> syntax highlighted in emacs. They used to be syntax highlighted when I
> specified the block's language. Any ideas?
>
> I use Doom, so it might have been a third-party feature.
>
> I know org-mode officially suggests that example blocks should not have
> syntax highlighting, but I want it anyway. IMO `:eval never` is just not as
> good,

Why is `:eval never` not as good ?  You don't have to write it on each
code block; you may set it globally, per file, per headline, etc.

See https://orgmode.org/manual/Using-Header-Arguments.html.


> not to mention tools like `pandoc` also produce example blocks from
> markdown source blocks.

My version of pandoc generates org source blocks from markdown code
blocks (pandoc 3.1.11.1).

I'm not sure you'll get a better answer than this SE answer:

  https://emacs.stackexchange.com/questions/76466/how-do-i-get-syntax-highlighting-in-example-blocks-when-exporting-org-mode-to-ht

I.e. "begin_example" is just not designed for source blocks.  It
may have worked by chance before.

Hoping this helps,

Bruno

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Rudi C]

[-- Attachment #1: Type: text/plain, Size: 8639 bytes --]

> Why is `:eval never` not as good ?  You don't have to write it on each
> code block; you may set it globally, per file, per headline, etc.

The main reason is that I also use source blocks for babel blocks that are
runnable. So I cannot use per headline etc. solutions. I can use snippets
to automatically insert `:eval never`, but it's not as good a UX.

I also already have lots of org-mode notes written using `example lang`
blocks. Rewriting all of these will be a PITA.

> My version of pandoc generates org source blocks from markdown code
> blocks (pandoc 3.1.11.1).

I upgraded my pandoc, and it now indeed does this if the markdown block has
a source lang defined.

---

NickD has identified the commit that introduced this breaking change:

```
commit 616e80a9f10c4bd085d7b5ac96fd6ea23e9c9191
Author: <elided>
Date:   Thu Apr 6 08:49:20 2023 -0500

    Handle block-type when checking `org-src-fontify-natively`

    * lisp/org.el (org-fontify-meta-lines-and-blocks-1): Only fontify
      natively for blocks of src type.

 lisp/org.el | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
```

Thanks to the above commit (
https://github.com/bzg/org-mode/commit/616e80a9f10c4bd085d7b5ac96fd6ea23e9c9191),
I wrote this patch which can be put in one's config to make this behavior
configurable.

Can you merge this upstream? emacs is all about user customizability, and
in general, breaking backward compatibility without giving a way out is bad.

```lisp
(defvar night/org-fontify-block-types '("src" "example")
  "List of block types to fontify in org-mode.")

(defun night/org-fontify-block-p (block-type)
  "Check if BLOCK-TYPE is one of the types to fontify."
  (member block-type night/org-fontify-block-types))

(defun night/org-fontify-meta-lines-and-blocks-1 (limit)
  "Fontify #+ lines and blocks."
  (let ((case-fold-search t))
    (when (re-search-forward
  (rx bol (group (zero-or-more (any " \t")) "#"
 (group (group (or (seq "+" (one-or-more (any "a-zA-Z")) (optional ":"))
   (any " \t")
   eol))
(optional (group "_" (group (one-or-more (any "a-zA-Z"))))))
 (zero-or-more (any " \t"))
 (group (group (zero-or-more (not (any " \t\n"))))
(zero-or-more (any " \t"))
(group (zero-or-more any)))))
  limit t)
      (let ((beg (match-beginning 0))
   (end-of-beginline (match-end 0))
   ;; Including \n at end of #+begin line will include \n
   ;; after the end of block content.
   (block-start (match-end 0))
   (block-end nil)
   (lang (match-string 7))     ; The language, if it is a source block.
   (bol-after-beginline (line-beginning-position 2))
   (dc1 (downcase (match-string 2)))
   (dc3 (downcase (match-string 3)))
   (whole-blockline org-fontify-whole-block-delimiter-line)
   beg-of-endline end-of-endline nl-before-endline quoting block-type)
(cond
((and (match-end 4) (equal dc3 "+begin"))
 ;; Truly a block
 (setq block-type (downcase (match-string 5))
;; Src, example, export, maybe more.
quoting (member block-type org-protecting-blocks))
 (when (re-search-forward
(rx-to-string `(group bol (or (seq (one-or-more "*") space)
      (seq (zero-or-more (any " \t"))
   "#+end"
   ,(match-string 4)
   word-end
   (zero-or-more any)))))
;; We look further than LIMIT on purpose.
nil t)
   ;; We do have a matching #+end line.
   (setq beg-of-endline (match-beginning 0)
 end-of-endline (match-end 0)
 nl-before-endline (1- (match-beginning 0)))
   (setq block-end (match-beginning 0)) ; Include the final newline.
   (when quoting
     (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
     (remove-text-properties beg end-of-endline
     '(display t invisible t intangible t)))
   (add-text-properties
    beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
   (org-remove-flyspell-overlays-in beg bol-after-beginline)
   (org-remove-flyspell-overlays-in nl-before-endline end-of-endline)
            (cond
    ((and org-src-fontify-natively
                   (night/org-fontify-block-p block-type))
     (save-match-data
                (org-src-font-lock-fontify-block (or lang "") block-start
block-end))
     (add-text-properties bol-after-beginline block-end '(src-block t)))
    (quoting
     (add-text-properties
      bol-after-beginline beg-of-endline
      (list 'face
    (list :inherit
  (let ((face-name
 (intern (format "org-block-%s" lang))))
    (append (and (facep face-name) (list face-name))
    '(org-block)))))))
    ((not org-fontify-quote-and-verse-blocks))
    ((string= block-type "quote")
     (add-face-text-property
      bol-after-beginline beg-of-endline 'org-quote t))
    ((string= block-type "verse")
     (add-face-text-property
      bol-after-beginline beg-of-endline 'org-verse t)))
   ;; Fontify the #+begin and #+end lines of the blocks
   (add-text-properties
    beg (if whole-blockline bol-after-beginline end-of-beginline)
    '(face org-block-begin-line))
   (unless (eq (char-after beg-of-endline) ?*)
     (add-text-properties
      beg-of-endline
      (if whole-blockline
  (let ((beg-of-next-line (1+ end-of-endline)))
    (min (point-max) beg-of-next-line))
(min (point-max) end-of-endline))
      '(face org-block-end-line)))
   t))
((member dc1 '("+title:" "+subtitle:" "+author:" "+email:" "+date:"))
 (org-remove-flyspell-overlays-in
  (match-beginning 0)
  (if (equal "+title:" dc1) (match-end 2) (match-end 0)))
 (add-text-properties
  beg (match-end 3)
  (if (member (intern (substring dc1 1 -1)) org-hidden-keywords)
      '(font-lock-fontified t invisible t)
    '(font-lock-fontified t face org-document-info-keyword)))
 (add-text-properties
  (match-beginning 6) (min (point-max) (1+ (match-end 6)))
  (if (string-equal dc1 "+title:")
      '(font-lock-fontified t face org-document-title)
    '(font-lock-fontified t face org-document-info))))
((string-prefix-p "+caption" dc1)
 (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
 (remove-text-properties (match-beginning 0) (match-end 0)
 '(display t invisible t intangible t))
 ;; Handle short captions
 (save-excursion
   (forward-line 0)
   (looking-at (rx (group (zero-or-more (any " \t"))
  "#+caption"
  (optional "[" (zero-or-more any) "]")
  ":")
   (zero-or-more (any " \t")))))
 (add-text-properties (line-beginning-position) (match-end 1)
      '(font-lock-fontified t face org-meta-line))
 (add-text-properties (match-end 0) (line-end-position)
      '(font-lock-fontified t face org-block))
 t)
((member dc3 '(" " ""))
 ;; Just a comment, the plus was not there
 (org-remove-flyspell-overlays-in beg (match-end 0))
 (add-text-properties
  beg (match-end 0)
  '(font-lock-fontified t face font-lock-comment-face)))
(t ;; Just any other in-buffer setting, but not indented
 (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
 (remove-text-properties (match-beginning 0) (match-end 0)
 '(display t invisible t intangible t))
 (add-text-properties beg (match-end 0)
      '(font-lock-fontified t face org-meta-line))
 t))))))
(advice-add 'org-fontify-meta-lines-and-blocks-1 :override
#'night/org-fontify-meta-lines-and-blocks-1)
```

On Thu, Feb 22, 2024 at 7:17 PM Bruno Barbier <brubar.cs@gmail.com> wrote:

>
> Hi Rudi,
>
> Rudi C <rudiwillalwaysloveyou@gmail.com> writes:
>
> > After upgrading to emacs 29.2 and org 9.7, my example blocks are no
> longer
> > syntax highlighted in emacs. They used to be syntax highlighted when I
> > specified the block's language. Any ideas?
> >
> > I use Doom, so it might have been a third-party feature.
> >
> > I know org-mode officially suggests that example blocks should not have
> > syntax highlighting, but I want it anyway. IMO `:eval never` is just not
> as
> > good,
>
> Why is `:eval never` not as good ?  You don't have to write it on each
> code block; you may set it globally, per file, per headline, etc.
>
> See https://orgmode.org/manual/Using-Header-Arguments.html.
>
>
> > not to mention tools like `pandoc` also produce example blocks from
> > markdown source blocks.
>
> My version of pandoc generates org source blocks from markdown code
> blocks (pandoc 3.1.11.1).
>
> I'm not sure you'll get a better answer than this SE answer:
>
>
> https://emacs.stackexchange.com/questions/76466/how-do-i-get-syntax-highlighting-in-example-blocks-when-exporting-org-mode-to-ht
>
> I.e. "begin_example" is just not designed for source blocks.  It
> may have worked by chance before.
>
> Hoping this helps,
>
> Bruno
>
>

[-- Attachment #2: Type: text/html, Size: 11112 bytes --]

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Bruno Barbier]

Hi Rudy,

Note that I am not an org maintainer; just trying to help :-)

Rudi C <rudiwillalwaysloveyou@gmail.com> writes:

>> Why is `:eval never` not as good ?  You don't have to write it on each
>> code block; you may set it globally, per file, per headline, etc.
>
> The main reason is that I also use source blocks for babel blocks that are
> runnable. So I cannot use per headline etc. solutions. I can use snippets
> to automatically insert `:eval never`, but it's not as good a UX.

You could customize `org-insert-structure-template' like this:

    (add-to-list 'org-structure-template-alist
                  (cons "p"  "src python :eval never"))

Then:

    C-c C-, p
    
should insert your code example:

   #+begin_src python :eval never
   #+end_src


> I also already have lots of org-mode notes written using `example lang`
> blocks. Rewriting all of these will be a PITA.

You could teach Emacs to fix/rewrite those for you.  Ask here if you
need help.  You're probably not the only one that will need to fix
them.


> NickD has identified the commit that introduced this breaking change:
>
> ...
> https://github.com/bzg/org-mode/commit/616e80a9f10c4bd085d7b5ac96fd6ea23e9c9191
> I wrote this patch which can be put in one's config to make this behavior
> configurable.

Thanks for the link.

(your code layout was broken, it's more reliable to attach these kinds
of documents)


> Can you merge this upstream? emacs is all about user customizability, and
> in general, breaking backward compatibility without giving a way out is bad.

According to the email thread, the change was made only to match the
documentation (see
https://list.orgmode.org/87zg7lghzz.fsf@gmail.com/).

So, maybe a new option, like you suggested will be accepted, something like:

   (defvar org-fontify-block-types '("src")
      "Which block types to fontify.
      Prior versions of Org may fontify other blocks than src.  That
      invalid behavior is now fixed.  If you have documents relying on
      this invalid behavior, and, you cannot fix them, add the block
      types to this list.")
      
But:
   -  Fontification must be fast. The more options, the harder to
      optimize.

   -  It might just be opening a can of worms: Why is editing not
      working too then ? Why is the indentation incorrect? Why are
      they not exported correctly ? Why is pandoc not able to handle
      them correctly ?  etc.

Let see what other people think.

Bruno

org-mode: example blocks are no longer syntax highlighted in emacs

[Rudi C]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
syntax highlighted in emacs. They used to be syntax highlighted when I
specified the block's language. Any ideas?

I use Doom, so it might have been a third-party feature.

I know org-mode officially suggests that example blocks should not have
syntax highlighting, but I want it anyway. IMO `:eval never` is just not as
good, not to mention tools like `pandoc` also produce example blocks from
markdown source blocks.

Current Behavior:
[![enter image description here][1]][1]

Old (Correct) Behavior:
[![enter image description here][2]][2]

I have asked about this on SE:
- [org-mode: example blocks are no longer syntax highlighted in emacs -
Emacs Stack Exchange](
https://emacs.stackexchange.com/questions/80459/org-mode-example-blocks-are-no-longer-syntax-highlighted-in-emacs
)

PS: Please use reply-to-all, I am not subscribed to the list.


  [1]: https://i.stack.imgur.com/U9II9.png
  [2]: https://i.stack.imgur.com/lNNtO.png

[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Ihor Radchenko]

Rudi C <rudiwillalwaysloveyou@gmail.com> writes:

> After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
> syntax highlighted in emacs. They used to be syntax highlighted when I
> specified the block's language. Any ideas?

This is an undocumented feature.
Yet, there is no reason to remote it.
I recovered example block fontification on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9daad41cc

Fixed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Rudi C]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Thanks!

On Sun, Feb 25, 2024 at 1:30 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Rudi C <rudiwillalwaysloveyou@gmail.com> writes:
>
> > After upgrading to emacs 29.2 and org 9.7, my example blocks are no
> longer
> > syntax highlighted in emacs. They used to be syntax highlighted when I
> > specified the block's language. Any ideas?
>
> This is an undocumented feature.
> Yet, there is no reason to remote it.
> I recovered example block fontification on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9daad41cc
>
> Fixed.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>

[-- Attachment #2: Type: text/html, Size: 1585 bytes --]

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Max Nikulin]

On 25/02/2024 17:04, Ihor Radchenko wrote:
> Rudi C writes:
> 
>> After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
>> syntax highlighted in emacs.
> 
> Yet, there is no reason to remote it.
> I recovered example block fontification on main.

At least some people are against the feature:

https://list.orgmode.org/orgmode/CA+G3_PPYmiiwHYKkgiJDZQ=o7DvaG=0G3aQnPHSBKEMZsOyG-g@mail.gmail.com/T/#u
Tom Gillespie. Re: [DISCUSSION] Refactoring fontification system. Tue, 7 
Jun 2022 21:23:21 -0700.

I do not mind to have syntax highlighting support for #+begin_example 
blocks, but it should be coherent with ox exporters. My idea to unify 
#+begin_src and #+begin_example exporters was not meet warmly.

To notify users that fontification will be lost during export, the 
feature may be turned into an opt-in one by adding a defcustom user 
option disabled by default and having docstring clearly describing that 
language will be ignored during export and users need to suppress 
`org-lint' warnings.

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
>>> syntax highlighted in emacs.
>> 
>> Yet, there is no reason to remote it.
>> I recovered example block fontification on main.
>
> At least some people are against the feature:
>
> https://list.orgmode.org/orgmode/CA+G3_PPYmiiwHYKkgiJDZQ=o7DvaG=0G3aQnPHSBKEMZsOyG-g@mail.gmail.com/T/#u
> Tom Gillespie. Re: [DISCUSSION] Refactoring fontification system. Tue, 7 
> Jun 2022 21:23:21 -0700.

Tom is against babel support for example blocks. However, indicating
language and fontification have nothing to do with babel.

> I do not mind to have syntax highlighting support for #+begin_example 
> blocks, but it should be coherent with ox exporters. My idea to unify 
> #+begin_src and #+begin_example exporters was not meet warmly.

It just got no responses, right? I personally did not reply because it
was out of scope of that thread.

> To notify users that fontification will be lost during export, the 
> feature may be turned into an opt-in one by adding a defcustom user 
> option disabled by default and having docstring clearly describing that 
> language will be ignored during export and users need to suppress 
> `org-lint' warnings.

Colored export might be reasonable to add for example blocks,
but we should discuss that separately. Not in this thread.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>