Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Please, add the ability to fontify code inside

#+BEGIN_LATEX
...
#+END_LATEX

blocks. Embedded LaTeX doesn't get native fontification and SRC
blocks aren't the place to just get fancy colors (besides, latex inline
preview doesn't work for them). Currently there is this trick:

#+BEGIN_LATEX latex
...
#+END_LATEX

which is a bit redundant. I propose to add a new option or to just
fontify LATEX blocks if fontification is enabled for latex SRC blocks.
(I prefer the second alternative but maybe there is a reason to keep
both options separated).

Thanks.

---

Emacs  : GNU Emacs 26.1.50 (build 13, x86_64-pc-linux-gnu, GTK+ Version 3.24.1)
 of 2018-10-25
Package: Org mode version 9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)

current state:
==============
(setq
 org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer)
 org-fontify-whole-heading-line t
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-refile-targets '((nil :maxlevel . 2) (("gtd.org" "diario.org") :maxlevel . 2))
 org-html-format-inlinetask-function 'org-html-format-inlinetask-default-function
 org-agenda-files '("gtd.org" "inbox.org" "phone.org" "tablet.org")
 org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
 org-reveal-start-hook '(org-decrypt-entry)
 org-mode-hook '((lambda nil
                  (let*
                   ((in-goto (string= (buffer-name) "*org-goto*")) (in-notes (string-prefix-p (expand-file-name "~/notes/") buffer-file-name))
                    (note-name (and in-notes (file-name-base buffer-file-name))))
                   (if in-notes (progn (org-indent-mode) (auto-revert-mode)))
                   (if (string= note-name "inbox") (progn (set (make-local-variable (quote auto-save-timeout)) 10) (auto-save-visited-mode)))
                   (if (or in-goto (member note-name (quote ("inbox" "tablet" "phone"))))
                    (progn
                     (let ((--dolist-tail-- (quote (org-level-1 org-level-2 org-level-3 org-level-4 org-level-5))) face)
                      (while --dolist-tail-- (setq face (car --dolist-tail--)) (face-remap-add-relative face :inherit (quote default))
                       (setq --dolist-tail-- (cdr --dolist-tail--)))
                      )
                     )
                    )
                   (add-hook (quote completion-at-point-functions) (quote pcomplete-completions-at-point) nil t))
                  )
                 #[0 "\300\301\302\303\304$\207" [add-hook change-major-mode-hook org-show-block-all append local] 5]
                 #[0 "\300\301\302\303\304$\207" [add-hook change-major-mode-hook org-babel-show-result-all append local] 5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-bibtex-headline-format-function #[257 "\300.\236A\207" [:title] 3 "\n\n(fn ENTRY)"]
 org-latex-format-drawer-function #[514 "\207" [] 3 "\n\n(fn _ CONTENTS)"]
 org-agenda-deadline-faces '((1.001 . error) (1.0 . org-warning) (0.5 . org-upcoming-deadline) (0.0 . org-upcoming-distant-deadline))
 org-babel-pre-tangle-hook '(save-buffer)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 org-hide-leading-stars t
 org-babel-load-languages '((emacs-lisp . t) (python . t))
 org-link-abbrev-alist '(("google" . "https://www.google.com/#q=") ("docs" . my-find-in-docs))
 org-export-backends '(ascii beamer html latex md)
 org-highlight-latex-and-related '(latex script entities)
 org-ascii-format-drawer-function #[771 ".\207" [] 4 "\n\n(fn NAME CONTENTS WIDTH)"]
 org-babel-python-command "ipython -i --simple-prompt --matplotlib=agg"
 org-fontify-quote-and-verse-blocks t
 org-fontify-done-headline t
 org-occur-hook '(org-first-headline-recenter)
 org-cycle-separator-lines 1
 org-footnote-auto-adjust t
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers org-cycle-show-empty-lines org-optimize-window-after-visibility-change)
 org-todo-keywords '((sequence "TODO(t)" "SOMEDAY(s)" "|" "DONE(d)"))
 org-speed-command-hook '(org-speed-command-activate org-babel-speed-command-activate)
 org-babel-tangle-lang-exts '(("python" . "py") ("emacs-lisp" . "el") ("elisp" . "el"))
 org-format-latex-options '(:foreground default :background default :scale 1.3 :html-foreground "Black" :html-background "Transparent" :html-scale
                            1.0 :matchers ("begin" "$1" "$" "$$" "\\(" "\\["))
 org-refile-use-outline-path t
 org-font-lock-set-keywords-hook '(doom-org-custom-fontification)
 org-confirm-shell-link-function 'yes-or-no-p
 org-stuck-projects '("-someday+LEVEL=2/-DONE" ("TODO") nil "")
 org-link-parameters '(("id" :follow org-id-open) ("rmail" :follow org-rmail-open :store org-rmail-store-link)
                       ("mhe" :follow org-mhe-open :store org-mhe-store-link) ("irc" :follow org-irc-visit :store org-irc-store-link)
                       ("info" :follow org-info-open :export org-info-export :store org-info-store-link)
                       ("gnus" :follow org-gnus-open :store org-gnus-store-link)
                       ("docview" :follow org-docview-open :export org-docview-export :store org-docview-store-link)
                       ("bibtex" :follow org-bibtex-open :store org-bibtex-store-link)
                       ("bbdb" :follow org-bbdb-open :export org-bbdb-export :complete org-bbdb-complete-link :store org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link) ("file+sys") ("file+emacs") ("doi" :follow org--open-doi-link)
                       ("elisp" :follow org--open-elisp-link) ("file" :complete org-file-complete-link)
                       ("ftp" :follow (lambda (path) (browse-url (concat "ftp:" path)))) ("help" :follow org--open-help-link)
                       ("http" :follow (lambda (path) (browse-url (concat "http:" path))))
                       ("https" :follow (lambda (path) (browse-url (concat "https:" path))))
                       ("mailto" :follow (lambda (path) (browse-url (concat "mailto:" path))))
                       ("news" :follow (lambda (path) (browse-url (concat "news:" path)))) ("shell" :follow org--open-shell-link))
 org-latex-format-headline-function 'org-latex-format-headline-default-function
 org-capture-templates '(("i" "Inbox Note" entry (file "inbox.org") "* %?" :empty-lines 1)
                         ("p" "Protocol Link" entry (file "inbox.org") "* %:description\n\n%:link\n\n%i%?" :empty-lines 1)
                         ("L" "Protocol Link" entry (file "inbox.org") "* %:description\n\n%:link%?" :empty-lines 1))
 org-latex-format-inlinetask-function 'org-latex-format-inlinetask-default-function
 org-html-format-drawer-function #[514 "\207" [] 3 "\n\n(fn NAME CONTENTS)"]
 org-latex-classes '(("beamer" "\\documentclass[presentation]{beamer}" ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
                     ("article" "\\documentclass[11pt]{article}" ("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" . "\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}") ("\\paragraph{%s}" . "\\paragraph*{%s}")
                      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("report" "\\documentclass[11pt]{report}" ("\\part{%s}" . "\\part*{%s}") ("\\chapter{%s}" . "\\chapter*{%s}")
                      ("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" . "\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
                     ("book" "\\documentclass[11pt]{book}" ("\\part{%s}" . "\\part*{%s}") ("\\chapter{%s}" . "\\chapter*{%s}")
                      ("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" . "\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
                     )
 org-html-format-headline-function 'org-html-format-headline-default-function
 org-confirm-babel-evaluate nil
 org-directory "~/notes"
 org-preview-latex-image-directory "/tmp/ltximg/"
 )

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Ok, I got the "#+BEGIN_LATEX latex" idea from reddit, but exploring
the source code I realized that the right thing to do here is to
"#+BEGIN_EXPORT latex" which makes more sense and is less redundant.

Now the problem still exists that "#+BEGIN_EXPORT latex" won't do
inline preview.

So, to sum up:

1. #+BEGIN_LATEX could be changed to honor current fontification setting

2. #+BEGIN_EXPORT latex should be changed to allow inline preview

I would say 2 is a bug.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Hello,

Carlos Pita <carlosjosepita@gmail.com> writes:

> So, to sum up:
>
> 1. #+BEGIN_LATEX could be changed to honor current fontification
> setting

This construct doesn't exist in Org.

> 2. #+BEGIN_EXPORT latex should be changed to allow inline preview
>
> I would say 2 is a bug.

It is not a bug. Why would it be one?

Regards,

-- 
Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

> > 1. #+BEGIN_LATEX could be changed to honor current fontification
> > setting
>
> This construct doesn't exist in Org.

Yes, I realized after posting, it was suggested in reddit but I was
unable to find any reference to it in the source code. Maybe it did
exist in the past since it's often mentioned in stackexchange and
reddit.

> > 2. #+BEGIN_EXPORT latex should be changed to allow inline preview
> It is not a bug. Why would it be one?

The behavior of

   (P) previews,
   (E) C-c ' editing and
   (F) native fontification

is very inconsistent for latex:

1. $xxx$ and $$xxx$$ no (F) no (E) but (P)

2. \begin{xxx}...\end{xxx} no (F) but (E) and (P)

3. #+BEGIN_EXPORT latex and #+LATEX  no (P) but (E) and (F)

I understand the limited behavior of $...$ since it's intended to be
for short snippets.

The lack of features for $$...$$ is more questionable. And you can do
\begin{displaymath}, which is exactly the same, and get (E).

Anyway, if there were an option for long code fragments to be
correctly folded, fontified, edited and previewed I would pay the
price of the "#+BEGIN_EXPORT latex" extra typing. The only problem is
that previews are not working there. So why not activate previews
inside export environments of latex type to get the full combo?

Regards
--
Carlos





>
> Regards,
>
> --
> Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 86 bytes --]

Indeed it would be nice to have preview working for SRC blocks with lang =
latex too.

[-- Attachment #2: Type: text/html, Size: 108 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

One last suggestion. I see the latex export backend takes any BEGIN_XXX and
outputs a \begin{XXX} environment. That would be almost what I was asking
for when XXX = equation or displaymath or whatever. Except that in that
case the inline preview wont work (for BEGIN_LATEX it was working just
because of the extra $$...$$ I added there). Again, this makes me believe
that in the same way a latex lang parameter activates native latex
fontification in *any* kind of block it should also activate inline latex
previewing. The relevant blocks here are SRC, EXPORT and any XXX exported
as a latex env.

[-- Attachment #2: Type: text/html, Size: 621 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Hello,

Carlos Pita <carlosjosepita@gmail.com> writes:

> The behavior of
>
>    (P) previews,
>    (E) C-c ' editing and
>    (F) native fontification
>
> is very inconsistent for latex:
>
> 1. $xxx$ and $$xxx$$ no (F) no (E) but (P)

I guess (E) could be added, we already do this for inline source code,
but no one bothered implementing it.

> 2. \begin{xxx}...\end{xxx} no (F) but (E) and (P)

This is consistent with 1., with the (E) feature.

> 3. #+BEGIN_EXPORT latex and #+LATEX  no (P) but (E) and (F)

This is something entirely different. #+BEGIN_EXPORT latex ...
#+END_EXPORT latex, in Org lingo, is not LaTeX #code, but raw text for
an export back-end called "latex". There is no guarantee export back-end
"latex" correspond to real LaTeX code.

You cannot expect the same behavior for apples and oranges, can you?

> The lack of features for $$...$$ is more questionable. And you can do
> \begin{displaymath}, which is exactly the same, and get (E).

A missing feature is still not a bug, i.e., feature requests are not bug
reports.

> Anyway, if there were an option for long code fragments to be
> correctly folded, fontified, edited and previewed I would pay the
> price of the "#+BEGIN_EXPORT latex" extra typing.

I'm not sure what you mean by "folded". You can see fontification and
you can edit #+begin_export latex blocks with <C-c '>.

There are not fontified in the Org buffer, but Org only fontifies source
blocks.

> The only problem is
> that previews are not working there. So why not activate previews
> inside export environments of latex type to get the full combo?

There may not be something to preview in the first place. More
precisely, they shouldn't contain something that can be previewed. For
that, you can use environments, e.g.
\begin{something}...\end{something}.

Regards,

-- 
Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

Hi Nicolas, thank you for your detailed answer!

To be honest I was wrapping my head around this until late tonight when it
all suddenly made sense and I more or less concluded the same you explained
about the role of each block and was going to say that you could indeed
dismiss this request.

Nevertheless there is one area I still believe that could be improved which
is native fontification of inline code. I almost got it by advicing
org-do-latex-and-related so that it does the equivalent of:

        (org-src-font-lock-fontify-block "latex"
                                         (+ offset (match-beginning 0))
                                         (match-end 0))
        ;; (font-lock-prepend-text-property
        ;;  (+ offset (match-beginning 0)) (match-end 0)
        ;;  'face 'org-latex-and-related)

That correctly fontifies latex if org-highlight-latex-and-related includes
'latex. The problem is that once you break the beginning of the
environment, say by adding a space so that "\begin" -> "\be gin" or
whatever, all fontification is lost for the entire environment and never
reapplied, no matter whether it became valid latex again afterwards or not.
The next statement in org-do-latex-and-related is:

        (add-text-properties (+ offset (match-beginning 0)) (match-end 0)
                  '(font-lock-multiline t)))

But this doesn't seem to help keeping font locking in sync. I will dig
deeper into emacs font locking mechanism until I get this working but,
since I'm no expert, maybe you can point in the right direction and spare
me some time.

A missing feature is still not a bug, i.e., feature requests are not bug
> reports.
>

I understand this, I just followed the instructions in the manual and
thought org-submit-bug-request was intended for everything feedback
related, so sorry if I should have written directly to the list or maybe
removed the Bug: prefix in the header.

Regards
--
Carlos

[-- Attachment #2: Type: text/html, Size: 2705 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

> all fontification is lost for the entire environment and never reapplied

The problem is with the limit argument passed by the font locking
framework to org-do-latex-and-related .

Once you natively fontified the environment this limit arg became non
nil and it's in general not enough to match the entire latex
environment. I assume this is because of the way native fontification
annotates the code which makes the framework think it would be enough
to look up to some limit. But this is not true anymore when you have
this hybrid syntax that forces to refontify the entire fragment in a
temporary buffer.

So in order to get native fontification working for inline latex you need to:

1. Ignore the limit argument.
2. Replace the calls to font-lock-prepend-text-property and
add-text-properties by a single call to
org-src-font-lock-fontify-block
3. Also this makes most sense if org-highlight-latex-and-related is
set to '(latex).

Maybe an option could be added that did something like 1, 2 and 3
above, in case the user prefers native fontification for inline latex.

In any case, here is an advice that will do the trick:

(advice-add 'org-do-latex-and-related
            :around (lambda (f limit)
                      (cl-letf
                          (((symbol-function 'font-lock-prepend-text-property)
                            (lambda (start end &rest args)
                              (org-src-font-lock-fontify-block "latex"
start end))))
                        (funcall f nil))))

Regards
---
Carlos

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

>      (lambda (start end &rest args)
>          (org-src-font-lock-fontify-block "latex" start end))))

Or a safer version:

         (lambda (start end prop face)
           (when (eq face 'org-latex-and-related)
              (org-src-font-lock-fontify-block "latex" start end)))))

Also note that the advice doesn't avoid the double setting of
(font-lock-multiline t). This is harmless but it's unnecessary
nevertheless. It's just intended as a workaround and to show that it's
pretty simple to extend native fontification to inline snippets.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Well, now I do have found a bug ;). It's related to the limit issue I
mentioned before. Indeed the problem with rehighlighting broken
environments and highlighting new environments also happens for basic
latex fontification when org-highlight-latex-and-related is set to
'(latex), it has nothing to do with my modification in order to call
org-src-font-lock-fontify-block. It seems that once you break the
multiline structure (for example, by putting a space between "be" and
"gin") org mode syntax highlighter is unable to identify it again once
recomposed. More generally, it's unable to identify environments
entered after the initial loading of the file into the buffer, being
them slight "repairments" of previously existing ones or just brand
new environments. Of course, systematically ignoring the search bound
is not the right solution. I think some kind of region extension
function is missing here.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

As a possible fix I implemented this
font-lock-extend-after-change-region-function:

(defun my-org-extend-region (beg end old-len)
  (let ((begin-re "[\t ]*\\(#\\+BEGIN\\|\\\\begin{\\)")
        (end-re "[\t ]*\\(#\\+END\\|\\\\end{\\)"))
    (save-excursion
      (goto-char beg)
      (beginning-of-line)
      (if (looking-at end-re)
          (setq beg (or (re-search-backward begin-re nil t) beg))
        (when (looking-at begin-re)
          (setq end (or (re-search-forward end-re nil t) end))))))
  (cons beg end))

It's simple and efficient. The idea is that if you have changed one
endpoint of a #+BEGIN/#+END or \begin{/\end{ block, then you'd better
extend the fontification region up to the other end. This way the
extension only triggers when a BEGIN/END/begin/end match is found in
the current line after a change, making the whole thing quite
lightweight.

Notice that I added the BEGIN/END case because I was having some
issues with org blocks also, although they were less frequent, just
some border cases. This hook made the behavior more robust and
predictable.

Maybe you would like to add this (or something similar or better)
while initializing org mode font lock stuff.

Once this is in place it's trivial to get native fontification of
latex snippets working as explained above, without the need to
disrespect search limits.

So, to sum up:

1. There is a bug and some kind of region extension is in order to
address it. Here I'm proposing one simple solution.

2. Then there is a feature request: add the ability to natively
fontify latex embedded fragments. I've also shown how to do that in a
couple of lines, but obviously some new customization option is also
required.

HIH
--
Carlos


On Wed, Nov 28, 2018 at 3:38 PM Carlos Pita <carlosjosepita@gmail.com> wrote:
>
> Well, now I do have found a bug ;). It's related to the limit issue I
> mentioned before. Indeed the problem with rehighlighting broken
> environments and highlighting new environments also happens for basic
> latex fontification when org-highlight-latex-and-related is set to
> '(latex), it has nothing to do with my modification in order to call
> org-src-font-lock-fontify-block. It seems that once you break the
> multiline structure (for example, by putting a space between "be" and
> "gin") org mode syntax highlighter is unable to identify it again once
> recomposed. More generally, it's unable to identify environments
> entered after the initial loading of the file into the buffer, being
> them slight "repairments" of previously existing ones or just brand
> new environments. Of course, systematically ignoring the search bound
> is not the right solution. I think some kind of region extension
> function is missing here.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Hi all,

I've been adding some improvements to the region extension function
(pasted below) in order to better support nesting, but anyway I'm
convinced that the current approach to org-highlight-latex-and-related
== '(latex) is hopeless except for small fragments and should be
reworked or dropped or kept with a caveat. This is because it quite
easily bumps into jit-lock-chunk-size (which is set at 500 by
default). Multiline matchers, specially medium to large ones, are
explicitly disadvised for a good reason. Once you have a larger chunk
of latex no region extension function will do it. But immediately
after you wrap the large latex environment inside a #+BEGIN/#+END pair
the extension function becomes helpful again: this because the
#+BEGIN/#+END matcher is incremental and eventually the region to
refontify spans the entire extended region and the latex matcher is
able to do its thing again.

Anyway, here is the extension function:

(defun my-org-extend-region (beg end old-len)
  (let ((begin-re "[\t ]*\\(#\\+BEGIN_\\|\\\\begin{\\)%s[ \t}]?")
        (end-re "[\t ]*\\(#\\+END_\\|\\\\end{\\)%s[ \t}]?")
        (name-re "\\([a-zA-Z0-9_\\*]+\\)")
        (extend (lambda (re pos dir)
                  (let ((re (format re (regexp-quote (match-string 2)))))
                    (or (re-search-forward re nil t dir) pos)))))
    (save-match-data
      (save-excursion
        (goto-char beg)
        (beginning-of-line)
        (if (looking-at (format end-re name-re))
            (setq beg (funcall extend begin-re beg -1))
          (when (looking-at (format begin-re name-re))
            (setq end (funcall extend end-re end 1)))))))
  (message "%s--%s" beg end)
  (cons beg end))

Best regards
--
Carlos

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

> the extension function becomes helpful again: this because the
> #+BEGIN/#+END matcher is incremental and eventually the region to

Well, on a closer inspection it's not that incremental, it's just
ignoring the limit...

(defun org-fontify-meta-lines-and-blocks-1 (limit)
      ....
      (when (re-search-forward
         (concat "^[ \t]*#\\+end" (match-string 4) "\\>.*")
         nil t)  ;; on purpose, we look further than LIMIT
       ....

This is against the rules! But if it's allowed here it should be
allowed in org-do-latex-and-related by the same token, shouldn't it?

So in order to get all this properly working:

1. Also ignore the limit in org-do-latex-and-related.

2. Install a region extension function so that multiline regions are
properly identified at the start and after syntax breaking change
(that is, at the beginning or at the end).

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Hello,

Carlos Pita <carlosjosepita@gmail.com> writes:

> So in order to get all this properly working:
>
> 1. Also ignore the limit in org-do-latex-and-related.
>
> 2. Install a region extension function so that multiline regions are
> properly identified at the start and after syntax breaking change
> (that is, at the beginning or at the end).

I pushed a different fix for LaTeX environment fontification. Please let
me know if it fixes your issue.

Regards,

-- 
Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Hi Nicolas, thank you for letting me know.

I've been giving it a try but it doesn't really fix the main problem
AFAICS (I've just redefined org-do-latex-and-related as in [1], maybe
I'm missing something). It's not surprising because I don't think you
can work around it without installing a region extension function.

For example, in order to reproduce one possible problem:

1) write \begin{xxx}
2) in the next line write xxx
3) now, in a third line, write \end{xxx}

This basic example already fails because you're not entering the start
and end delimiters at the same time (maybe you're using some snippet
facility for that?). Then you have an array of more complex cases:
blocks that are larger than jit-lock-chunk-size, changing one of the
ends of the block and restoring it, etc.

There is no perfect solution that I know of, but at the bare least you
have to let the font lock framework know that some entire  syntactical
region has changed. Also you have to ignore search limits since one
chunk won't do it as a syntactical unit, but you are already doing
this. Even doing all this, some border cases remain, in particular if
you have a really large block and you go to the end of it and change
its delimiter and then restore it, the jit framework will proceed by
chunks upwards but only when you scroll up and progressively reveal
the block, so in general it won't be refontified until you get near
the top delimiter and the search finally identifies a syntactic unit
from the top to the bottom. I don't think this case is important in
practice, but it's a good illustration of how things are working under
the hood.

Also, there is the fact that this limitations are shared by the font
locking of #+BEGIN/END blocks. Even if search limits are ignored, the
region extension is still missing there too.

I've considerably improved my region extension function since my last
post, if you're interested in adding it I could write a proper patch,
but here it is anyway:

(defun my-org-extend-region (beg end _old-len)
  (let ((begin-re "\\(\\\\\\[\\|\\(#\\+begin_\\|\\\\begin{\\)\\S-+\\)")
        (end-re "\\(\\\\\\]\\|\\(#\\+end_\\|\\\\end{\\)\\S-+\\)")
        (extend (lambda (r1 r2 dir)
                  (let ((re (replace-regexp-in-string "\\(begin\\|end\\)" r1
                             (replace-regexp-in-string "[][]" r2
                              (match-string-no-properties 0)))))
                    (re-search-forward (regexp-quote re) nil t dir)))))
    (save-match-data
      (save-excursion
        (goto-char beg)
        (back-to-indentation)
        (cond ((looking-at end-re)
               (cons (or (funcall extend "begin" "[" -1) beg) end))
              ((looking-at begin-re)
               (cons beg (or (funcall extend "end" "]" 1) end)))
              (t (cons beg end)))))))

Now it better supports nested blocks (of different types at least) and
also displaymath \[ \] delimiters.

Best regards
--
Carlos

[1] https://code.orgmode.org/bzg/org-mode/commit/af3e2b1856be4088071bd01f386560cff1e292bc?style=unified

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Oh, I see there have been some changes in
org-compute-latex-and-related-regexp in a previous commit. Now it
seems to be working at least for some simple examples. I'm working in
a long and complex beamer document so I will be able to more seriously
test this during the next few days. Also, I'm interested in
understanding your take on the problem. Thanks again.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

One problem I found after uninstalling my extension function is with
#+BEGIN/END blocks. For example, enable fontification for python,
elisp or whatever and write a long block like this:

#+BEGIN_XXX python

def xxx(x):
   pass

def xxx(x):
   pass

...

#+END_XXX python

Now scroll down to the end and change python to pyton and back. Even
when the delimiter is restored fontification isn't. I see that, for
latex blocks, you have added a case for searching backwards for an
opening delimiter. But org-fontify-meta-lines-and-blocks-1 lacks a
similar logic.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Also, in the example above, when you change and restore the opening
clause, refontification doesn't happen immediately, you have to
somewhat move the point to retrigger fontification.

> #+END_XXX python

Just #+END_XXX, of course.

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

Nicolas, I have modified org-fontify-meta-lines-and-blocks-1 following
your approach:

@@ -5991,6 +5991,13 @@ by a #."
         (dc3 (downcase (match-string 3)))
         end end1 quoting block-type)
     (cond
+     ((and (match-end 4) (equal dc3 "+end"))
+          (save-match-data
+            (save-excursion
+              (when (re-search-backward
+                     (concat "^[ \t]*#\\+begin" (match-string 4) "\\>.*")
+                     nil t)  ;; on purpose, we look further than LIMIT
+                (org-fontify-meta-lines-and-blocks-1 limit)))))
      ((and (match-end 4) (equal dc3 "+begin"))
       ;; Truly a block
       (setq block-type (downcase (match-string 5))

Nevertheless I'm not quite satisfied. If you play a bit with opening
and ending delimiters and moving around the buffer you will quite
easily find border cases in which the block is indeed recognized and
properties adequately set but still font locking is only partially
refreshed until you scroll, move to another buffer, etc.

I tried any combination of save-match-data and save-excursion wrapping
the recursive call, even without any of them, to no avail.

What I think it's happening here is that ignoring the limit forwards
could be a tolerable trick, but when you start jumping forward and
backwards in you fontification function, font-lock routines lose track
of what is meant to be refreshed; and if, instead, you save everything
before jumping around it's like nothing or too little have been
refontified from the point of view of the framework.

I still believe that explicitly extending the region is a saner
approach, it's the documented one at least. Maybe you could rewrite my
function in a better way using the precomputed latex regexps and
augmenting them with #+BEGIN/END counterparts. That still relies on
ignoring limits in the font locking function but from a cursory
inspection I understand the framework is quite robust when you move
the point further than allowed. Also, as I've said before, some border
cases remain when the block is too large (larger than the jit chunk)
and you break the ending delimiter, but nothing to worry about.

Best regards
--
Carlos

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Hello,

Carlos Pita <carlosjosepita@gmail.com> writes:

> I still believe that explicitly extending the region is a saner
> approach, it's the documented one at least.

The only sane approach for fontification is to use the parser. Any
solution relying to regexps is bound to fail somewhere, because Org
syntax cannot be parsed by regexps only. This would require to rewrite
the fontification process completely.

> Maybe you could rewrite my function in a better way using the
> precomputed latex regexps and augmenting them with #+BEGIN/END
> counterparts.

Per above, I'm not really interested in writing yet another workaround
for something that is going to fail anyway. For the time being, I just
accept that fontification is not perfect.

Regards,

-- 
Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

>
>
> > Maybe you could rewrite my function in a better way using the
> > precomputed latex regexps and augmenting them with #+BEGIN/END
> > counterparts.
>
> Per above, I'm not really interested in writing yet another workaround
> for something that is going to fail anyway. For the time being, I just
> accept that fontification is not perfect.
>

Hi Nicolas, I understand this. It's just that I have been using an
extension function for editing complex documents with lots of blocks (latex
and org) and I've not yet seen any problem, while I quickly spot rough
edges otherwise. It's not like writing yet another workaround in an
accumulative fashion, since the fontification functions just work forward
as always and the extension function is the one taking care of finding the
opposite end, plus it's just one function for all types of blocks.

I've no problem in contributing a patch if you're not inclined to work in
this anymore. It will have to move the end searching logic out of
org-do-latex-and-related though. All in all I don't think there would be
more lines of code or complexity and all block types would be given the
same treatment, moreover it would be a bit more by the book. Give it a
chance :) Anyway, thank you for your work.

Best regards
--
Carlos

[-- Attachment #2: Type: text/html, Size: 1694 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

Here is a patch on top of latest revision. It just adds an extension
function while initializing font locking, as documented in [1]. It
indeed reduces lines of code to some extent by moving the dirty stuff
out of the fontification function (but I tried to keep the code
formatting of your last commit, since it was rather different than the
one shipped with emacs). The approach is more robust AFAICS since the
font locking framework is correctly informed of which blocks should be
refontified before fontification itself happens. It covers all types
of blocks: #+BEGIN/#+END, \begin\end, \[\]. It's not a full parser but
at least it matches corresponding delimiters by type by using a simple
heuristic.

Of course, I cannot ensure that it's bug free but I was *really* using
and improving this for a couple of weeks now and I can say it does the
job. If you want me to change anything just let me know, I will have
some time for this for the next few days.

Best regards
--
Carlos

---

[1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Multiline-Font-Lock.html

[-- Attachment #2: region-extend.patch --]
[-- Type: text/x-patch, Size: 4311 bytes --]

diff --git a/lisp/org.el b/lisp/org.el
index 5dd5c02..115fa4d 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -6102,6 +6102,24 @@ by a #."
 	    (add-text-properties closing-start end '(invisible t)))
 	  t)))))
 
+(defun org-fontify-extend-region (beg end _old-len)
+  (let ((begin-re "\\(\\\\\\[\\|\\(#\\+begin_\\|\\\\begin{\\)\\S-+\\)")
+	(end-re "\\(\\\\\\]\\|\\(#\\+end_\\|\\\\end{\\)\\S-+\\)")
+	(extend (lambda (r1 r2 dir)
+		  (let ((re (replace-regexp-in-string "\\(begin\\|end\\)" r1
+			     (replace-regexp-in-string "[][]" r2
+			      (match-string-no-properties 0)))))
+		    (re-search-forward (regexp-quote re) nil t dir)))))
+    (save-match-data
+      (save-excursion
+	(goto-char beg)
+	(back-to-indentation)
+	(cond ((looking-at end-re)
+	       (cons (or (funcall extend "begin" "[" -1) beg) end))
+	      ((looking-at begin-re)
+	       (cons beg (or (funcall extend "end" "]" 1) end)))
+	      (t (cons beg end)))))))
+
 (defun org-activate-footnote-links (limit)
   "Add text properties for footnotes."
   (let ((fn (org-footnote-next-reference-or-definition limit)))
@@ -6233,16 +6251,11 @@ Result depends on variable `org-highlight-latex-and-related'."
 	       (org-use-sub-superscripts (list org-match-substring-regexp))))
 	(re-latex
 	 (when (memq 'latex org-highlight-latex-and-related)
-	   (let* ((matchers (plist-get org-format-latex-options :matchers))
-		  (regexps (and (member "begin" matchers)
-				'("\\\\end{[a-zA-Z0-9\\*]+}[ \t]*$"))))
-	     (dolist (matcher matchers)
-	       (pcase (assoc matcher org-latex-regexps)
-		 (`("begin" . ,_) (push "^[ \t]*\\\\begin{[a-zA-Z0-9\\*]+}"
-					regexps))
-		 (`(,_ ,regexp . ,_) (push regexp regexps))
-		 (_ nil)))
-	     (nreverse regexps))))
+	   (let ((matchers (plist-get org-format-latex-options :matchers)))
+	     (delq nil
+		   (mapcar (lambda (x)
+			     (and (member (car x) matchers) (nth 1 x)))
+			   org-latex-regexps)))))
 	(re-entities
 	 (when (memq 'entities org-highlight-latex-and-related)
 	   (list "\\\\\\(there4\\|sup[123]\\|frac[13][24]\\|[a-zA-Z]+\\)\
@@ -6259,38 +6272,20 @@ highlighted object, if any.  Return t if some highlighting was
 done, nil otherwise."
   (when (org-string-nw-p org-latex-and-related-regexp)
     (catch 'found
-      (while (re-search-forward org-latex-and-related-regexp limit t)
+      (while (re-search-forward org-latex-and-related-regexp
+				nil t) ;; on purpose, we ignore LIMIT
 	(unless (cl-some (lambda (f) (memq f '(org-code org-verbatim underline
-						   org-special-keyword)))
+					       org-special-keyword)))
 			 (save-excursion
 			   (goto-char (1+ (match-beginning 0)))
 			   (face-at-point nil t)))
-	  (let* ((start (if (memq (char-after (1+ (match-beginning 0)))
+	  (let ((offset (if (memq (char-after (1+ (match-beginning 0)))
 				  '(?_ ?^))
-			    (1+ (match-beginning 0))
-			  (match-beginning 0)))
-		 (end
-		  (let* ((b (match-beginning 0))
-			 (e (match-end 0))
-			 (m (buffer-substring-no-properties b e)))
-		    (cond
-		     ((string-match "\\`[ \t]*\\\\begin{\\([a-zA-Z0-9\\*]+\\)}"
-				    m)
-		      (let ((re (format "\\\\end{%s}[ \t]*$"
-					(regexp-quote (match-string 1 m)))))
-			(or (re-search-forward re nil t) e)))
-		     ((string-match "\\\\end{\\([a-zA-Z0-9\\*]+\\)}[ \t]*\\'" m)
-		      (let ((re (format "^[ \t]*\\\\begin{%s}"
-					(regexp-quote (match-string 1 m)))))
-			(setq start
-			      (or (save-excursion (re-search-backward re nil t))
-				  b))
-			(line-end-position)))
-		     ((string-match "\\`\\\\[a-zA-Z]+\\*?{\\'" m)
-		      (search-forward "}" nil t))
-		     (t e)))))
+			    1
+			  0)))
 	    (font-lock-prepend-text-property
-	     start end 'face 'org-latex-and-related)
+	     (+ offset (match-beginning 0)) (match-end 0)
+	     'face 'org-latex-and-related)
 	    (add-text-properties start end '(font-lock-multiline t)))
 	  (throw 'found t)))
       nil)))
@@ -6480,6 +6475,8 @@ needs to be inserted at a specific position in the font-lock sequence.")
     (setq-local org-font-lock-keywords org-font-lock-extra-keywords)
     (setq-local font-lock-defaults
 		'(org-font-lock-keywords t nil nil backward-paragraph))
+    (setq-local font-lock-extend-after-change-region-function
+		#'org-fontify-extend-region)
     (kill-local-variable 'font-lock-keywords)
     nil))
 

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Hello,

Carlos Pita <carlosjosepita@gmail.com> writes:

> Here is a patch on top of latest revision. It just adds an extension
> function while initializing font locking, as documented in [1]. It
> indeed reduces lines of code to some extent by moving the dirty stuff
> out of the fontification function (but I tried to keep the code
> formatting of your last commit, since it was rather different than the
> one shipped with emacs). The approach is more robust AFAICS since the
> font locking framework is correctly informed of which blocks should be
> refontified before fontification itself happens. It covers all types
> of blocks: #+BEGIN/#+END, \begin\end, \[\]. It's not a full parser but
> at least it matches corresponding delimiters by type by using a simple
> heuristic.

I still think it is not a good long-term solution for fontification. I'd
rather see work to integrate fontification with the parser.

Meanwhile, I applied your patch in master. I also added a commit
message.

Thank you.

Regards,

-- 
Nicolas Goaziou

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

>
>
> I'd
> rather see work to integrate fontification with the parser.
>

I could take a look at this sometime in the next few months but I'm not
sure what do you mean. In particular:

1. Is the parser doing partial parsings of every inserted/removed character
or line in a more or less complete syntactical environment around the
insertion/removal?

2. If the parser identifies a block how is it in a better position to flag
it as outdated regarding fontification? Do you mean completely
circumventing the search based font lock mechanism with its multiline
property and region extensions? That is, directly setting face related
properties and forcing a refresh of the region?


> Meanwhile, I applied your patch in master. I also added a commit
> message.
>

Thanks, Nicolas!

Best regards
--
Carlos

[-- Attachment #2: Type: text/html, Size: 1634 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

3. Or maybe you mean a region extension function that uses the parser to
find out the opposite delimiter of a block?

[-- Attachment #2: Type: text/html, Size: 139 bytes --]

Re: Bug: Add option to fontify latex blocks [9.1.9 (release_9.1.9-65-g5e4542 @ /home/carlos/local/stow/emacs-26/share/emacs/26.1.50/lisp/org/)]

[Nicolas Goaziou]

Carlos Pita <carlosjosepita@gmail.com> writes:

> 2. If the parser identifies a block how is it in a better position to flag
> it as outdated regarding fontification? Do you mean completely
> circumventing the search based font lock mechanism with its multiline
> property and region extensions? That is, directly setting face related
> properties and forcing a refresh of the region?

Indeed. Barring headlines, info lines and properties drawers, I suggest
to have Font lock delegate fontification to a function that would parse
an appropriate region and apply faces according to the AST.

One subtle problem is to determine the region to parse according to the
changes, but this is not specific to fontification. I.e., this function
needs to be written for proper parser caching anyway.