Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Nathaniel Nicandro <nathanielnicandro@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3232 bytes --]


Ihor Radchenko <yantar92@posteo.net> writes:

> I think that the most reasonable approach to fontify ANSI sequences will
> be the following:
>
> 1. We will consider ANSI within (a) all greater elements and lesser
>    elements that have RESULTS affiliated keyword (indicating that they
>    are result of code block evaluation); (b) otherwise, just lesser
>    elements, like paragraph, src block, example block, export block,
>    etc., but _not_ tables (c) otherwise, within verbatim-like objects,
>    like code, export-snippet, inline-src-block, table-cell, verbatim.
>
>    The three groups above should be declared via variables, so that
>    users can tweak them as necessary.
>
> 2. If ANSI sequence is encountered inside a verbatim-like object and we
>    did not see any ANSI sequences within parent element or greater
>    element, limit ANSI triggers to the current object.
>
>    Example:
>
>    #+RESULTS:
>    Lorem upsum =<ANSI>valor=. Some more text.
>
>    (only "valor" will be affected)
>
> 3. If the first ANSI sequence is encountered inside element and outside
>    verbatim-like object, the rest of the element is affected, including
>    all the objects.
>
>    Example:
>
>    #+RESULTS:
>    <ANSI>Lorem upsum =<ANSI>valor=. Some more text.
>
>    (the first ANSI affects everything, including verbatim; the second
>    ANSI also affects everything)
>
> 4. If the first ANSI sequence is encountered inside greater element with
>    RESULTS affiliated keyword, all the lesser elements inside will be
>    affected.
>
>    Example:
>
>    #+RESULTS:
>    :drawer:
>    <ANSI>Lorem upsum =valor=. Some more text.
>
>    Another paragraph inside drawer.
>    :end:
>
>    (everything down to :end: is affected)
>
>    or
>
>    #+RESULTS:
>    - <ANSI>list
>    - one
>    - two
>    - three
>

Hello Ihor,

Attached is the updated version of the patch.  I've also attached an
updated file that I've been using for testing the feature.

What I have is essentially a function, org-fontify-ansi-sequences, that
scans the buffer for an ANSI sequence and depending on the
element-context processes the region that should be affected according
to the rules you stated (see above).  The org-fontify-ansi-sequences-1
function scans the buffer element-wise and processes the appropriate
regions of the elements, even if no sequences appear in those regions,
according to an ansi-context.  This is to support the fourth rule you
mentioned.

Note that modifications to highlighted regions hasn't really been
considered so if you have a scenario like

#+RESULTS:
- <ANSI>Paragraph one
- Paragraph two
  Line 3

where the sequence affects everything down to "Line 3" and you make a
modification to line three, the fontification due to the sequence
disappears on that line.

Also note that lesser elements contained in greater elements that
don't have a RESULTS keyword are handled at the lesser element level
so if you have something like

#+begin_center
Paragraph <ANSI>one.

Paragraph two.
#+end_center

It would be the same as if you just had the paragraphs without the
greater element.

Please review the code and let me know what you think and how you
think we can move forward on this feature.

Thanks in advance.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Example Org file --]
[-- Type: text/x-org, Size: 2718 bytes --]

#+TITLE: Test

Section 1

* Greater ^[[42melements^[[0m
:PROPERTIES:
:CUSTOM_ID: 123
:END:

#+begin_center
Inner ^[[31mparagraph one

Inner paragraph two
#+end_center

:drawer:
Inner ^[[31mparagraph one

Inner paragraph two
:end:

#+BEGIN: dblock1 :scope subtree :maxlevel 2
- Item 1
- ^[[31mItem 2
  - Item 3
- ^[[32mItem 4
#+END:

[fn:1] Footnote ^[[42mdefinition

*************** TODO Inline ^[[42mtask 1^[[0m
Inner ^[[31mcontents^[[0m
*************** END

*************** TODO Inline ^[[42mtask 2^[[0m

- Paragraph ^[[31mone^[[0m
- Paragraph ^[[31mtwo
  - Paragraph three
- Paragraph four

| ^[[31mcell 1 | cell 2 |
| cell 3       | cell 4 |

#+begin_quote
open ^[[43m
#+end_quote

should not be highlighted

#+begin_quote
close ^[[0m
#+end_quote

* Lesser elements
:PROPERTIES:
:DESCRIPTION: ^[[31mvalue
:END:

#+CALL: fn(str="^[[31mtext^[[0m")

# Line ^[[31mone^[[0m

#+begin_comment
Line ^[[31mone^[[0m
Line ^[[32mtwo^[[0m
#+end_comment

%%(diary-anniversary 10 31 1948) Arthur's ^[[32mBirthday

#+begin_example
Line ^[[31mone^[[0m
Line ^[[32mtwo^[[0m
#+end_example

#+begin_export latex
Line ^[[31mone^[[0m
Line ^[[32ttwo^[[0m
#+end_export

: Line ^[[31mone^[[0m
: Line ^[[32mtwo^[[0m

#+AUTHOR: First ^[[31mLast

\begin{quote}
Line ^[[31mone^[[0m
Line ^[[32mtwo^[[0m
\end{quote}

Paragraph ^[[31mone
Line ^[[32mtwo^[[0m

#+begin_src python
for x in y:
    print(x + "^[[43mtest^[[0m")
#+end_src

* Object contexts

=ver^[[43mbatim= one

^[[42mLorem upsum =^[[43mvalor=. Some more text.

This is a paragraph src_python{return "t^[[43mest^[[0ming"} {{{results(=t^[[43mest^[[0ming=)}}} with
multiple inline src_python{return 5*4} {{{results(=20=)}}} source blocks.

An inline source block src_python{return 1+ 1 without an
end.  src_python{return "t^[[43mest^[[0ming"}.

^[[42m Paragraph =^[[43mone=

_underlined ^[[43m text *bold ^[[42m text ^[[0m* underlined ^[[0m text_

_underlined ^[[43m text_ plain^[[32m text _underlined ^[[42m text_

_underlined ^[[43m text *bold ^[[42m te /ita^[[31mlic/ xt ^[[0m end* underlined ^[[0m text_

_underlined ^[[43m text *bold ^[[42m te /ita^[[31mlic/ xt* underlined ^[[0m text_

_underlined ^[[43m text_ plain _underlined ^[[0m text_

* Greater elements with RESULTS keyword

#+RESULTS:
:drawer:
^[[42mLorem upsum =valor=. Some more text.

Another paragraph inside drawer.
:end:

#+RESULTS:
:RESULTS:
Paragraph ^[[42mone.

#+begin_example
- ^[[32mtest^[[0m
- ^[[31mtest^[[0m
#+end_example

Paragraph ^[[43mtwo.
:END:

#+RESULTS:
- ^[[42mlist
  - one
    - three 
- two
- three

#+RESULTS:
- [ ] ^[[42mCheckbox
- 
- 

#+RESULTS:
- ^[[42mList item

- [@5] List item

  :drawer:
  Interior
  
  - list inner

	- one two three
	  four five six
  :end:

- tag :: description
  
  
#+RESULTS:
- ^[[42mItem 1 

- Item 2
  
  | cell 1 | cell 2 |
  | cell 3 | cell 4 |





[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Patch --]
[-- Type: text/x-patch, Size: 13587 bytes --]

From 66baf6e1d435974fb4c51cc47eb5b3ace3feb22c Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro <nathanielnicandro@gmail.com>
Date: Tue, 9 May 2023 19:58:11 -0500
Subject: [PATCH] Highlight ANSI escape sequences

* etc/ORG-NEWS: Describe the new feature.
* org.el (org-fontify-ansi-sequences): New customization variable and
function which does the work of fontifying the sequences.
(org-ansi-highlightable-elements)
(org-ansi-highlightable-objects): New customization variables.
(org-ansi-new-context, org-ansi-process-region)
(org-ansi-process-block, org-ansi-process-paragraph)
(org-ansi-process-fixed-width)
(org-fontify-ansi-sequences-1): New functions.
(org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences`
function to the font-lock keywords.
(org-ansi-mode): New minor mode to enable/disable highlighting of the
sequences.  Enabled in Org buffers by default.
---
 etc/ORG-NEWS |  12 +++
 lisp/org.el  | 236 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 248 insertions(+)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 1207d6f..76a81e3 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -492,6 +492,18 @@ Currently implemented options are:
   iCalendar programs support this usage.
 
 ** New features
+*** ANSI escape sequences are now highlighted in the whole buffer
+
+A new customization ~org-fontify-ansi-sequences~ is available which
+tells Org to highlight all ANSI sequences in the buffer if non-nil and
+the new minor mode ~org-ansi-mode~ is enabled.
+
+To disable highlighting of the sequences you can either
+disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~
+and =M-x org-mode-restart RET=.  Doing the latter will disable
+highlighting of sequences in all newly opened Org buffers whereas
+doing the former disables highlighting locally to the current buffer.
+
 *** =ob-plantuml.el=: Support tikz file format output
 
 =ob-plantuml.el= now output =tikz= :file format via
diff --git a/lisp/org.el b/lisp/org.el
index d2cd0b9..64a853c 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -81,6 +81,7 @@ (eval-when-compile (require 'gnus-sum))
 (require 'calendar)
 (require 'find-func)
 (require 'format-spec)
+(require 'ansi-color)
 
 (condition-case nil
     (load (concat (file-name-directory load-file-name)
@@ -3608,6 +3609,12 @@ (defcustom org-fontify-whole-block-delimiter-line t
   :group 'org-appearance
   :type 'boolean)
 
+(defcustom org-fontify-ansi-sequences t
+  "Non-nil means to highlight ANSI escape sequences."
+  :group 'org-appearance
+  :type 'boolean
+  :package-version '(Org . "9.7"))
+
 (defcustom org-highlight-latex-and-related nil
   "Non-nil means highlight LaTeX related syntax in the buffer.
 When non-nil, the value should be a list containing any of the
@@ -5598,6 +5605,208 @@ (defun org-fontify-extend-region (beg end _old-len)
 	     (cons beg (or (funcall extend "end" "]" 1) end)))
 	    (t (cons beg end))))))
 
+(defcustom org-ansi-highlightable-elements
+  '(plain-list drawer
+    example-block export-block fixed-width paragraph)
+  "A list of element types that will have ANSI sequences processed."
+  :type '(list (symbol :tag "Element Type"))
+  :version "9.7"
+  :group 'org-appearance)
+
+(defcustom org-ansi-highlightable-objects
+  '(bold code export-snippet italic macro
+    strike-through table-cell underline verbatim)
+  "A list of object types that will have ANSI sequences processed."
+  :type '(list (symbol :tag "Object Type"))
+  :version "9.7"
+  :group 'org-appearance)
+
+(defun org-ansi-new-context (pos)
+  (list (list (make-bool-vector 8 nil)
+              nil nil)
+        (copy-marker pos)))
+
+(defun org-ansi-process-region (beg end &optional context)
+  (or context (setq context (org-ansi-new-context beg)))
+  (move-marker (cadr context) beg)
+  (let ((ansi-color-context-region context)
+        (ansi-color-apply-face-function
+         (lambda (beg end face)
+           (font-lock-prepend-text-property beg end 'face face))))
+    (ansi-color-apply-on-region beg end t)))
+
+(defun org-ansi-process-block (el &optional context)
+  (let ((beg (org-element-property :begin el))
+        (end (org-element-property :end el)))
+    (save-excursion
+      (goto-char beg)
+      (while (org-at-keyword-p)
+        (forward-line))
+      (setq beg (line-beginning-position 2)))
+    (save-excursion
+      (goto-char end)
+      (skip-chars-backward " \t\n")
+      (setq end (line-beginning-position)))
+    (org-ansi-process-region beg end context)))
+
+(defun org-ansi-process-paragraph (el &optional context)
+  ;; Compute the regions of the paragraph excluding inline
+  ;; source blocks.
+  (let ((pend (org-element-property :contents-end el)) beg end)
+    (push (point) beg)
+    (while (re-search-forward
+            "\\<src_\\([^ \t\n[{]+\\)[{[]" pend t)
+      (let ((el (org-element-context)))
+        (when (eq (org-element-type el) 'inline-src-block)
+          (push (org-element-property :begin el) end)
+          (goto-char (org-element-property :end el))
+          (push (point) beg))))
+    (push pend end)
+    (let ((ansi-context (or context (org-ansi-new-context (point)))))
+      (while beg
+        (org-ansi-process-region (pop beg) (pop end) ansi-context)))))
+
+(defun org-ansi-process-fixed-width (el &optional context)
+  (org-ansi-process-region
+   (org-element-property :begin el)
+   (save-excursion
+     (goto-char (org-element-property :end el))
+     (skip-chars-backward " \t\n")
+     (point))
+   context))
+
+(defun org-fontify-ansi-sequences-1 (limit &optional ansi-context)
+  (let ((skip-to-end-p
+         (lambda (el)
+           (or (null (org-element-property :contents-begin el))
+               (<= (org-element-property :contents-end el)
+                   (point)
+                   (org-element-property :end el))))))
+    (while (< (point) limit)
+      (let* ((el (org-element-at-point))
+             (type (org-element-type el)))
+        (pcase type
+          ;; Greater elements
+          ((or `headline `inlinetask `item
+               `center-block `quote-block `special-block
+               `drawer)
+           (if (funcall skip-to-end-p el)
+               (goto-char (org-element-property :end el))
+             (goto-char (org-element-property :contents-begin el))))
+          ((or `dynamic-block `footnote-definition `property-drawer)
+           (goto-char (org-element-property :end el)))
+          (`plain-list
+           (let ((end (org-element-property :end el)))
+             (goto-char (org-element-property :contents-begin el))
+             (while (< (point) end)
+               ;; Move to within the first item of a list.
+               (forward-char)
+               (let* ((item (org-element-at-point))
+                      (cbeg (org-element-property :contents-begin item)))
+                 (when cbeg
+                   (goto-char cbeg)
+                   (org-fontify-ansi-sequences-1
+                    (org-element-property :contents-end item)
+                    ansi-context))
+                 (goto-char (org-element-property :end item))
+                 (skip-chars-forward " \t\n")))))
+          (`table
+           (if (funcall skip-to-end-p el)
+               (goto-char (org-element-property :end el))
+             (goto-char (org-element-property :contents-begin el))
+             ;; Move to within the table-row of a table to continue
+             ;; processing it.
+             (forward-char)))
+          ;; Lesser elements
+          (`table-row
+           (if (eq (org-element-property :type el) 'rule)
+               (goto-char (org-element-property :end el))
+             (let ((end-1 (1- (org-element-property :end el))))
+               (goto-char (org-element-property :contents-begin el))
+               (while (< (point) end-1)
+                 (let ((cell (org-element-context)))
+                   (org-ansi-process-region
+                    (org-element-property :contents-begin cell)
+                    (org-element-property :contents-end cell)
+                    ansi-context)
+                   (goto-char (org-element-property :end cell))))
+               (forward-char))))
+          ((or `example-block `export-block)
+           (org-ansi-process-block el ansi-context)
+           (goto-char (org-element-property :end el)))
+          (`fixed-width
+           (org-ansi-process-fixed-width el ansi-context)
+           (goto-char (org-element-property :end el)))
+          (`paragraph
+           (org-ansi-process-paragraph el ansi-context)
+           (goto-char (org-element-property :end el)))
+          (_
+           (goto-char (org-element-property :end el))))))))
+
+(defvar org-ansi-mode)
+
+(defun org-fontify-ansi-sequences (limit)
+  "Fontify ANSI sequences."
+  (when (and org-fontify-ansi-sequences org-ansi-mode)
+    (while (< (point) limit)
+      (if (re-search-forward ansi-color-control-seq-regexp limit t)
+          (let* ((ctx (progn
+                        (goto-char (match-beginning 0))
+                        (org-element-context)))
+                 (type (org-element-type ctx)))
+            (cond
+             ((memq type org-ansi-highlightable-objects)
+              ;; If the element-context is an object then there has not
+              ;; been a sequence at the element level so limit the
+              ;; effect of the sequence to the object.
+              (org-ansi-process-region
+               (point)
+               (or (org-element-property :contents-end ctx)
+                   (- (org-element-property :end ctx)
+                      (org-element-property :post-blank ctx)
+                      1))
+               (org-ansi-new-context (point)))
+              (goto-char (org-element-property :end ctx)))
+             ((memq type org-ansi-highlightable-elements)
+              (let ((el ctx))
+                (while (and el (not (org-element-property :results el)))
+                  (setq el (org-element-property :parent el)))
+                (if (and el (not (eq el ctx)))
+                    ;; If the element-context is a highlightable element
+                    ;; that has an ancestor with a RESULTS affiliated
+                    ;; keyword, process the full greater element with
+                    ;; that keyword.
+                    (if (not (memq (org-element-type el) org-ansi-highlightable-elements))
+                        ;; Skip over the greater element if not
+                        ;; highlightable.
+                        (goto-char (org-element-property :end el))
+                      (goto-char (org-element-property :begin el))
+                      (org-fontify-ansi-sequences-1
+                       (or (org-element-property :contents-end el)
+                           (org-element-property :end el))
+                       (org-ansi-new-context (point)))
+                      (goto-char (org-element-property :end el)))
+                  ;; If the element-context is not a part of a greater
+                  ;; element with a RESULTS affiliated keyword, then it
+                  ;; is just a highlightable lesser element.  Process
+                  ;; the element.
+                  (pcase type
+                    ((or `example-block `export-block)
+                     (org-ansi-process-block ctx))
+                    (`fixed-width
+                     (org-ansi-process-fixed-width ctx))
+                    (`paragraph
+                     (org-ansi-process-paragraph ctx)))
+                  (goto-char (org-element-property :end ctx)))))
+             (t
+              (pcase type
+                ((or `headline `inlinetask)
+                 (goto-char (or (org-element-property :contents-begin ctx)
+                                (org-element-property :end ctx))))
+                (_
+                 (goto-char (org-element-property :end ctx)))))))
+        (goto-char limit)))))
+
 (defun org-activate-footnote-links (limit)
   "Add text properties for footnotes."
   (let ((fn (org-footnote-next-reference-or-definition limit)))
@@ -5915,6 +6124,7 @@ (defun org-set-font-lock-defaults ()
 	  ;; Blocks and meta lines
 	  '(org-fontify-meta-lines-and-blocks)
           '(org-fontify-inline-src-blocks)
+          '(org-fontify-ansi-sequences)
           ;; Citations.  When an activate processor is specified, if
           ;; specified, try loading it beforehand.
           (progn
@@ -15582,6 +15792,32 @@ (defun org-agenda-prepare-buffers (files)
     (when org-agenda-file-menu-enabled
       (org-install-agenda-files-menu))))
 
+\f
+;;;; ANSI minor mode
+
+(define-minor-mode org-ansi-mode
+  "Toggle the minor `org-ansi-mode'.
+This mode adds support to highlight ANSI sequences in Org mode.
+The sequences are highlighted only if the customization
+`org-fontify-ansi-sequences' is non-nil when the mode is enabled.
+\\{org-ansi-mode-map}"
+  :lighter " OANSI"
+  (org-restart-font-lock)
+  (unless org-ansi-mode
+    (org-with-wide-buffer
+     (goto-char (point-min))
+     (while (re-search-forward ansi-color-control-seq-regexp nil t)
+       (let ((beg (match-beginning 0))
+             (end (point)))
+         (dolist (ov (overlays-at beg))
+           (when (and (= beg (overlay-start ov))
+                      (= end (overlay-end ov))
+                      (plist-get (overlay-properties ov) 'invisible))
+             ;; Assume this is the overlay added by `ansi-color-apply-on-region'
+             (delete-overlay ov))))))))
+
+(add-hook 'org-mode-hook #'org-ansi-mode)
+
 \f
 ;;;; CDLaTeX minor mode
 
-- 
2.39.1


[-- Attachment #4: Type: text/plain, Size: 15 bytes --]


-- 
Nathaniel