[PATCH v2] Fix background color of latex previews

[Roshan Shariff <roshan.shariff@gmail.com>]

This change addresses two issues:

1. Latex previews in headings have the background color of the
fontified Latex code, rather than the rest of the heading.

2. If Latex code is fontified with a face that has the :extend
attribute, and the preview overlay wraps to the next line, then the
empty space after the end of the line uses the background color of the
Latex code rather than that of the surrounding text.

For example, setting org-highlight-latex-and-related to '(native)
applies the org-block face to Latex code.  Since the default heading
faces do not set :background, the org-block background is used for
preview overlays in headings.  And since org-block has :extend t, its
background color is also used for the empty space after the end of the
line preceding a wrapped latex preview.

With this change, we create a new anonymous face that inherits from
the default face and sets the :extend attribute, and use it as a
fallback for any attributes unspecified in the faces around the Latex
code.

1. Since the fallback face inherits from default, it is guaranteed to
be completely specified.  The preview overlay thus appears exactly
like the text around it; any unspecified attributes are taken from the
default face rather than the fontified Latex code.

2. When Emacs displays the empty space at the end of a line, it only
considers faces with a non-nil :extend attribute.  Since the fallback
face sets :extend, so will the merged preview overlay face.  Thus the
background color of the empty space will be taken from the surrounding
text face if it sets :extend, or the default face otherwise.
---

Please ignore the previous patch and look at this one instead :)

* I've fixed a bug in the org-latex-preview--face-with-fallback
function.

* I also noticed that the patch fixes a broader issue with background
colors for previews in headings.  I've updated the commit message to
describe both issues.

 lisp/org-latex-preview.el | 51 +++++++++++++++++++++++++++++++++------
 lisp/org.el               |  5 ++--
 2 files changed, 46 insertions(+), 10 deletions(-)

diff --git a/lisp/org-latex-preview.el b/lisp/org-latex-preview.el
index 03c198e32..96ffe894e 100644
--- a/lisp/org-latex-preview.el
+++ b/lisp/org-latex-preview.el
@@ -505,8 +505,7 @@ overlay face is set to `org-latex-preview-processing-face'."
         (overlay-put
          ov 'hidden-face
          (or (and errors 'error)
-             (org-latex-preview--face-around
-              (overlay-start ov) (overlay-end ov)))))
+             (org-latex-preview--overlay-face ov))))
        (errors
         (overlay-put
          ov 'before-string
@@ -526,8 +525,7 @@ overlay face is set to `org-latex-preview-processing-face'."
 (defun org-latex-preview--face-around (start end)
   "Return the relevant face symbol around the region START to END.
 A relevant face symbol before START is prefered, with END
-examined if none could be found, and finally the default face
-used as the final fallback.
+examined if none could be found.
 Faces in `org-latex-preview--ignored-faces' are ignored."
   (or (and (> start (point-min))
            (not (eq (char-before start) ?\n))
@@ -544,8 +542,47 @@ Faces in `org-latex-preview--ignored-faces' are ignored."
               ((consp face)
                (cl-set-difference face org-latex-preview--ignored-faces))
               ((not (memq face org-latex-preview--ignored-faces))
-               face))))
-      'default))
+               face))))))
+
+(defun org-latex-preview--face-with-fallback (face fallback)
+  "Return a face spec combining FACE with FALLBACK.
+FACE should be valid as a value of the `face' text or overlay
+property, as specified in Info node `(elisp)Special properties'.
+The return value will also be valid as a `face' text or overlay
+property.  FALLBACK should be a either a face name (symbol or
+string) or an anonymous face (plist); see the Info
+node `(elisp)Faces'."
+  (pcase face
+    ('nil
+     fallback)
+    (`(:filtered ,filter ,face)
+     `(:filtered ,filter ,(org-latex-preview--face-with-fallback face fallback)))
+    (`(foreground-color . ,color)
+     `((:foreground ,color) ,fallback))
+    (`(background-color . ,color)
+     `((:background ,color) ,fallback))
+    (`(,(pred (not keywordp)) . ,_)
+     (append face (list fallback)))
+    (_
+     (list face fallback))))
+
+(defun org-latex-preview--overlay-face (ov)
+  "Return `face' property to be used for preview overlay OV.
+Return the face found by `org-latex-preview--face-around', with
+unset properties falling back to an anonymous face that inherits
+from `default' and sets the `:extend' attribute.
+
+The fallback ensures that if the overlay wraps to the next line
+and the text around the overlay has a face without `:extend', the
+empty space after the end of the line uses the default background
+color.  Otherwise, the space would take on the background color
+of the latex code itself if it has a face with `:extend'.  For
+example, when `org-highlight-latex-and-related' includes
+`native', latex code is fontified with the `org-block' face."
+  (org-latex-preview--face-with-fallback
+   (org-latex-preview--face-around
+    (overlay-start ov) (overlay-end ov))
+   '(:inherit default :extend t)))
 
 ;; Code for `org-latex-preview-auto-mode':
 ;;
@@ -1479,7 +1516,7 @@ is either the substring between BEG and END or (when provided) VALUE."
 
 (defun org-latex-preview--colors-around (start end)
   "Find colors for LaTeX previews occuping the region START to END."
-  (let* ((face (org-latex-preview--face-around start end))
+  (let* ((face (or (org-latex-preview--face-around start end) 'default))
          (fg (pcase (plist-get org-latex-preview-options :foreground)
                ('auto
                 (org-latex-preview--resolved-faces-attr face :foreground))
diff --git a/lisp/org.el b/lisp/org.el
index 6cc8f1a07..b76f551e8 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -5439,7 +5439,7 @@ Result depends on variable `org-highlight-latex-and-related'."
 			   "\\|"))))
 
 (defvar org-latex-preview-options) ; Defined in org-latex-preview.el.
-(declare-function org-latex-preview--face-around "org-latex-preview" (start end))
+(declare-function org-latex-preview--overlay-face "org-latex-preview" (ov))
 
 (defun org-do-latex-and-related (limit)
   "Highlight LaTeX snippets and environments, entities and sub/superscript.
@@ -5490,8 +5490,7 @@ highlighting was done, nil otherwise."
                              (overlay-get ov 'face)
                              (not (eq (overlay-get ov 'face) 'error)))
                     (overlay-put ov 'face
-                                 (org-latex-preview--face-around
-                                  (overlay-start ov) (overlay-end ov))))))
+                                 (org-latex-preview--overlay-face ov)))))
 	      (throw 'found t)))))
 	nil))))
 
-- 
2.41.0