Re: [RFC] [export] synctex support for ox-latex

[Aaron Ecay <aaronecay@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]

Hi Rasmus,

Thank you very much for testing the patch!

2013ko urriak 29an, Rasmus-ek idatzi zuen:

[...]

> It doesn't work for me however.  Or perhaps I just don't get it.  I am
> expecting it to work similarly to AUCTeX-synctex and other non-Emacs
> editors supporting synctex.  E.g. a red box usually pops up,
> highlighting the correct line and I'm able to jump back from the pdf
> to the source.  Neither works.

Indeed, the jump from emacs -> pdf is not implemented.  I’ve got some
complicated dbus voodoo (attached to this email, in case it is useful
for you or others) in my init.el to get that working with AucTeX – I am
also using evince as the viewer).  So I will have to investigate how to
add that feature.

The jump from pdf viewer -> org should be working though, assuming it
does for you in auctex.  (I also have some dbus arcana for that portion
in my init.el file; I assume you do too).

Ideally, I think ox-synctex should eventually include the configurations
for as many pdf viewers as possible, so that users do not need to
configure this themselves.  (Ideally x 2, some external library would
already provide this...)

> 
> I tested it with emacs -q.  
> 
> Note, there should probably be some require statement in the top of
> your file.  ox-latex would work, but you might only need org-element.

This is right; I added both just to be safe (and dired, which was also
needed since I borrow a defcustom from there).

One thing I forgot to mention in my last email is that you need to
customize your org-latex-pdf-process to make the latex compiler generate
a synctex file.  (This is a command line switch, but it differs across
(pdf/xe/lua)tex.)  So this could be why it does not work for you in
emacs -q.  The following bit of code suffices, after M-x load-library
ox-latex:

(setq org-latex-pdf-process 
      '("pdflatex -synctex=1 -interaction nonstopmode -output-directory %o %f"))

I attach another version of the patch, with the missing requires as well
as some more logging code.  If it doesn’t work for you, could you
perhaps send me the lines in *Messages* generated by the export?


[-- Attachment #2: evince.el --]
[-- Type: application/emacs-lisp, Size: 2381 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-add-synctex-support.patch --]
[-- Type: text/x-diff, Size: 10279 bytes --]

From 05333e2cde7520d81daec986bdab3235a5d0c348 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Wed, 23 Oct 2013 15:29:56 -0400
Subject: [PATCH] add synctex support

* contrib/lisp/ox-synctex.el: new file
---
 contrib/lisp/ox-synctex.el | 260 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 260 insertions(+)
 create mode 100644 contrib/lisp/ox-synctex.el

diff --git a/contrib/lisp/ox-synctex.el b/contrib/lisp/ox-synctex.el
new file mode 100644
index 0000000..e0ef4a3
--- /dev/null
+++ b/contrib/lisp/ox-synctex.el
@@ -0,0 +1,260 @@
+;;; ox-synctex.el --- Synctex functionality for org LaTeX export
+
+;; Copyright (C) 2013 Aaron Ecay
+
+;; Author: Aaron Ecay <aaronecay@gmail.com>
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; This code provides synctex support for org mode export to latex.
+;; To activate, execute (ox-synctex-activate).  To deactivate,
+;; (ox-synctex-deactivate)
+
+;; TODOs:
+;; - support multi-file documents through #+include and friends
+;; - do something so that clicks on a minted source code block go to
+;;   the .org file and not the .pyg intermediate
+;; - ...
+
+;;; Code:
+
+(require 'org-element)
+(require 'ox-latex)
+(require 'dired) ;; for `dired-touch-program'
+
+;;;; Internal functions and variable
+
+(defvar ox-synctex--concordance nil
+  "The concordance resulting from the last export operation.")
+
+(defun ox-synctex--read-concordance (concordance src-line)
+  "Get the output line number from CONCORDANCE for input line SRC-LINE."
+  ;; TODO: not robust against malformed concordances
+  (while (and (caadr concordance)
+              (<= (caadr concordance) src-line))
+    (setq concordance (cdr concordance)))
+  (cdar concordance))
+
+(defun ox-synctex--propertize-buffer ()
+  "Put line-number text properties on a buffer.
+
+Each line gets a org-line-num-pre property, which is its line
+number in the buffer.  When export operations change the buffer,
+the text property will still reflect the original state of affairs."
+  (save-restriction
+    (widen)
+    (while (= 0 (forward-line 1))
+      (put-text-property (point) (point-at-eol)
+			 'ox-synctex-line-num
+			 (line-number-at-pos)))))
+
+(defun ox-synctex--line-number-at-pos (pos)
+  "Return the buffer line number at POS, widening if necessary.
+
+This function first looks for text properties set by
+`ox-synctex--propertize-buffer' which allow it to return an
+accurate line number in a buffer copy modified during export.  It
+falls back to the usual method of calculating line numbers if no
+text properties are found."
+  (or (get-text-property pos 'ox-synctex-line-num)
+      (save-excursion
+        (widen)
+        (line-number-at-pos pos))))
+
+(defun ox-synctex--add-line-to-element (element)
+  "Add begin and end line numbers to an element as returned by `org-element'."
+  (let* ((plist (cadr element))
+	 (beg (plist-get plist :begin))
+	 (end (plist-get plist :end)))
+    (and beg (plist-put plist :begin-line (ox-synctex--line-number-at-pos beg)))
+    (and end (plist-put plist :end-line (ox-synctex--line-number-at-pos end)))
+    element))
+
+
+(defun ox-synctex--propertize-string (data string)
+  "Add line number text properties to STRING, based on DATA.
+
+The function works by copying the properties added by
+`ox-synctex--add-line-to-element' to the string.  This will allow
+the construction of a concordance from the exported string."
+  (let ((len (length string)))
+    (when (> len 1)
+      (put-text-property 0 1 'org-line-num
+                         (org-element-property :begin-line data)
+                         string)
+      (put-text-property (1- len) len 'org-line-num
+                         (org-element-property :end-line data)
+                         string)))
+  string)
+
+(defun ox-synctex--build-concordance ()
+  "Build a concordance, based on text properties added by
+`ox-synctex--propertize-string' and accumulated in in an export
+result buffer.
+
+Has the form ((OUT-LINE . IN-LINE) ...)"
+  (save-excursion
+    (let ((res '())
+          next)
+      (goto-char (point-min))
+      (while (setq next (next-single-property-change (point) 'org-line-num))
+        (goto-char next)
+	(let ((ln (get-text-property (point) 'org-line-num)))
+	  ;; TODO: `ln' should never be nil, but sometimes it is.  For
+	  ;; now, we hack around that with this `when'.
+	  (when ln
+	    (setq res (cons (cons (line-number-at-pos) ln)
+			    res))))
+        (forward-char 1))
+      (setq res (nreverse res))
+      (setq next res)
+      (while (cdr next)
+        (if (equal (caar next) (caadr next))
+            (setcdr next (cddr next))
+          (setq next (cdr next))))
+      res)))
+
+(defun ox-synctex--patch-synctex (file)
+  "Patch the synctex file resulting from the last export
+operation, using the information stored in
+`ox-synctex--concordance'."
+  (let* ((file-base (file-name-nondirectory
+		     (replace-regexp-in-string "\\.tex\\'" "." file)))
+	 (synctex-file (concat file-base "synctex.gz")))
+    (cond
+     ((not ox-synctex--concordance)
+      (message "ox-synctex: no concordance, not patching."))
+     ((not (file-exists-p synctex-file))
+      (message "ox-synctex: no synctex file found, not patching."))
+     (t
+      (message "ox-synctex: patching synctex file")
+      (let* ((conc ox-synctex--concordance)
+	     (buf (find-file-noselect synctex-file)))
+	(with-current-buffer buf
+	  (let ((max-index 0)
+		the-index extra-path new-index)
+	    (goto-char (point-min))
+	    (while (re-search-forward "^Input:\\([0-9]+\\):" nil t)
+	      (setq max-index (max max-index (string-to-number (match-string 1)))))
+	    (setq new-index (number-to-string (1+ max-index)))
+	    (goto-char (point-min))
+	    (when (re-search-forward (concat "^Input:\\([0-9]+\\):\\(.*\\)"
+					     (regexp-quote file-base) "tex$")
+				     nil t)
+	      (setq the-index (string-to-number (match-string 1)))
+	      (setq extra-path (match-string 2))
+	      (goto-char (line-end-position))
+	      (insert (format "\nInput:%s:%s%sorg" new-index extra-path file-base)))
+	    (goto-char (point-min))
+	    (while (re-search-forward (format "^[vhxkgr$[()]\\(%s\\),\\([0-9]+\\):"
+					      the-index)
+				      nil t)
+	      (let ((new-line (ox-synctex--read-concordance
+			       ox-synctex--concordance
+			       (string-to-number (match-string 2)))))
+		(when new-line
+		  (replace-match new-index nil t nil 1)
+		  (replace-match (int-to-string new-line)
+				 nil t nil 2))))
+	    (save-buffer)))
+	(kill-buffer buf))))))
+
+;;;; Hooks and advice
+
+(defun ox-synctex--before-processing-hook (&rest ignore)
+  (ox-synctex--propertize-buffer))
+
+(defconst ox-synctex--parsers-to-patch
+  (append org-element-greater-elements org-element-all-elements))
+
+;;; Patch all `org-element' parsers to add line number info to their
+;;; return values.
+(dolist (parser ox-synctex--parsers-to-patch)
+  (let ((parser-fn (intern (format "org-element-%s-parser"
+				   (symbol-name parser)))))
+    (eval `(defadvice ,parser-fn (around ox-synctex)
+	     "Advice added by `ox-synctex'."
+	     ad-do-it
+	     (setq ad-return-value (ox-synctex--add-line-to-element
+				    ad-return-value))))))
+
+;;; Patch element->string conversion to carry through the line numbers
+;;; added above
+(defadvice org-export-transcoder (around ox-synctex)
+  ad-do-it
+  (when (and ad-return-value
+	     (org-export-derived-backend-p
+	      (plist-get (ad-get-arg 1) :back-end)
+	      'latex))
+    (setq ad-return-value
+	  `(lambda (data contents &optional info)
+	     (ox-synctex--propertize-string
+	      data
+	      (if info
+		  (,ad-return-value data contents info)
+		;; The plain text transcoder takes only 2 arguments;
+		;; here contents is really info.  I couldn't find a
+		;; better way to inspect the arity of elisp
+		;; functions...?
+		(,ad-return-value data contents)))))))
+
+;;; Patch to build the concordance once we have the export result.  We
+;;; need to hack around the fact that the original function strips
+;;; text properties from its return value which we need.
+(defadvice org-export-as (around ox-synctex)
+  (cl-letf (((symbol-function 'org-no-properties)
+	     (lambda (s &optional _restricted) s)))
+    ad-do-it)
+  (when (org-export-derived-backend-p (ad-get-arg 0) 'latex)
+    (message "ox-synctex: patching org-export-as return")
+    (with-temp-buffer
+      (insert ad-return-value)
+      (setq ox-synctex--concordance (ox-synctex--build-concordance))))
+  (setq ad-return-value (org-no-properties ad-return-value)))
+
+;;; Actually do the patching after compilation
+(defadvice org-latex-compile (around ox-synctex)
+  (message "ox-synctex: active during latex compile")
+  ad-do-it
+  (ox-synctex--patch-synctex (ad-get-arg 0))
+  ;; Some PDF viewers (eg evince) don't notice changes to the synctex
+  ;; file, so we need to poke them to reload the pdf after we've
+  ;; finished changing it.
+  (call-process dired-touch-program nil nil nil
+		(replace-regexp-in-string "\\.tex\\'" "." (ad-get-arg 0)))
+  (message "ox-synctex: done, hoorah!"))
+
+;;;; User-facing functions
+
+(defun ox-synctex-activate ()
+  (interactive)
+  (add-hook 'org-export-before-processing-hook
+            #'ox-synctex--before-processing-hook)
+  (ad-activate-regexp "ox-synctex"))
+
+(defun ox-synctex-deactivate ()
+  (interactive)
+  (remove-hook 'org-export-before-processing-hook
+	       #'ox-synctex--before-processing-hook)
+  (ad-deactivate-regexp "ox-synctex"))
+
+(provide 'ox-synctex)
+
+;; Local Variables:
+;; lexical-binding: t
+;; End:
+
+;;; ox-synctex.el ends here
-- 
1.8.4.2


[-- Attachment #4: Type: text/plain, Size: 25 bytes --]


Thanks,

-- 
Aaron Ecay