Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Oleg Sivokon]

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     http://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org-mode mailing list.
------------------------------------------------------------------------

Hello,

I've been having this problem for a while, and now I think I found its
culprit: The org-babel-ref-parse function will try to move the point in
a wrong buffer (a temporary buffer with the contents of the buffer being
exported), while doing so, it will not switch to this temporary buffer.
Below is what I did to deal with it:

(defun org-babel-ref-parse (assignment)
  "Parse a variable ASSIGNMENT in a header argument.
If the right hand side of the assignment has a literal value
return that value, otherwise interpret as a reference to an
external resource and find its value using
`org-babel-ref-resolve'.  Return a list with two elements.  The
first element of the list will be the name of the variable, and
the second will be an emacs-lisp representation of the value of
the variable."
  (when (string-match org-babel-ref-split-regexp assignment)
    (let ((var (match-string 1 assignment))
	  (ref (match-string 2 assignment)))
      (cons (intern var)
	    (let ((out (save-excursion
			 (when org-babel-current-src-block-location
               ;; Added this line, to avoid
               ;; "Marker points into wrong buffer" error
			   (set-buffer (marker-buffer org-babel-current-src-block-location))
			   (goto-char org-babel-current-src-block-location))
			 (org-babel-read ref))))
	      (if (equal out ref)
		  (if (string-match "^\".*\"$" ref)
		      (read ref)
		    (org-babel-ref-resolve ref))
		out))))))

Best,

Oleg

Emacs  : GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.14.12)
 of 2015-05-03 on wvxvvw-laptop
Package: Org-mode version 8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)

current state:
==============
(setq
 org-export-backends '(ascii html icalendar latex man texinfo)
 org-tab-first-hook '(org-hide-block-toggle-maybe org-src-native-tab-command-maybe
                      org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 org-latex-classes '(("djcb-org-article"
                      "\\documentclass[11pt,a4paper]{article}\n\\usepackage[T1]{fontenc}\n\\usepackage{fontspec}\n\\usepackage{graphicx}\n\\usepackage{hyperref}\n\\usepackage[normalem]{ulem}\n\\defaultfontfeatures{Mapping=tex-text}\n\\setromanfont{Gentium}\n\\setromanfont [BoldFont={Gentium Basic Bold},\n                ItalicFont={Gentium Basic Italic}]{Gentium Basic}\n\\setsansfont{Charis SIL Compact}\n\\setmonofont[Scale=0.8]{DejaVuSansMono}\n\\usepackage{geometry}\n\\geometry{a4paper, textwidth=6.5in, textheight=10in,\n            marginparsep=7pt, marginparwidth=.6in}\n\\pagestyle{empty}\n\\title{}\n      [NO-DEFAULT-PACKAGES]\n      [NO-PACKAGES]"
                      ("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" . "\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}") ("\\paragraph{%s}" . "\\paragraph*{%s}")
                      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("koma-article" "\\documentclass{scrartcl}" ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
                      ("\\paragraph{%s}" . "\\paragraph*{%s}") ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("org-article"
                      "\\documentclass{org-article}\n             [NO-DEFAULT-PACKAGES]\n             [PACKAGES]\n             [EXTRA]"
                      ("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" . "\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}") ("\\paragraph{%s}" . "\\paragraph*{%s}")
                      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("memoir"
                      "\\documentclass[oneside]{memoir}\n             [DEFAULT-PACKAGES]\n             [PACKAGES]\n             [EXTRA]"
                      ("\\chapter{%s}" . "\\chapter*{%s}") ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
                      ("\\paragraph{%s}" . "\\paragraph*{%s}") ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("article" "\\documentclass[11pt]{article}" ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
                      ("\\paragraph{%s}" . "\\paragraph*{%s}") ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("report" "\\documentclass[11pt]{report}" ("\\part{%s}" . "\\part*{%s}")
                      ("\\chapter{%s}" . "\\chapter*{%s}") ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
                     ("book" "\\documentclass[11pt]{book}" ("\\part{%s}" . "\\part*{%s}")
                      ("\\chapter{%s}" . "\\chapter*{%s}") ("\\section{%s}" . "\\section*{%s}")
                      ("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
                     )
 org-speed-command-hook '(org-speed-command-default-hook org-babel-speed-command-hook)
 org-occur-hook '(org-first-headline-recenter)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-confirm-shell-link-function 'yes-or-no-p
 org-latex-pdf-process '("latexmk -pdflatex='pdflatex -shell-escape -interaction nonstopmode' -pdf -f %f")
 org-latex-format-headline-function 'org-latex-format-headline-default-function
 org-default-notes-file "~/org/notes.org"
 org-babel-latex-htlatex "htlatex"
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-from-is-user-regexp "\\<Oleg Sivokon\\>"
 org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer)
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-babel-pre-tangle-hook '(save-buffer)
 org-mode-hook '(#[nil "\300\301\302\303\304$\207"
                   [org-add-hook change-major-mode-hook org-show-block-all append local] 5]
                 #[nil "\300\301\302\303\304$\207"
                   [org-add-hook change-major-mode-hook org-babel-show-result-all append local] 5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point org-babel-execute-safely-maybe)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers org-cycle-hide-inline-tasks
                  org-cycle-show-empty-lines org-optimize-window-after-visibility-change)
 org-plantuml-jar-path "/home/wvxvw/Projects/org-mode/contrib/scripts/plantuml.jar"
 org-latex-packages-alist '(("AUTO" "inputenc" t))
 org-babel-tangle-lang-exts '(("latex" . "tex") ("maxima" . "max") ("python" . "py") ("lisp" . "lisp")
                              ("haxe" . "hx") ("prolog" . "pl") ("emacs-lisp" . "el"))
 org-confirm-elisp-link-function 'yes-or-no-p
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-latex-listings t
 org-babel-load-languages '((haxe . t) (emacs-lisp . t) (plantuml . t) (lisp . t) (prolog . t) (python . t)
                            (dot . t) (maxima . t) (latex . t) (R . t) (sql . t) (sqlite . t) (sh . t))
 org-agenda-files '("~/.org/tasks.org" "~/Projects/walla/README.org")
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-src-fontify-natively t
 org-publish-project-alist '(("example" :base-directory "~/org/example" :base-extension "org" :publishing-directory
                              "~/org/out/example" :publishing-function org-html-publish-to-html :setupfile
                              "~/org/setupfile.org")
                             )
 )

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Nicolas Goaziou]

Hello,

Oleg Sivokon <olegsivokon@gmail.com> writes:

> I've been having this problem for a while, and now I think I found its
> culprit: The org-babel-ref-parse function will try to move the point in
> a wrong buffer (a temporary buffer with the contents of the buffer being
> exported), while doing so, it will not switch to this temporary buffer.
> Below is what I did to deal with it:
>
> (defun org-babel-ref-parse (assignment)
>   "Parse a variable ASSIGNMENT in a header argument.
> If the right hand side of the assignment has a literal value
> return that value, otherwise interpret as a reference to an
> external resource and find its value using
> `org-babel-ref-resolve'.  Return a list with two elements.  The
> first element of the list will be the name of the variable, and
> the second will be an emacs-lisp representation of the value of
> the variable."
>   (when (string-match org-babel-ref-split-regexp assignment)
>     (let ((var (match-string 1 assignment))
> 	  (ref (match-string 2 assignment)))
>       (cons (intern var)
> 	    (let ((out (save-excursion
> 			 (when org-babel-current-src-block-location
>                ;; Added this line, to avoid
>                ;; "Marker points into wrong buffer" error
> 			   (set-buffer (marker-buffer org-babel-current-src-block-location))
> 			   (goto-char org-babel-current-src-block-location))
> 			 (org-babel-read ref))))

Using `with-current-buffer' is more appropriate since the change is
temporary.


Could you provide a patch using git format-patch, with a proper commit
message?

Thank you.


Regards,

-- 
Nicolas Goaziou

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Oleg Sivokon]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Using `with-current-buffer' is more appropriate since the change is
> temporary.
>
>
> Could you provide a patch using git format-patch, with a proper commit
> message?

Hello Nicolas,

Please let me know if I didn't export the patch properly (patch
attached).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: marker in wrong buffer patch --]
[-- Type: text/x-patch, Size: 993 bytes --]

From fdc1409602b48ba5f4cd1b5be6264d8311b8b11c Mon Sep 17 00:00:00 2001
From: wvxvw <olegsivokon@gmail.com>
Date: Fri, 8 May 2015 00:14:11 +0300
Subject: [PATCH] Making sure the buffer pointed by
 org-babel-current-src-block-location is active when moving to the source
 block.

---
 lisp/ob-ref.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-ref.el b/lisp/ob-ref.el
index b8a921e..81d4471 100644
--- a/lisp/ob-ref.el
+++ b/lisp/ob-ref.el
@@ -90,7 +90,9 @@ the variable."
 	    (let ((out (save-excursion
 			 (when org-babel-current-src-block-location
 			   (goto-char (if (markerp org-babel-current-src-block-location)
-					  (marker-position org-babel-current-src-block-location)
+					  (with-current-buffer
+					      (marker-buffer org-babel-current-src-block-location)
+					    (marker-position org-babel-current-src-block-location))
 					org-babel-current-src-block-location)))
 			 (org-babel-read ref))))
 	      (if (equal out ref)
-- 
2.1.0


[-- Attachment #3: Type: text/plain, Size: 13 bytes --]


Best,

Oleg

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Nicolas Goaziou]

Oleg Sivokon <olegsivokon@gmail.com> writes:

> Please let me know if I didn't export the patch properly (patch
> attached).

Thank you. Some comments follow.

> From fdc1409602b48ba5f4cd1b5be6264d8311b8b11c Mon Sep 17 00:00:00 2001
> From: wvxvw <olegsivokon@gmail.com>
> Date: Fri, 8 May 2015 00:14:11 +0300
> Subject: [PATCH] Making sure the buffer pointed by
>  org-babel-current-src-block-location is active when moving to the source
>  block.

The summary line should be shorter.  Also you need to add a reference to
the function being modified, e.g.,

* list/ob-ref.el (org-babel-ref-parse): Make sure the buffer pointed by...

Eventually, you need to add "TINYCHANGE" string at the end of the commit
message if you haven't signed FSF papers.  See
<http://orgmode.org/worg/org-contribute.html#orgheadline8> for details.

>  			   (goto-char (if (markerp org-babel-current-src-block-location)
> -					  (marker-position org-babel-current-src-block-location)
> +					  (with-current-buffer
> +					      (marker-buffer org-babel-current-src-block-location)
> +					    (marker-position org-babel-current-src-block-location))
>  					org-babel-current-src-block-location)))
>  			 (org-babel-read ref))))

This won't work as `org-babel-read' is not called within the scope of
`with-current-buffer'.

I think it should be something like this:

  (with-current-buffer
      (if (markerp org-babel-current-src-block-location)
          (marker-buffer org-babel-current-src-block-location)
        (current-buffer))
    (save-excursion
      (goto-char org-babel-current-src-block-location)
      (org-babel-read ref)))

Bonus points if you can write a test about it in "test-ob.el".

Regards,

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Oleg Sivokon]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> This won't work as `org-babel-read' is not called within the scope of
> `with-current-buffer'.
>
> I think it should be something like this:
>
>   (with-current-buffer
>       (if (markerp org-babel-current-src-block-location)
>           (marker-buffer org-babel-current-src-block-location)
>         (current-buffer))
>     (save-excursion
>       (goto-char org-babel-current-src-block-location)
>       (org-babel-read ref)))
>
> Bonus points if you can write a test about it in "test-ob.el".

Hi Nicolas,

I've started working on the test as I realized that the behavior no
longer reproduces (since I've filed the bug, I've rebuilt my
Emacs...).  Now even if the marker is initially in the wrong buffer, if
you call `marker-position', it still succeeds.  I checked the history of
`marker-position' source and `CHECK_MARKER' source, but there was no
change there recent enough to suspect them to be the reason of the
behavior I've seen.  I've tried going back to Emacs 24.5 (the one
installed from distribution package manager), but it's not compatible
with the Org version that I've build from the master yesterday.  If this
is important, it is incompatible in that ox-latex expects there to be a
`cl-struct-define' macro, but it probably didn't exist yet in the 24.5
times.

So, I'm not sure, is this still relevant?

Best,

Oleg

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Nicolas Goaziou]

Oleg Sivokon <olegsivokon@gmail.com> writes:

> I've started working on the test as I realized that the behavior no
> longer reproduces (since I've filed the bug, I've rebuilt my
> Emacs...).  Now even if the marker is initially in the wrong buffer, if
> you call `marker-position', it still succeeds.  I checked the history of
> `marker-position' source and `CHECK_MARKER' source, but there was no
> change there recent enough to suspect them to be the reason of the
> behavior I've seen.

Odd. Anyway, if you can't reproduce it, I guess there's nothing to fix
at the moment.

> I've tried going back to Emacs 24.5 (the one installed from
> distribution package manager), but it's not compatible with the Org
> version that I've build from the master yesterday.

It should be. Development version currently tries to be compatible with
both Emacs 23 and Emacs 24. How does this incompatibility manifest
itself?

> If this is important, it is incompatible in that ox-latex expects
> there to be a `cl-struct-define' macro, but it probably didn't exist
> yet in the 24.5 times.

Do you mean `defstruct'? If so, it has been in cl.el for ages. I doubt
this is related to the incompatibility you are experiencing.

Regards,

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Oleg Sivokon]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Hello, Nicolas.

> Do you mean `defstruct'? If so, it has been in cl.el for ages. I doubt
> this is related to the incompatibility you are experiencing.

Yes, that's `defstruct'. Its ABI changed between 24.5 and 25.0, The
older one had this in the end:

    (push `(setq ,tag-symbol (list ',tag)) forms)
    (push `(cl-eval-when (compile load eval)
             (put ',name 'cl-struct-slots ',descs)
             (put ',name 'cl-struct-type ',(list type (eq named t)))
             (put ',name 'cl-struct-include ',include)
             (put ',name 'cl-struct-print ,print-auto)
             ,@(mapcar (lambda (x)
                         `(put ',(car x) 'side-effect-free ',(cdr x)))
                       side-eff))
          forms)
    `(progn ,@(nreverse (cons `',name forms)))))

And the younger one looks like this:

    `(progn
       (defvar ,tag-symbol)
       ,@(nreverse forms)
       ;; Call cl-struct-define during compilation as well, so that
       ;; a subsequent cl-defstruct in the same file can correctly include this
       ;; struct as a parent.
       (eval-and-compile
         (cl-struct-define ',name ,docstring ',include-name
                           ',type ,(eq named t) ',descs ',tag-symbol ',tag
                           ',print-auto))
       ',name)))

So, if I compile Org against the CL library from Emacs 25.0, it's not
backwards compatible with Emacs 24.5.  I could put a different compiled
version elsewhere for the older Emacs and will experiment with it.

The reason I thought that compatibility is no longer needed is because
Linux distributions usually package the bytecompiled Emacs Lisp
code. So, if someone was to get a newer version of Org, compiled with
the younger CL, that wouldn't work for them either.

Best,

Oleg

Re: Bug: org-babel-ref-parse needs to set current buffer to the buffer of the marker [8.2.2 (release_8.2.2-188-gc57372 @ /home/wvxvw/Projects/org-mode/lisp/)]

[Nicolas Goaziou]

Oleg Sivokon <olegsivokon@gmail.com> writes:

> Yes, that's `defstruct'. Its ABI changed between 24.5 and 25.0, The
> older one had this in the end:
>
>     (push `(setq ,tag-symbol (list ',tag)) forms)
>     (push `(cl-eval-when (compile load eval)
>              (put ',name 'cl-struct-slots ',descs)
>              (put ',name 'cl-struct-type ',(list type (eq named t)))
>              (put ',name 'cl-struct-include ',include)
>              (put ',name 'cl-struct-print ,print-auto)
>              ,@(mapcar (lambda (x)
>                          `(put ',(car x) 'side-effect-free ',(cdr x)))
>                        side-eff))
>           forms)
>     `(progn ,@(nreverse (cons `',name forms)))))
>
> And the younger one looks like this:
>
>     `(progn
>        (defvar ,tag-symbol)
>        ,@(nreverse forms)
>        ;; Call cl-struct-define during compilation as well, so that
>        ;; a subsequent cl-defstruct in the same file can correctly include this
>        ;; struct as a parent.
>        (eval-and-compile
>          (cl-struct-define ',name ,docstring ',include-name
>                            ',type ,(eq named t) ',descs ',tag-symbol ',tag
>                            ',print-auto))
>        ',name)))

Thanks for the info.

> So, if I compile Org against the CL library from Emacs 25.0, it's not
> backwards compatible with Emacs 24.5.  I could put a different compiled
> version elsewhere for the older Emacs and will experiment with it.

I doubt this is a problem in practice. If you get Org from ELPA, it
compiles files with your current Emacs. If you get it from an archive or
directly from git repository, you have to compile files yourself anyway.

> The reason I thought that compatibility is no longer needed is because
> Linux distributions usually package the bytecompiled Emacs Lisp
> code. So, if someone was to get a newer version of Org, compiled with
> the younger CL, that wouldn't work for them either.

Using Linux distribution packaging system is not among suggested
installation methods in the manual.


Regards,