* Async Python src block behavior with :dir header property @ 2024-01-31 7:29 Nasser Alkmim 2024-01-31 11:23 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Nasser Alkmim @ 2024-01-31 7:29 UTC (permalink / raw) To: emacs-orgmode Hi, I'm in the latest Emacs GUI. This works, the =return figname= apparently can recognizee that we are in the =:dir=. #+begin_src python :dir asyncpy :return figname :results file value import matplotlib.pyplot as plt plt.figure(figsize=(4*.5, 3*.5)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname, transparent=True, bbox_inches='tight') #+end_src This does not work. Even though the python execution is in the right directory and the file is saved correctly. The =return= does not produces the right file path considering the directory. #+begin_src python :dir asyncpy :async :session :return figname :results file value import matplotlib.pyplot as plt plt.figure(figsize=(4*.5, 3*.5)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname, transparent=True, bbox_inches='tight') #+end_src If we inform the right path to the return statement, it works. #+begin_src python :dir asyncpy :async :session :epilogue figdir='asyncpy/'+figname :return figdir :results file value import matplotlib.pyplot as plt plt.figure(figsize=(4*.5, 3*.5)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname, transparent=True, bbox_inches='tight') #+end_src Another alternative, we can use the =os= package to get the current directory. #+header: :epilogue import os;figdir=os.getcwd()+'/'+figname #+begin_src python :dir asyncpy :async :session :return figdir :results file value import matplotlib.pyplot as plt plt.figure(figsize=(4*.5, 3*.5)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname, transparent=True, bbox_inches='tight') #+end_src Is this behavior as expected? Is there a better way to have the figures from the python block with async execution? -- Nasser Alkmim +43 677 6408 9171 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 7:29 Async Python src block behavior with :dir header property Nasser Alkmim @ 2024-01-31 11:23 ` Ihor Radchenko 2024-01-31 12:17 ` Nasser Alkmim 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-01-31 11:23 UTC (permalink / raw) To: Nasser Alkmim; +Cc: emacs-orgmode Nasser Alkmim <nasser.alkmim@gmail.com> writes: > This does not work. > Even though the python execution is in the right directory and the file is saved correctly. > The =return= does not produces the right file path considering the directory. > > #+begin_src python :dir asyncpy :async :session :return figname :results file value These header arguments are not correctly formatted. Try M-x org-lint -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 11:23 ` Ihor Radchenko @ 2024-01-31 12:17 ` Nasser Alkmim 2024-01-31 12:47 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Nasser Alkmim @ 2024-01-31 12:17 UTC (permalink / raw) To: emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Nasser Alkmim <nasser.alkmim@gmail.com> writes: > >> This does not work. >> Even though the python execution is in the right directory and the file is saved >> correctly. >> The =return= does not produces the right file path considering the directory. >> >> #+begin_src python :dir asyncpy :async :session :return figname :results file value > > These header arguments are not correctly formatted. > Try M-x org-lint I tried org-lint and it says that the header argument is "unknown". However, I can execute the source blocks. I tried with 'emacs -Q' and '(require 'ob-python)' and I noticed that I missed the header argument ':mkdirp "yes"'. With that I can reproduce the examples. In any case, thanks for taking a look. -- Nasser Alkmim ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 12:17 ` Nasser Alkmim @ 2024-01-31 12:47 ` Ihor Radchenko 2024-01-31 15:00 ` Nasser Alkmim 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-01-31 12:47 UTC (permalink / raw) To: Nasser Alkmim; +Cc: emacs-orgmode Nasser Alkmim <nasser.alkmim@gmail.com> writes: >>> #+begin_src python :dir asyncpy :async :session :return figname :results file value >> >> These header arguments are not correctly formatted. >> Try M-x org-lint > > I tried org-lint and it says that the header argument is "unknown". You should have got 1 low Empty value in header argument ":session" 1 low Empty value in header argument ":async" 1 nil Unknown header argument ":async" You need to specify (1) :session name; (2) :async yes. > However, I can execute the source blocks. There is no guarantee that execution will work as expected in malformed src blocks like yours. > I tried with 'emacs -Q' and '(require 'ob-python)' and I noticed that I > missed the header argument ':mkdirp "yes"'. > With that I can reproduce the examples. Do you mean that you have no problem after you add :mkdirp yes? -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 12:47 ` Ihor Radchenko @ 2024-01-31 15:00 ` Nasser Alkmim 2024-01-31 16:21 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Nasser Alkmim @ 2024-01-31 15:00 UTC (permalink / raw) To: emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Do you mean that you have no problem after you add :mkdirp yes? No. Just that I can reproduce the examples in the first message. Here is a clearer description of the unwanted behavior: 1. emacs -Q ~/Desktop/testasync.org Then evaluate (require 'ob-python) and paste this source block: #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes import matplotlib.pyplot as plt plt.figure(figsize=(1, 1)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname) #+end_src Execute the source block, which results in #+RESULTS: [[file:fig.svg]] However, the figure is in ~/Desktop/otherdir/fig.svg and the link is wrong. If I run the same block without the ':async' and ':session', I get the right relative path to the figure: #+RESULTS: [[file:otherdir/fig.svg]] Best regards, -- Nasser Alkmim +43 677 6408 9171 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 15:00 ` Nasser Alkmim @ 2024-01-31 16:21 ` Ihor Radchenko 2024-02-01 4:16 ` Jack Kamm 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-01-31 16:21 UTC (permalink / raw) To: Nasser Alkmim, Jack Kamm; +Cc: emacs-orgmode [ CCing ob-python maintainer ] Nasser Alkmim <nasser.alkmim@gmail.com> writes: > Here is a clearer description of the unwanted behavior: > > 1. emacs -Q ~/Desktop/testasync.org > > Then evaluate (require 'ob-python) and paste this source block: > > #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes > import matplotlib.pyplot as plt > plt.figure(figsize=(1, 1)) > plt.plot([1, 2]) > figname = 'fig.svg' > plt.savefig(figname) > #+end_src > > Execute the source block, which results in > > #+RESULTS: > [[file:fig.svg]] I don't even see this. Confirmed. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-01-31 16:21 ` Ihor Radchenko @ 2024-02-01 4:16 ` Jack Kamm 2024-02-01 8:40 ` Nasser Alkmim 2024-02-01 11:59 ` Ihor Radchenko 0 siblings, 2 replies; 24+ messages in thread From: Jack Kamm @ 2024-02-01 4:16 UTC (permalink / raw) To: Ihor Radchenko, Nasser Alkmim; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 976 bytes --] Ihor Radchenko <yantar92@posteo.net> writes: > [ CCing ob-python maintainer ] > > Nasser Alkmim <nasser.alkmim@gmail.com> writes: > >> Here is a clearer description of the unwanted behavior: >> >> 1. emacs -Q ~/Desktop/testasync.org >> >> Then evaluate (require 'ob-python) and paste this source block: >> >> #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes >> import matplotlib.pyplot as plt >> plt.figure(figsize=(1, 1)) >> plt.plot([1, 2]) >> figname = 'fig.svg' >> plt.savefig(figname) >> #+end_src >> >> Execute the source block, which results in >> >> #+RESULTS: >> [[file:fig.svg]] > > I don't even see this. > Confirmed. Does the attached patch fix the issue? It seems the problem is with async sessions generally (not just ob-python), and happens because `org-babel-comint-async-filter' does not set `default-directory' before calling `org-babel-insert-result', unlike `org-babel-execute-src-block'. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-comint-Make-file-results-from-async-sessions-resp.patch --] [-- Type: text/x-patch, Size: 2644 bytes --] From 1430a27e4416d5e88094a64360015a6a2ae7315c Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Wed, 31 Jan 2024 20:06:00 -0800 Subject: [PATCH] ob-comint: Make file results from async sessions respect :dir header * lisp/ob-comint.el (org-babel-comint-async-filter): Set default-directory before calling `org-babel-insert-result' https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t --- lisp/ob-comint.el | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 7d258ea0e..ecce18a95 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -248,7 +248,14 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (tmp-file (expand-file-name tmp-file)) + (dir (cdr (assq :dir params))) + (default-directory + (if dir + (file-name-as-directory + (expand-file-name dir)) + default-directory))) (org-babel-insert-result (funcall file-callback (nth @@ -291,7 +298,13 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (dir (cdr (assq :dir params))) + (default-directory + (if dir + (file-name-as-directory + (expand-file-name dir)) + default-directory))) (org-babel-insert-result res-str result-params info)) t)))) -- 2.43.0 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-01 4:16 ` Jack Kamm @ 2024-02-01 8:40 ` Nasser Alkmim 2024-02-01 11:59 ` Ihor Radchenko 1 sibling, 0 replies; 24+ messages in thread From: Nasser Alkmim @ 2024-02-01 8:40 UTC (permalink / raw) To: Jack Kamm; +Cc: Ihor Radchenko, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: > > Does the attached patch fix the issue? > I briefly tested it here and it works. Thanks for taking a look into it. -- Nasser Alkmim +43 677 6408 9171 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-01 4:16 ` Jack Kamm 2024-02-01 8:40 ` Nasser Alkmim @ 2024-02-01 11:59 ` Ihor Radchenko 2024-02-03 1:20 ` Jack Kamm 1 sibling, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-01 11:59 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: >> Confirmed. > > Does the attached patch fix the issue? > > It seems the problem is with async sessions generally (not just > ob-python), and happens because `org-babel-comint-async-filter' does not > set `default-directory' before calling `org-babel-insert-result', unlike > `org-babel-execute-src-block'. The patch generally looks reasonable, although I am slightly concerned about interaction between :dir and session we describe in the manual: When ‘dir’ is used with ‘session’, Org sets the starting directory for a new session. But Org does not alter the directory of an already existing session. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-01 11:59 ` Ihor Radchenko @ 2024-02-03 1:20 ` Jack Kamm 2024-02-03 15:51 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-03 1:20 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > The patch generally looks reasonable, although I am slightly concerned > about interaction between :dir and session we describe in the manual: > > When ‘dir’ is used with ‘session’, Org sets the starting directory > for a new session. But Org does not alter the directory of an already > existing session. I agree it's a problem -- if there are multiple blocks with the same session but different ":dir" arguments, then a ":file" result of the second block will be relative to the wrong :dir. This seems like a longstanding problem, and affects both async and non-async session blocks. Ideally, ":dir" should be set at session level rather than block level. This could be done via PROPERTY if there is only one session in the file or subtree, but there isn't a mechanism for this more generally. Perhaps the simplest solution is to add a check to org-lint, requiring all blocks with the same session to have the same :dir. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-03 1:20 ` Jack Kamm @ 2024-02-03 15:51 ` Ihor Radchenko 2024-02-04 1:26 ` Jack Kamm 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-03 15:51 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1296 bytes --] Jack Kamm <jackkamm@gmail.com> writes: > Ihor Radchenko <yantar92@posteo.net> writes: > >> The patch generally looks reasonable, although I am slightly concerned >> about interaction between :dir and session we describe in the manual: >> >> When ‘dir’ is used with ‘session’, Org sets the starting directory >> for a new session. But Org does not alter the directory of an already >> existing session. > > I agree it's a problem -- if there are multiple blocks with the same > session but different ":dir" arguments, then a ":file" result of the > second block will be relative to the wrong :dir. > > This seems like a longstanding problem, and affects both async and > non-async session blocks. Maybe something like the attached? > Ideally, ":dir" should be set at session level rather than block > level. This could be done via PROPERTY if there is only one session in > the file or subtree, but there isn't a mechanism for this more > generally. Or we can ask babel backends to respect :dir and issue the necessary "cd" command in the session for the duration of the code block evaluation. > Perhaps the simplest solution is to add a check to org-lint, requiring > all blocks with the same session to have the same :dir. Or this. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --] [-- Type: text/x-patch, Size: 1864 bytes --] From 1e87de13d44a1bb31985622bfc9c9a6df6b7548c Mon Sep 17 00:00:00 2001 Message-ID: <1e87de13d44a1bb31985622bfc9c9a6df6b7548c.1706975370.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Sat, 3 Feb 2024 16:47:57 +0100 Subject: [PATCH] org-babel-execute-src-block: Force :dir according to live session * lisp/ob-core.el (org-babel-execute-src-block): Force eval directory to follow live session buffer, if any. This is consistent with what we promise in the manual. Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com --- lisp/ob-core.el | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 1de3af6ad..2cd542012 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (dir (cdr (assq :dir params))) (mkdirp (cdr (assq :mkdirp params))) (default-directory - (cond - ((not dir) default-directory) - ((member mkdirp '("no" "nil" nil)) - (file-name-as-directory (expand-file-name dir))) - (t - (let ((d (file-name-as-directory (expand-file-name dir)))) - (make-directory d 'parents) - d)))) + (cond + ((not dir) default-directory) + ((when-let ((session (assq :session params))) + (when (org-babel-comint-buffer-livep session) + (buffer-local-value 'default-directory (get-buffer session))))) + ((member mkdirp '("no" "nil" nil)) + (file-name-as-directory (expand-file-name dir))) + (t + (let ((d (file-name-as-directory (expand-file-name dir)))) + (make-directory d 'parents) + d)))) (cmd (intern (concat "org-babel-execute:" lang))) result exec-start-time) (unless (fboundp cmd) -- 2.43.0 [-- Attachment #3: Type: text/plain, Size: 225 bytes --] -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-03 15:51 ` Ihor Radchenko @ 2024-02-04 1:26 ` Jack Kamm 2024-02-04 16:30 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-04 1:26 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 612 bytes --] Ihor Radchenko <yantar92@posteo.net> writes: >> I agree it's a problem -- if there are multiple blocks with the same >> session but different ":dir" arguments, then a ":file" result of the >> second block will be relative to the wrong :dir. >> >> This seems like a longstanding problem, and affects both async and >> non-async session blocks. > > Maybe something like the attached? Nice, that seems like the right way to do it. I updated the patch for `org-babel-comint-async-filter' to follow the same approach, setting default-directory based on the session buffer's value rather than the :dir header arg. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-comint-Make-file-results-from-async-sessions-resp.patch --] [-- Type: text/x-patch, Size: 2360 bytes --] From 1a1a22e4f4a12ebe83c3fef26fe727066fb14476 Mon Sep 17 00:00:00 2001 From: Jack Kamm <jackkamm@gmail.com> Date: Wed, 31 Jan 2024 20:06:00 -0800 Subject: [PATCH] ob-comint: Make file results from async sessions respect :dir header * lisp/ob-comint.el (org-babel-comint-async-filter): Set default-directory before calling `org-babel-insert-result' https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t --- lisp/ob-comint.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 7d258ea0e..349524701 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string) (file-callback org-babel-comint-async-file-callback) (combined-string (concat org-babel-comint-async-dangling string)) (new-dangling combined-string) + ;; Assumes comint filter called with session buffer current + (session-dir default-directory) ;; list of UUID's matched by `org-babel-comint-async-indicator' uuid-list) (with-temp-buffer @@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result (funcall file-callback (nth @@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result res-str result-params info)) t)))) -- 2.43.0 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-04 1:26 ` Jack Kamm @ 2024-02-04 16:30 ` Ihor Radchenko 2024-02-05 0:59 ` Jack Kamm 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-04 16:30 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 360 bytes --] Jack Kamm <jackkamm@gmail.com> writes: > I updated the patch for `org-babel-comint-async-filter' to follow the > same approach, setting default-directory based on the session buffer's > value rather than the :dir header arg. Thanks! Attaching the two patches combined with some fixed to my patch. Please check if these two patches solve the discussed bug. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --] [-- Type: text/x-patch, Size: 1946 bytes --] From 66bbac4207d7132ebc9f91f60384809ae556c2ef Mon Sep 17 00:00:00 2001 Message-ID: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Sat, 3 Feb 2024 16:47:57 +0100 Subject: [PATCH 1/2] org-babel-execute-src-block: Force :dir according to live session * lisp/ob-core.el (org-babel-execute-src-block): Force eval directory to follow live session buffer, if any. This is consistent with what we promise in the manual section "16.4 Environment of a Code Block">Choosing a working directory. Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com --- lisp/ob-core.el | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 1de3af6ad..fd8d06c5d 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (dir (cdr (assq :dir params))) (mkdirp (cdr (assq :mkdirp params))) (default-directory - (cond - ((not dir) default-directory) - ((member mkdirp '("no" "nil" nil)) - (file-name-as-directory (expand-file-name dir))) - (t - (let ((d (file-name-as-directory (expand-file-name dir)))) - (make-directory d 'parents) - d)))) + (cond + ((not dir) default-directory) + ((when-let ((session (cdr (assq :session params)))) + (when (org-babel-comint-buffer-livep session) + (buffer-local-value 'default-directory (get-buffer session))))) + ((member mkdirp '("no" "nil" nil)) + (file-name-as-directory (expand-file-name dir))) + (t + (let ((d (file-name-as-directory (expand-file-name dir)))) + (make-directory d 'parents) + d)))) (cmd (intern (concat "org-babel-execute:" lang))) result exec-start-time) (unless (fboundp cmd) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-ob-comint-Make-file-results-from-async-sessions-resp.patch --] [-- Type: text/x-patch, Size: 2635 bytes --] From e9765a146bf27e6769a2563862d52b0efb77eb2e Mon Sep 17 00:00:00 2001 Message-ID: <e9765a146bf27e6769a2563862d52b0efb77eb2e.1707064153.git.yantar92@posteo.net> In-Reply-To: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net> References: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net> From: Jack Kamm <jackkamm@gmail.com> Date: Wed, 31 Jan 2024 20:06:00 -0800 Subject: [PATCH 2/2] ob-comint: Make file results from async sessions respect :dir header * lisp/ob-comint.el (org-babel-comint-async-filter): Set default-directory before calling `org-babel-insert-result' https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t --- lisp/ob-comint.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 7d258ea0e..349524701 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string) (file-callback org-babel-comint-async-file-callback) (combined-string (concat org-babel-comint-async-dangling string)) (new-dangling combined-string) + ;; Assumes comint filter called with session buffer current + (session-dir default-directory) ;; list of UUID's matched by `org-babel-comint-async-indicator' uuid-list) (with-temp-buffer @@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result (funcall file-callback (nth @@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result res-str result-params info)) t)))) -- 2.43.0 [-- Attachment #4: Type: text/plain, Size: 224 bytes --] -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-04 16:30 ` Ihor Radchenko @ 2024-02-05 0:59 ` Jack Kamm 2024-02-05 14:10 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-05 0:59 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Thanks! > Attaching the two patches combined with some fixed to my patch. > > Please check if these two patches solve the discussed bug. The original bug for async sessions looks fixed. But I encountered a complication regarding the non-async bug of changing :dir. In particular the following example: #+begin_src python :dir otherdir :session pysession :return figname :results file value :mkdirp yes import matplotlib.pyplot as plt plt.figure(figsize=(1, 1)) plt.plot([1, 2]) figname = 'fig.svg' plt.savefig(figname) #+end_src #+RESULTS: [[file:otherdir/fig.svg]] #+begin_src python :dir otherdir2 :session pysession :return figname :results file value :mkdirp yes import matplotlib.pyplot as plt plt.figure(figsize=(1, 1)) plt.plot([1, 2]) figname = 'fig5.svg' plt.savefig(figname) #+end_src #+RESULTS: [[file:otherdir2/fig5.svg]] As you can see the second result points to the wrong directory. However, if replacing ":session pysession" with ":session *pysession*", then it works. It's because ob-python starts the session in buffer "*pysession*" (it adds earmuffs around the session name when missing). So the following doesn't find the inferior Python: > + ((when-let ((session (cdr (assq :session params)))) > + (when (org-babel-comint-buffer-livep session) Honestly, I'm not sure why ob-python insists on the session buffer having the earmuffs, that behavior is from before my time. In particular, I'm not yet sure if the motivation is just cosmetic (because of the general convention like *shell*, *Python*, *R*, etc), or whether it actually affects python.el behavior. But if it's just cosmetic, then we might consider removing this behavior to simplify things. Though note that ob-python is not the only one with this sort of behavior -- looks like ob-lua might behave similarly, and until recently ob-R also had some surprising behavior depending on whether the session name had earmuffs. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-05 0:59 ` Jack Kamm @ 2024-02-05 14:10 ` Ihor Radchenko 2024-02-06 6:56 ` Jack Kamm 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-05 14:10 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: > It's because ob-python starts the session in buffer "*pysession*" (it > adds earmuffs around the session name when missing). So the following > doesn't find the inferior Python: Good point. Some backends indeed do not have comint buffer named the same as :session name. What we can do is to introduce a new backend template function org-babel-session-buffer:<lang> that will be passed a session name and src block params and return the session buffer name. If such function is not defined, we fall back to assumption that session buffer is named the same as the session. WDYT? -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-05 14:10 ` Ihor Radchenko @ 2024-02-06 6:56 ` Jack Kamm 2024-02-06 14:06 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-06 6:56 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > What we can do is to introduce a new backend template function > org-babel-session-buffer:<lang> that will be passed a session name and > src block params and return the session buffer name. > > If such function is not defined, we fall back to assumption that session > buffer is named the same as the session. > > WDYT? Sounds good -- I think this is the best solution. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-06 6:56 ` Jack Kamm @ 2024-02-06 14:06 ` Ihor Radchenko 2024-02-08 3:08 ` Jack Kamm 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-06 14:06 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 502 bytes --] Jack Kamm <jackkamm@gmail.com> writes: > Ihor Radchenko <yantar92@posteo.net> writes: > >> What we can do is to introduce a new backend template function >> org-babel-session-buffer:<lang> that will be passed a session name and >> src block params and return the session buffer name. >> >> If such function is not defined, we fall back to assumption that session >> buffer is named the same as the session. >> >> WDYT? > > Sounds good -- I think this is the best solution. See the updated patchset. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --] [-- Type: text/x-patch, Size: 1946 bytes --] From b877a440c65cb83bd29182daf4c13cd9b14ed182 Mon Sep 17 00:00:00 2001 Message-ID: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Sat, 3 Feb 2024 16:47:57 +0100 Subject: [PATCH 1/3] org-babel-execute-src-block: Force :dir according to live session * lisp/ob-core.el (org-babel-execute-src-block): Force eval directory to follow live session buffer, if any. This is consistent with what we promise in the manual section "16.4 Environment of a Code Block">Choosing a working directory. Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com --- lisp/ob-core.el | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 1de3af6ad..fd8d06c5d 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (dir (cdr (assq :dir params))) (mkdirp (cdr (assq :mkdirp params))) (default-directory - (cond - ((not dir) default-directory) - ((member mkdirp '("no" "nil" nil)) - (file-name-as-directory (expand-file-name dir))) - (t - (let ((d (file-name-as-directory (expand-file-name dir)))) - (make-directory d 'parents) - d)))) + (cond + ((not dir) default-directory) + ((when-let ((session (cdr (assq :session params)))) + (when (org-babel-comint-buffer-livep session) + (buffer-local-value 'default-directory (get-buffer session))))) + ((member mkdirp '("no" "nil" nil)) + (file-name-as-directory (expand-file-name dir))) + (t + (let ((d (file-name-as-directory (expand-file-name dir)))) + (make-directory d 'parents) + d)))) (cmd (intern (concat "org-babel-execute:" lang))) result exec-start-time) (unless (fboundp cmd) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-ob-comint-Make-file-results-from-async-sessions-resp.patch --] [-- Type: text/x-patch, Size: 2635 bytes --] From a31c7b3f5f730b63d3753e07a7a5bb5e73a97ce4 Mon Sep 17 00:00:00 2001 Message-ID: <a31c7b3f5f730b63d3753e07a7a5bb5e73a97ce4.1707228351.git.yantar92@posteo.net> In-Reply-To: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net> References: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net> From: Jack Kamm <jackkamm@gmail.com> Date: Wed, 31 Jan 2024 20:06:00 -0800 Subject: [PATCH 2/3] ob-comint: Make file results from async sessions respect :dir header * lisp/ob-comint.el (org-babel-comint-async-filter): Set default-directory before calling `org-babel-insert-result' https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t --- lisp/ob-comint.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 7d258ea0e..349524701 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string) (file-callback org-babel-comint-async-file-callback) (combined-string (concat org-babel-comint-async-dangling string)) (new-dangling combined-string) + ;; Assumes comint filter called with session buffer current + (session-dir default-directory) ;; list of UUID's matched by `org-babel-comint-async-indicator' uuid-list) (with-temp-buffer @@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result (funcall file-callback (nth @@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result res-str result-params info)) t)))) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: 0003-org-babel-New-babel-backend-API-function-org-babel-s.patch --] [-- Type: text/x-patch, Size: 6415 bytes --] From 5afbd3b5dce89d3da9f0532c6358392175a93a9f Mon Sep 17 00:00:00 2001 Message-ID: <5afbd3b5dce89d3da9f0532c6358392175a93a9f.1707228351.git.yantar92@posteo.net> In-Reply-To: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net> References: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Tue, 6 Feb 2024 15:02:48 +0100 Subject: [PATCH 3/3] org-babel: New babel backend API function org-babel-session-buffer:<lang> * lisp/ob-core.el (org-babel-session-buffer): New API function that return session buffer, if such buffer exists and is live. (org-babel-execute-src-block): Use `org-babel-session-buffer'. * lisp/ob-lua.el (org-babel-session-buffer:lua): * lisp/ob-python.el (org-babel-session-buffer:python): Provide API to retrieve session buffer name. (org-babel-python-initiate-session-by-key): Use `org-babel-session-buffer:python'. * etc/ORG-NEWS (Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~): Declare the API addition. Link: https://orgmode.org/list/87r0hr9f3b.fsf@localhost --- etc/ORG-NEWS | 11 +++++++++++ lisp/ob-core.el | 32 +++++++++++++++++++++++++++----- lisp/ob-lua.el | 5 +++++ lisp/ob-python.el | 8 ++++++-- 4 files changed, 49 insertions(+), 7 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 965872d23..246d83ffe 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -13,6 +13,17 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org. * Version 9.7 (not released yet) ** Important announcements and breaking changes +*** Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~ + +Org babel now uses session buffer (if it exists) to retrieve +~default-directory~ environment during src block evaluation. + +By default, buffer named like session is checked. All the backends +that create sessions inside buffers named differently should provide a +function ~org-babel-session-buffer:<lang>~. The function must accept +two arguments - session name and info list (as returned by +~org-babel-get-src-block-info~); and return the session buffer name. + *** Org mode now fontifies whole table lines (including newline) according to ~org-table~ face Previously, leading indentation and trailing newline in table rows diff --git a/lisp/ob-core.el b/lisp/ob-core.el index fd8d06c5d..a812887fa 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -767,8 +767,31 @@ (defun org-babel--file-desc (params result) (`(:file-desc) result) (`(:file-desc . ,(and (pred stringp) val)) val))) -(defvar *this*) ; Dynamically bound in `org-babel-execute-src-block' - ; and `org-babel-read' +(defvar *this*) +;; Dynamically bound in `org-babel-execute-src-block' +;; and `org-babel-read' + +(defun org-babel-session-buffer (&optional info) + "Return buffer name for session associated with current code block. +Return nil when no such live buffer with process exists. +When INFO is non-nil, it should be a list returned by +`org-babel-get-src-block-info'. +This function uses org-babel-session-buffer:<lang> function to +retrieve backend-specific session buffer name." + (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval))) + (lang (nth 0 info)) + (session (cdr (assq :session (nth 2 info)))) + (cmd (intern (concat "org-babel-session-buffer:" lang))) + buffer-name) + (setq buffer-name + (if (fboundp cmd) + (funcall cmd session info) + ;; Use session name as buffer name by default. + session)) + (when buffer-name + (require 'ob-comint) + (when (org-babel-comint-buffer-livep buffer-name) + buffer-name)))) ;;;###autoload (defun org-babel-execute-src-block (&optional arg info params executor-type) @@ -842,9 +865,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (default-directory (cond ((not dir) default-directory) - ((when-let ((session (cdr (assq :session params)))) - (when (org-babel-comint-buffer-livep session) - (buffer-local-value 'default-directory (get-buffer session))))) + ((when-let ((session (org-babel-session-buffer info))) + (buffer-local-value 'default-directory (get-buffer session)))) ((member mkdirp '("no" "nil" nil)) (file-name-as-directory (expand-file-name dir))) (t diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el index a2b830ca3..b241fccdc 100644 --- a/lisp/ob-lua.el +++ b/lisp/ob-lua.el @@ -184,6 +184,11 @@ (defun org-babel-lua-with-earmuffs (session) name (format "*%s*" name)))) +(defun org-babel-session-buffer:lua (session &optional _) + "Return session buffer name for SESSION." + (or (org-babel-lua-session-buffer session) + (org-babel-lua-with-earmuffs session))) + (defun org-babel-lua-without-earmuffs (session) "Remove stars around *SESSION*, leaving SESSION." (let ((name (if (stringp session) session (format "%s" session)))) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index cfc0fdcb6..89cdf4c47 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -260,6 +260,11 @@ (defun org-babel-python-without-earmuffs (session) (substring name 1 (- (length name) 1)) name))) +(defun org-babel-session-buffer:python (session &optional _) + "Return session buffer name for SESSION." + (or (org-babel-python-session-buffer session) + (org-babel-python-with-earmuffs session))) + (defun org-babel-python--python-util-comint-end-of-output-p () "Return non-nil if the last prompt matches input prompt. Backport of `python-util-comint-end-of-output-p' to emacs28. To @@ -302,8 +307,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session) initialized session." (save-window-excursion (let* ((session (if session (intern session) :default)) - (py-buffer (or (org-babel-python-session-buffer session) - (org-babel-python-with-earmuffs session))) + (py-buffer (org-babel-session-buffer:python session)) (python-shell-buffer-name (org-babel-python-without-earmuffs py-buffer)) (existing-session-p (comint-check-proc py-buffer)) -- 2.43.0 [-- Attachment #5: Type: text/plain, Size: 224 bytes --] -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-06 14:06 ` Ihor Radchenko @ 2024-02-08 3:08 ` Jack Kamm 2024-02-08 15:43 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-08 3:08 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > +(defun org-babel-session-buffer (&optional info) > + "Return buffer name for session associated with current code block. > +Return nil when no such live buffer with process exists. > +When INFO is non-nil, it should be a list returned by > +`org-babel-get-src-block-info'. > +This function uses org-babel-session-buffer:<lang> function to > +retrieve backend-specific session buffer name." > + (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval))) > + (lang (nth 0 info)) > + (session (cdr (assq :session (nth 2 info)))) > + (cmd (intern (concat "org-babel-session-buffer:" lang))) > + buffer-name) On executing any python session block I am getting the following error which I think is caused by the above: Debugger entered--Lisp error: (void-variable buffer-name) Also, make shows a byte-compiler warning about this: Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el... In org-babel-session-buffer: ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-08 3:08 ` Jack Kamm @ 2024-02-08 15:43 ` Ihor Radchenko 2024-02-08 17:46 ` Bruno Barbier 2024-02-10 18:14 ` Jack Kamm 0 siblings, 2 replies; 24+ messages in thread From: Ihor Radchenko @ 2024-02-08 15:43 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 597 bytes --] Jack Kamm <jackkamm@gmail.com> writes: > On executing any python session block I am getting the following error > which I think is caused by the above: > > Debugger entered--Lisp error: (void-variable buffer-name) That's a mystery. > Also, make shows a byte-compiler warning about this: > > Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el... > > In org-babel-session-buffer: > ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’ But this is a real problem. See the attached updated version of the patch set. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: v2-0001-org-babel-execute-src-block-Force-dir-according-t.patch --] [-- Type: text/x-patch, Size: 1949 bytes --] From 1185e7a3569048b465b697fe173ce50a3ab7f0cc Mon Sep 17 00:00:00 2001 Message-ID: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Sat, 3 Feb 2024 16:47:57 +0100 Subject: [PATCH v2 1/3] org-babel-execute-src-block: Force :dir according to live session * lisp/ob-core.el (org-babel-execute-src-block): Force eval directory to follow live session buffer, if any. This is consistent with what we promise in the manual section "16.4 Environment of a Code Block">Choosing a working directory. Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com --- lisp/ob-core.el | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 1de3af6ad..fd8d06c5d 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (dir (cdr (assq :dir params))) (mkdirp (cdr (assq :mkdirp params))) (default-directory - (cond - ((not dir) default-directory) - ((member mkdirp '("no" "nil" nil)) - (file-name-as-directory (expand-file-name dir))) - (t - (let ((d (file-name-as-directory (expand-file-name dir)))) - (make-directory d 'parents) - d)))) + (cond + ((not dir) default-directory) + ((when-let ((session (cdr (assq :session params)))) + (when (org-babel-comint-buffer-livep session) + (buffer-local-value 'default-directory (get-buffer session))))) + ((member mkdirp '("no" "nil" nil)) + (file-name-as-directory (expand-file-name dir))) + (t + (let ((d (file-name-as-directory (expand-file-name dir)))) + (make-directory d 'parents) + d)))) (cmd (intern (concat "org-babel-execute:" lang))) result exec-start-time) (unless (fboundp cmd) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: v2-0002-ob-comint-Make-file-results-from-async-sessions-r.patch --] [-- Type: text/x-patch, Size: 2638 bytes --] From 8799d747ab2983f43dab117713342dbc92c6fc16 Mon Sep 17 00:00:00 2001 Message-ID: <8799d747ab2983f43dab117713342dbc92c6fc16.1707406936.git.yantar92@posteo.net> In-Reply-To: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net> References: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net> From: Jack Kamm <jackkamm@gmail.com> Date: Wed, 31 Jan 2024 20:06:00 -0800 Subject: [PATCH v2 2/3] ob-comint: Make file results from async sessions respect :dir header * lisp/ob-comint.el (org-babel-comint-async-filter): Set default-directory before calling `org-babel-insert-result' https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t --- lisp/ob-comint.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 7d258ea0e..349524701 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string) (file-callback org-babel-comint-async-file-callback) (combined-string (concat org-babel-comint-async-dangling string)) (new-dangling combined-string) + ;; Assumes comint filter called with session buffer current + (session-dir default-directory) ;; list of UUID's matched by `org-babel-comint-async-indicator' uuid-list) (with-temp-buffer @@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result (funcall file-callback (nth @@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params - (cdr (assq :result-params params)))) + (cdr (assq :result-params params))) + (default-directory session-dir)) (org-babel-insert-result res-str result-params info)) t)))) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: v2-0003-org-babel-New-babel-backend-API-function-org-babe.patch --] [-- Type: text/x-patch, Size: 6458 bytes --] From 0d496b8fcc205810c994b32b2ab732305a4d9a26 Mon Sep 17 00:00:00 2001 Message-ID: <0d496b8fcc205810c994b32b2ab732305a4d9a26.1707406936.git.yantar92@posteo.net> In-Reply-To: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net> References: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net> From: Ihor Radchenko <yantar92@posteo.net> Date: Tue, 6 Feb 2024 15:02:48 +0100 Subject: [PATCH v2 3/3] org-babel: New babel backend API function org-babel-session-buffer:<lang> * lisp/ob-core.el (org-babel-session-buffer): New API function that return session buffer, if such buffer exists and is live. (org-babel-execute-src-block): Use `org-babel-session-buffer'. * lisp/ob-lua.el (org-babel-session-buffer:lua): * lisp/ob-python.el (org-babel-session-buffer:python): Provide API to retrieve session buffer name. (org-babel-python-initiate-session-by-key): Use `org-babel-session-buffer:python'. * etc/ORG-NEWS (Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~): Declare the API addition. Link: https://orgmode.org/list/87r0hr9f3b.fsf@localhost --- etc/ORG-NEWS | 11 +++++++++++ lisp/ob-core.el | 31 ++++++++++++++++++++++++++----- lisp/ob-lua.el | 5 +++++ lisp/ob-python.el | 8 ++++++-- 4 files changed, 48 insertions(+), 7 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 92d363b80..7ac56702d 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -13,6 +13,17 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org. * Version 9.7 (not released yet) ** Important announcements and breaking changes +*** Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~ + +Org babel now uses session buffer (if it exists) to retrieve +~default-directory~ environment during src block evaluation. + +By default, buffer named like session is checked. All the backends +that create sessions inside buffers named differently should provide a +function ~org-babel-session-buffer:<lang>~. The function must accept +two arguments - session name and info list (as returned by +~org-babel-get-src-block-info~); and return the session buffer name. + *** Org mode now fontifies whole table lines (including newline) according to ~org-table~ face Previously, leading indentation and trailing newline in table rows diff --git a/lisp/ob-core.el b/lisp/ob-core.el index fd8d06c5d..bfeac257b 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -767,8 +767,30 @@ (defun org-babel--file-desc (params result) (`(:file-desc) result) (`(:file-desc . ,(and (pred stringp) val)) val))) -(defvar *this*) ; Dynamically bound in `org-babel-execute-src-block' - ; and `org-babel-read' +(defvar *this*) +;; Dynamically bound in `org-babel-execute-src-block' +;; and `org-babel-read' + +(defun org-babel-session-buffer (&optional info) + "Return buffer name for session associated with current code block. +Return nil when no such live buffer with process exists. +When INFO is non-nil, it should be a list returned by +`org-babel-get-src-block-info'. +This function uses org-babel-session-buffer:<lang> function to +retrieve backend-specific session buffer name." + (declare-function org-babel-comint-buffer-livep "ob-comint" (buffer)) + (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval))) + (lang (nth 0 info)) + (session (cdr (assq :session (nth 2 info)))) + (cmd (intern (concat "org-babel-session-buffer:" lang))) + (buffer-name + (if (fboundp cmd) + (funcall cmd session info) + ;; Use session name as buffer name by default. + session))) + (require 'ob-comint) + (when (org-babel-comint-buffer-livep buffer-name) + buffer-name))) ;;;###autoload (defun org-babel-execute-src-block (&optional arg info params executor-type) @@ -842,9 +864,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (default-directory (cond ((not dir) default-directory) - ((when-let ((session (cdr (assq :session params)))) - (when (org-babel-comint-buffer-livep session) - (buffer-local-value 'default-directory (get-buffer session))))) + ((when-let ((session (org-babel-session-buffer info))) + (buffer-local-value 'default-directory (get-buffer session)))) ((member mkdirp '("no" "nil" nil)) (file-name-as-directory (expand-file-name dir))) (t diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el index a2b830ca3..b241fccdc 100644 --- a/lisp/ob-lua.el +++ b/lisp/ob-lua.el @@ -184,6 +184,11 @@ (defun org-babel-lua-with-earmuffs (session) name (format "*%s*" name)))) +(defun org-babel-session-buffer:lua (session &optional _) + "Return session buffer name for SESSION." + (or (org-babel-lua-session-buffer session) + (org-babel-lua-with-earmuffs session))) + (defun org-babel-lua-without-earmuffs (session) "Remove stars around *SESSION*, leaving SESSION." (let ((name (if (stringp session) session (format "%s" session)))) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index cfc0fdcb6..89cdf4c47 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -260,6 +260,11 @@ (defun org-babel-python-without-earmuffs (session) (substring name 1 (- (length name) 1)) name))) +(defun org-babel-session-buffer:python (session &optional _) + "Return session buffer name for SESSION." + (or (org-babel-python-session-buffer session) + (org-babel-python-with-earmuffs session))) + (defun org-babel-python--python-util-comint-end-of-output-p () "Return non-nil if the last prompt matches input prompt. Backport of `python-util-comint-end-of-output-p' to emacs28. To @@ -302,8 +307,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session) initialized session." (save-window-excursion (let* ((session (if session (intern session) :default)) - (py-buffer (or (org-babel-python-session-buffer session) - (org-babel-python-with-earmuffs session))) + (py-buffer (org-babel-session-buffer:python session)) (python-shell-buffer-name (org-babel-python-without-earmuffs py-buffer)) (existing-session-p (comint-check-proc py-buffer)) -- 2.43.0 [-- Attachment #5: Type: text/plain, Size: 224 bytes --] -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-08 15:43 ` Ihor Radchenko @ 2024-02-08 17:46 ` Bruno Barbier 2024-02-10 18:14 ` Jack Kamm 1 sibling, 0 replies; 24+ messages in thread From: Bruno Barbier @ 2024-02-08 17:46 UTC (permalink / raw) To: Ihor Radchenko, Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Jack Kamm <jackkamm@gmail.com> writes: > >> On executing any python session block I am getting the following error >> which I think is caused by the above: >> >> Debugger entered--Lisp error: (void-variable buffer-name) > > That's a mystery. It looks like 'when-let*' doesn't accept symbols without values; it needs real bindings. The form `let*' assigns nil when there are no values, but, with `when-let*', assigning nil would make the whole `when-let' fails. Here is the expansion when using a symbol without a value, which explains both the void-variable error and the compiler message: #+begin_src elisp :results scalar (macroexpand-all '(when-let* (new-var) 7)) #+end_src #+RESULTS: : (let* ((new-var (and t new-var))) (if new-var 7)) Bruno ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-08 15:43 ` Ihor Radchenko 2024-02-08 17:46 ` Bruno Barbier @ 2024-02-10 18:14 ` Jack Kamm 2024-02-10 21:01 ` Ihor Radchenko 1 sibling, 1 reply; 24+ messages in thread From: Jack Kamm @ 2024-02-10 18:14 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Nasser Alkmim, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: >> Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el... >> >> In org-babel-session-buffer: >> ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’ > > But this is a real problem. > See the attached updated version of the patch set. Looks good, thanks. I tested it out it works without errors/warnings now. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-10 18:14 ` Jack Kamm @ 2024-02-10 21:01 ` Ihor Radchenko 2024-02-13 9:43 ` Nasser Alkmim 0 siblings, 1 reply; 24+ messages in thread From: Ihor Radchenko @ 2024-02-10 21:01 UTC (permalink / raw) To: Jack Kamm; +Cc: Nasser Alkmim, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: >> See the attached updated version of the patch set. > > Looks good, thanks. I tested it out it works without errors/warnings > now. Applied, onto main. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=13d0f8bf8 https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=644bf846d https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8c7313d39 Fixed. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-10 21:01 ` Ihor Radchenko @ 2024-02-13 9:43 ` Nasser Alkmim 2024-02-13 14:31 ` Ihor Radchenko 0 siblings, 1 reply; 24+ messages in thread From: Nasser Alkmim @ 2024-02-13 9:43 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Jack Kamm, emacs-orgmode Excellent job Ihor and Jack. On the same note, it would be convenient to have a hook to run after the async execution returns the output. Similar to 'org-babel-after-execute-hook'. So I could, for instance, redisplay the inline image previews. Is there an easy way to do that? -- Nasser Alkmim +43 677 6408 9171 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Async Python src block behavior with :dir header property 2024-02-13 9:43 ` Nasser Alkmim @ 2024-02-13 14:31 ` Ihor Radchenko 0 siblings, 0 replies; 24+ messages in thread From: Ihor Radchenko @ 2024-02-13 14:31 UTC (permalink / raw) To: Nasser Alkmim; +Cc: Jack Kamm, emacs-orgmode Nasser Alkmim <nasser.alkmim@gmail.com> writes: > On the same note, it would be convenient to have a hook to run after the > async execution returns the output. Similar to > 'org-babel-after-execute-hook'. So I could, for instance, redisplay the > inline image previews. Is there an easy way to do that? Maybe we can add a hook for `org-babel-insert-result'. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2024-02-13 14:27 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-01-31 7:29 Async Python src block behavior with :dir header property Nasser Alkmim 2024-01-31 11:23 ` Ihor Radchenko 2024-01-31 12:17 ` Nasser Alkmim 2024-01-31 12:47 ` Ihor Radchenko 2024-01-31 15:00 ` Nasser Alkmim 2024-01-31 16:21 ` Ihor Radchenko 2024-02-01 4:16 ` Jack Kamm 2024-02-01 8:40 ` Nasser Alkmim 2024-02-01 11:59 ` Ihor Radchenko 2024-02-03 1:20 ` Jack Kamm 2024-02-03 15:51 ` Ihor Radchenko 2024-02-04 1:26 ` Jack Kamm 2024-02-04 16:30 ` Ihor Radchenko 2024-02-05 0:59 ` Jack Kamm 2024-02-05 14:10 ` Ihor Radchenko 2024-02-06 6:56 ` Jack Kamm 2024-02-06 14:06 ` Ihor Radchenko 2024-02-08 3:08 ` Jack Kamm 2024-02-08 15:43 ` Ihor Radchenko 2024-02-08 17:46 ` Bruno Barbier 2024-02-10 18:14 ` Jack Kamm 2024-02-10 21:01 ` Ihor Radchenko 2024-02-13 9:43 ` Nasser Alkmim 2024-02-13 14:31 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).