* [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
@ 2024-02-01 11:58 Ihor Radchenko
2024-02-01 14:56 ` Bruno Barbier
0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2024-02-01 11:58 UTC (permalink / raw)
To: emacs-orgmode, Jack Kamm
This is a followup for another bug report
https://orgmode.org/list/87plxg95u4.fsf@gmail.com
Using the MWE from that report I am seeing a different bug and it is not
very clear for me what is causing it.
1. When I create /tmp/bug.org with the following contents:
#+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
import matplotlib.pyplot as plt
plt.figure(figsize=(1, 1))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname)
#+end_src
2. Go to Org git folder
3. make repro REPRO_ARGS="-l ob-python /tmp/bug.org"
4. Evaluate the code block
5. I get
#+RESULTS:
[[file:babel-GJy87v/python-xr8wq9]]
rather than expected [[file:otherdir/fig.svg]]
I use matplotlib 3.8.2-r1.
The contents of Python REPL is suspicious:
Python 3.11.7 (main, Jan 8 2024, 18:11:18) [GCC 13.2.1 20231216] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
ob_comint_async_python_file_/tmp/babel-s8nwvB/python-NmMFRv
>>>
This is most likely something about my current system setup - I can
reproduce with other Org mode and Emacs versions. But I have no clue
what is the cause.
Emacs : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0)
of 2024-01-30
Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-01 11:58 [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Ihor Radchenko @ 2024-02-01 14:56 ` Bruno Barbier 2024-02-03 1:30 ` Jack Kamm 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-02-01 14:56 UTC (permalink / raw) To: Ihor Radchenko, emacs-orgmode, Jack Kamm Hi Ihor, Ihor Radchenko <yantar92@posteo.net> writes: > > This is most likely something about my current system setup - I can > reproduce with other Org mode and Emacs versions. But I have no clue > what is the cause. I'm getting the same as you with your MWE. The tag, used by ob-comint async, is: "/tmp/babel-zqh04P/python-GL5N5d" but, in "/tmp/bug.org" it becomes: "babel-zqh04P/python-GL5N5d" (`org-babel-result-to-file' transformed it into a simpler relative path). The filter `org-babel-comint-async-filter' cannot spot it, because it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw". Here is how to reproduce: #+begin_src elisp :results table (let* ((tag "/tmp/babel-zqh04P/python-tXsdFw") (repro (lambda (fn) (let ((lnk (with-temp-buffer (org-mode) (let* ((default-directory "/tmp") (buffer-file-name fn) (cbuf (clone-indirect-buffer "tmp" nil))) (with-current-buffer cbuf (org-babel-result-to-file tag)))))) (list fn (not (eq nil (string-match-p (regexp-quote tag) lnk))) lnk))))) (cons (list "Filename" "string-match-p" "Org link") (cons 'hline (mapcar repro (list "/tmp/bug.org" "/somewhere/else/bug.org"))))) #+end_src #+RESULTS: | Filename | match-p | Org link | |-------------------------+---------+-----------------------------------------------| | /tmp/bug.org | nil | [[file:babel-zqh04P/python-tXsdFw]] | | /somewhere/else/bug.org | t | [[file:../../tmp/babel-zqh04P/python-tXsdFw]] | I don't know what a proper fix would be though. Hoping this help, Bruno > > Emacs : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0) > of 2024-01-30 > Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/) > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-01 14:56 ` Bruno Barbier @ 2024-02-03 1:30 ` Jack Kamm 2024-02-04 15:07 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Jack Kamm @ 2024-02-03 1:30 UTC (permalink / raw) To: Bruno Barbier, Ihor Radchenko, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > Hi Ihor, > > Ihor Radchenko <yantar92@posteo.net> writes: > >> >> This is most likely something about my current system setup - I can >> reproduce with other Org mode and Emacs versions. But I have no clue >> what is the cause. > > I'm getting the same as you with your MWE. > > The tag, used by ob-comint async, is: > > "/tmp/babel-zqh04P/python-GL5N5d" > > but, in "/tmp/bug.org" it becomes: > > "babel-zqh04P/python-GL5N5d" > > (`org-babel-result-to-file' transformed it into a simpler relative > path). > > The filter `org-babel-comint-async-filter' cannot spot it, because > it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw". Thanks, I believe your diagnosis is correct, and can confirm the bug occurs when the Org file is in /tmp. The problem goes away when `org-link-file-path-type' is set to absolute. I think the correct solution would be for `org-babel-insert-result' to not insert file results (or any other special results) for async session blocks. Perhaps in this case, `org-babel-insert-result' could return a new result type, named "async", "future", or similar. ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-03 1:30 ` Jack Kamm @ 2024-02-04 15:07 ` Ihor Radchenko 2024-02-05 1:37 ` Jack Kamm 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-04 15:07 UTC (permalink / raw) To: Jack Kamm; +Cc: Bruno Barbier, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: > I think the correct solution would be for `org-babel-insert-result' to > not insert file results (or any other special results) for async session > blocks. Perhaps in this case, `org-babel-insert-result' could return a > new result type, named "async", "future", or similar. That will not work. `org-babel-comint-async-filter' expects a unique result to be inserted into Org buffer, so that it can be located, and replaced by the async evaluation output. So, we have to insert some kind of indicator for async result. Of course, the existing scheme of coordination between `org-babel-insert-result' and `org-babel-comint-async-filter' is erroneous: 1. We have the problem with :results file value discussed here 2. We have a worse problem with :results file :file foo when the result may not be unique 3. We have :results append/prepend completely broken because `org-babel-comint-async-filter' simply calls `org-babel-insert-result' implicitly assuming that the existing indicator is replaced. The whole thing should be re-designed. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-04 15:07 ` Ihor Radchenko @ 2024-02-05 1:37 ` Jack Kamm 2024-02-05 14:29 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Jack Kamm @ 2024-02-05 1:37 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Jack Kamm <jackkamm@gmail.com> writes: > >> I think the correct solution would be for `org-babel-insert-result' to >> not insert file results (or any other special results) for async session >> blocks. Perhaps in this case, `org-babel-insert-result' could return a >> new result type, named "async", "future", or similar. > > That will not work. > `org-babel-comint-async-filter' expects a unique result to be inserted > into Org buffer, so that it can be located, and replaced by the async > evaluation output. > > So, we have to insert some kind of indicator for async result. I meant that we could return something like "async:uuid-abcd-1234" or "async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could still find the result. > Of course, the existing scheme of coordination between > `org-babel-insert-result' and `org-babel-comint-async-filter' is > erroneous: > > 1. We have the problem with :results file value discussed here > 2. We have a worse problem with :results file :file foo when the result > may not be unique > 3. We have :results append/prepend completely broken because > `org-babel-comint-async-filter' simply calls > `org-babel-insert-result' implicitly assuming that the existing > indicator is replaced. > > The whole thing should be re-designed. I agree that it would be good to redesign it, but am not sure where to start. A bit of a tangent, but if you are thinking about re-designing this, then it may be worth considering ob-jupyter's implementation of async sessions [1]. In particular, I believe it leaves a marker [2] where it needs to insert the future result. I don't remember the details, e.g. how it keeps track of which marker is for which result. But it seems neat, and might work better for some cases such as appending/prepending results. [1] https://github.com/emacs-jupyter/jupyter [2] https://www.gnu.org/software/emacs/manual/html_node/elisp/Markers.html ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-05 1:37 ` Jack Kamm @ 2024-02-05 14:29 ` Ihor Radchenko 2024-02-06 19:24 ` Bruno Barbier 2024-02-08 3:26 ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm 0 siblings, 2 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-02-05 14:29 UTC (permalink / raw) To: Jack Kamm; +Cc: Bruno Barbier, emacs-orgmode Jack Kamm <jackkamm@gmail.com> writes: >> So, we have to insert some kind of indicator for async result. > > I meant that we could return something like "async:uuid-abcd-1234" or > "async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could > still find the result. Of course, we could. But that would not solve all the possible problems. In particular, when header arguments tell Org babel to write result to a file, returning UUID will still not work. >> Of course, the existing scheme of coordination between >> `org-babel-insert-result' and `org-babel-comint-async-filter' is >> erroneous: >> >> 1. We have the problem with :results file value discussed here >> 2. We have a worse problem with :results file :file foo when the result >> may not be unique >> 3. We have :results append/prepend completely broken because >> `org-babel-comint-async-filter' simply calls >> `org-babel-insert-result' implicitly assuming that the existing >> indicator is replaced. >> >> The whole thing should be re-designed. > ... > A bit of a tangent, but if you are thinking about re-designing this, > then it may be worth considering ob-jupyter's implementation of async > sessions [1]. In particular, I believe it leaves a marker [2] where it > needs to insert the future result. I don't remember the details, > e.g. how it keeps track of which marker is for which result. But it > seems neat, and might work better for some cases such as > appending/prepending results. Markers are not as reliable as you think. If text around marker gets deleted, the marker will still exist potentially causing the async result to be inserted in the middle of unexpected place. Having an actual text indicator is more reliable - if user removes it before the async evaluation is completed, we will not write anything unexpected. Also, https://github.com/emacs-jupyter/jupyter/blob/master/ob-jupyter.el#L540 ;; KLUDGE: Remove the file result-parameter so that ;; `org-babel-insert-result' doesn't attempt to handle it while ;; async results are pending. Do the same in the synchronous ;; case, but not if link or graphics are also result-parameters, ;; only in Org >= 9.2, since those in combination with file mean ;; to interpret the result as a file link, a useful meaning that ;; doesn't interfere with Jupyter style result insertion. They also had to work around the same problem. > I agree that it would be good to redesign it, but am not sure where to > start. For example, 1. Change `org-babel-comint-async-register' to return UUID and to store PARAMS as passed by the backend (current approach with PARAMS being derived from src blocks prevents backends to transform src block PARAMS dynamically). 2. Change `org-babel-insert-result' to handle :async t specially, inserting something reliable, like #+async: <UUID> in place of result without performing extra transformations. 3. Change `org-babel-insert-result' to accept an internal parameter that will make it replace #+async: <UUID> keyword rather than perform normal result insertion. 4. Change `org-babel-comint-async-filter' to use the previously passed PARAMS, remove :async t from them, and arrange the call to `org-babel-insert-result' to replace the #+async: <UUID> keyword. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-05 14:29 ` Ihor Radchenko @ 2024-02-06 19:24 ` Bruno Barbier 2024-02-07 16:19 ` Ihor Radchenko ` (2 more replies) 2024-02-08 3:26 ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm 1 sibling, 3 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-06 19:24 UTC (permalink / raw) To: Ihor Radchenko, Jack Kamm; +Cc: emacs-orgmode Hi Ihor, Jack, Ihor Radchenko <yantar92@posteo.net> writes: > Jack Kamm <jackkamm@gmail.com> writes: > >> I agree that it would be good to redesign it, but am not sure where to >> start. > > For example, > > 1. Change `org-babel-comint-async-register' to return UUID and to store > PARAMS as passed by the backend (current approach with PARAMS being > derived from src blocks prevents backends to transform src block > PARAMS dynamically). > 2. Change `org-babel-insert-result' to handle :async t specially, > inserting something reliable, like #+async: <UUID> in place of result > without performing extra transformations. > 3. Change `org-babel-insert-result' to accept an internal parameter > that will make it replace #+async: <UUID> keyword rather than perform > normal result insertion. > 4. Change `org-babel-comint-async-filter' to use the previously passed > PARAMS, remove :async t from them, and arrange the call to > `org-babel-insert-result' to replace the #+async: <UUID> keyword. > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode. I think it would be good if ob-core itself could provide an asynchronous API. I've modified my Org so that it does have such an API. This is work in progress; let me describe it. I've modified ob-core itself to allow asynchronicity. In the asynchrosous case, instead of calling: (org-babel-execute:LANG body params) I'm calling: (org-babel-schedule:LANG body params handle-result) where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result). While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay. The overlay is used to know where to insert the result and to display the status/progress of the execution. If the file is closed and the execution fails, nothing is lost, the old result is still available. If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org. Let me know, Thanks, Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-06 19:24 ` Bruno Barbier @ 2024-02-07 16:19 ` Ihor Radchenko 2024-02-07 17:40 ` Bruno Barbier 2024-02-08 3:21 ` Jack Kamm 2024-02-15 20:02 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt 2 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-07 16:19 UTC (permalink / raw) To: Bruno Barbier; +Cc: Jack Kamm, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > FWIW, I've been trying to use asynchronous blocks for everything, not > only the source blocks that are based on the comint mode. I think it > would be good if ob-core itself could provide an asynchronous API. I've > modified my Org so that it does have such an API. This is work in > progress; let me describe it. > > I've modified ob-core itself to allow asynchronicity. In the > asynchrosous case, instead of calling: > > (org-babel-execute:LANG body params) > > I'm calling: > > (org-babel-schedule:LANG body params handle-result) > > where `org-babel-schedule:LANG' is in charge of calling `handle-result' > with the result (or the error) when it is known; `handle-result' takes > care to call `org-babel-insert-result' at the correct place (and > `org-babel-insert-result' is only called with a real result). LGTM. > While the execution is pending, I'm using the same technique that Org is > using when a source block is being edited: the result is left untouched, > but below an overlay. The overlay is used to know where to insert the > result and to display the status/progress of the execution. If the file > is closed and the execution fails, nothing is lost, the old result is > still available. > > If that technique looks safe enough and interesting, I can prepare a set > of patches so that we can discuss it further and, maybe, add it in Org. Using overlay also sounds good. I am wondering what you do when there is no result yet or when user edits whatever is under the overlay, but that's just a technical detail. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-07 16:19 ` Ihor Radchenko @ 2024-02-07 17:40 ` Bruno Barbier 0 siblings, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-07 17:40 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Jack Kamm, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> While the execution is pending, I'm using the same technique that Org is >> using when a source block is being edited: the result is left untouched, >> but below an overlay. The overlay is used to know where to insert the >> result and to display the status/progress of the execution. If the file >> is closed and the execution fails, nothing is lost, the old result is >> still available. >> >> If that technique looks safe enough and interesting, I can prepare a set >> of patches so that we can discuss it further and, maybe, add it in Org. > > Using overlay also sounds good. > > I am wondering what you do when there is no result yet or when user > edits whatever is under the overlay, > but that's just a technical detail. I don't think it's really robust yet. We'll see how to improve/change it. I'll prepare the set of patchs. Bruno > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-06 19:24 ` Bruno Barbier 2024-02-07 16:19 ` Ihor Radchenko @ 2024-02-08 3:21 ` Jack Kamm 2024-02-15 20:02 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt 2 siblings, 0 replies; 55+ messages in thread From: Jack Kamm @ 2024-02-08 3:21 UTC (permalink / raw) To: Bruno Barbier, Ihor Radchenko; +Cc: emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > FWIW, I've been trying to use asynchronous blocks for everything, not > only the source blocks that are based on the comint mode. I think it > would be good if ob-core itself could provide an asynchronous API. I've > modified my Org so that it does have such an API. This is work in > progress; let me describe it. > > I've modified ob-core itself to allow asynchronicity. In the > asynchrosous case, instead of calling: > > (org-babel-execute:LANG body params) > > I'm calling: > > (org-babel-schedule:LANG body params handle-result) > > where `org-babel-schedule:LANG' is in charge of calling `handle-result' > with the result (or the error) when it is known; `handle-result' takes > care to call `org-babel-insert-result' at the correct place (and > `org-babel-insert-result' is only called with a real result). Sounds interesting, a couple questions: 1. Which languages have you implemented/tested this on? 2. Does it apply for sessions, nonsessions, or both? > While the execution is pending, I'm using the same technique that Org is > using when a source block is being edited: the result is left untouched, > but below an overlay. The overlay is used to know where to insert the > result and to display the status/progress of the execution. If the file > is closed and the execution fails, nothing is lost, the old result is > still available. Also interesting, I think it's worth exploring/testing this overlay idea out. Does that mean that output is asynchronously printing into the Org buffer? It sounds cool but I wonder if it might cause problems while trying to edit another part of the buffer. ^ permalink raw reply [flat|nested] 55+ messages in thread
* Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-06 19:24 ` Bruno Barbier 2024-02-07 16:19 ` Ihor Radchenko 2024-02-08 3:21 ` Jack Kamm @ 2024-02-15 20:02 ` Matt 2024-02-16 17:52 ` Bruno Barbier 2 siblings, 1 reply; 55+ messages in thread From: Matt @ 2024-02-15 20:02 UTC (permalink / raw) To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode If I followed correctly, the topic switched to discussing async generally within Babel. I've started a new thread accordingly. ---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode. I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with. > I think it would be good if ob-core itself could provide an asynchronous API. Fortunately or unfortunately, depending on how you look at it, Babel already does. The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default. For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API: - "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. - "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. - "Synchronous" means that execution prevents the user from editing the document while results are obtained. - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained. > I've modified my Org so that it does have such an API. This is work in progress; let me describe it. > > I've modified ob-core itself to allow asynchronicity. In the asynchrosous case, instead of calling: > > (org-babel-execute:LANG body params) > > I'm calling: > > (org-babel-schedule:LANG body params handle-result) > > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result). > > While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay. The overlay is used to know where to insert the result and to display the status/progress of the execution. If the file is closed and the execution fails, nothing is lost, the old result is still available. The use of the overlay is a really cool idea! I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise. > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org. Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-15 20:02 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt @ 2024-02-16 17:52 ` Bruno Barbier 2024-02-18 21:14 ` Matt ` (2 more replies) 0 siblings, 3 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-16 17:52 UTC (permalink / raw) To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 8614 bytes --] Hi Matt, Jack, Ihor, Sorry for the late reply. Cleaning the code took me longer than expected. Jack Kamm <jackkamm@gmail.com> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> FWIW, I've been trying to use asynchronous blocks for everything, not >> only the source blocks that are based on the comint mode. >> ... > > Sounds interesting, a couple questions: > > 1. Which languages have you implemented/tested this on? I'm not using it with official org backends (yet). I'm using it with several custom backends that I'm working on. One of the backend delegate the block executions to emacs subprocesses: so I have a kind of asynchronous executions for free for any language, including elisp itself. > 2. Does it apply for sessions, nonsessions, or both? > The new API itself is more about how to wait for and display one block result. So, it's not really aware of sessions. But, I usually try to think as "no session" as a "one shot" session (like Matt wrote in his email). So, in that sense, it works for both anyway ;-) > Also interesting, I think it's worth exploring/testing this overlay idea > out. Does that mean that output is asynchronously printing into the Org > buffer? It sounds cool but I wonder if it might cause problems while > trying to edit another part of the buffer. Currently, I've limited the progress feedback to fit on one line to avoid anoying vertical display jumps. When the computation is successful, the result is inserted as usual, and, that may be annoying; when it updates a previous result of the same height, it's ok. Else, we could fold it to stay on one line too (using the overlay), until the user explicitly request to see it. Jack Kamm <jackkamm@gmail.com> writes: > > That all sounds reasonable...if you work on this, let me know if you > want any help with testing. Thanks. I'll definitely appreciate your help, to test the current code with the Python backend or any other backend if you prefer. Matt <matt@excalamus.com> writes: > If I followed correctly, the topic switched to discussing async > generally within Babel. I've started a new thread accordingly. Indeed. Thank you! > > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode. > > I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with. Good to know. Let's we make this happen! > > I think it would be good if ob-core itself could provide an asynchronous API. > > Fortunately or unfortunately, depending on how you look at it, Babel already does. > > The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default. > For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. > I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API: > > - "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. > > - "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. > > - "Synchronous" means that execution prevents the user from editing the document while results are obtained. > > - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained. I mostly think the same. Sessions (including the "none" session) definitely need some generic API and some generic tests that all backends could just reuse. To execute Python blocks, using the proposed async API: - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python). - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...). - The "no session" case is just about creating a new unique session and throwing it away immediately. But, some users may rely on some particular behavior, of the current implementations, that may be hard to implement in such a generic way. > The use of the overlay is a really cool idea! > > I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document). The backend is free to transform a failure into a success to make that result available though. > However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise. You should be able to test my current implementation, see below. It's almost too visual in my opinion. But, that's probably something that we should make easy to configure. > > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org. > > Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with. Good to know! So, here we go. You'll find attach a set of patchs. It works for me with Emacsc 30.50 and 9.7-pre (from today). I didn't check yet that the code and the commits follow all the guidelines. This is just a preliminary version for feedbacks. Corrections/critiques are welcome, but don't check the details until I check them myself. The 5 first patchs provide an API to handle asynchronous execution in ob-core, i.e. an API to report progress and to insert results in the asynchronous case. The 5th one isn't really about asynchronicity; but it adds a new keyword `:execute-with' that allows to delegate block executions to some other package; it's useful for testing or plugging other execution engines. That's a cleaned-up version of what I been using myself for a while, with 4 different values for `:execute-with'. The remaining patchs are new code that I've just written to show how to use this new API. I tried first to use it for ob-python, as an example. I just needed to figure out where to place the callbacks ... well ... "just" ... :) So, I decided to rewrite the whole thing, taking code from the synchronous case (following `org-babel-python-evaluate-session'). I also created a package that contains all the functions that should be reusable for any language. The patch [1] adds some generic functions to help dealing with asynchronicity (process, comint, etc.). The patch [2] shows a new possible way to execute python code blocks, both synchronously and asynchronously, with or without sessions. You should just need to open [3] and follow what's written there, and execute the existing bash and python code blocks. Note that this will create a folder `scratch/bba-ob-core-async' in your repository to place the temporary files there. If you know a better way to do this, let me know, thanks. I think the first 5 patchs could be included almost as-is in org. About the emaining ones, I'm not sure exactly how we should proceed. Let me know if you need help to test it. Feedbacks, corrections, critiques, etc are most welcome! Thanks! Bruno [1] lisp/org-elib-async.el: New package about async helpers [2] scratch/bba-ob-core-async: Some temporary test files [3] scratch/bba-ob-core-async/my-async-tests.org [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --] [-- Type: text/x-patch, Size: 1367 bytes --] From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:31:36 +0100 Subject: [PATCH 1/8] ob-core async: Add faces [1/5] lisp/org-faces.el (org-async-scheduled, org-async-pending, org-async-failure): new faces --- lisp/org-faces.el | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lisp/org-faces.el b/lisp/org-faces.el index 0e20de51a..5a8a8fd51 100644 --- a/lisp/org-faces.el +++ b/lisp/org-faces.el @@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun "Face used for clock display for overrun tasks in mode line." :group 'org-faces) +(defface org-async-scheduled '((t :inherit org-tag :background "gray")) + "Face for babel results for code blocks that are scheduled for execution." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + +(defface org-async-pending '((t :inherit org-checkbox :background "dark orange")) + "Face for babel results for code blocks that are running." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + +(defface org-async-failure '((t :inherit org-warning)) + "Face for babel results for code blocks that have failed." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + (provide 'org-faces) ;;; org-faces.el ends here -- 2.43.0 [-- Attachment #3: 0002-ob-core-async-Add-org-babel-async-tools-2-5.patch --] [-- Type: text/x-patch, Size: 10450 bytes --] From 8ecdc2159d85648949b477359793f007005bb7ca Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:00 +0100 Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5] --- lisp/ob-core.el | 219 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index bfeac257b..bb44f91cb 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -792,6 +792,225 @@ (defun org-babel-session-buffer (&optional info) (when (org-babel-comint-buffer-livep buffer-name) buffer-name))) +(defun org-babel--async-status-face (status) + (pcase status + (:scheduled 'org-async-scheduled) + (:pending 'org-async-pending) + (:failure 'org-async-failure) + (:success nil) + (_ (error "Not a status")) + )) + +(defun org-babel--async-make-overlay (beg end) + "Create an overlay between positions BEG and END and return it." + (let ((overlay (make-overlay beg end)) + (read-only + (list + (lambda (&rest _) + (user-error + "Cannot modify an area being updated")))) + ) + (cl-flet ((make-read-only + (ovl) + (overlay-put ovl 'modification-hooks read-only) + (overlay-put ovl 'insert-in-front-hooks read-only) + (overlay-put ovl 'insert-behind-hooks read-only)) + ) + (overlay-put overlay 'org-babel--async-type 'org-babel--async-note) + (overlay-put overlay 'face 'secondary-selection) + (overlay-put overlay 'help-echo "Pending src block result...") + (make-read-only overlay) + overlay))) + +(defun org-babel--async-result-region (inline-elem &optional info) + "Return the region of the results, for the source block at point." + (unless info (setq info (org-babel-get-src-block-info))) + (save-excursion + (when-let ((res-begin (org-babel-where-is-src-block-result nil info))) + (cons res-begin + (save-excursion + (goto-char res-begin) + (if inline-elem + ;; Logic copy/pasted from org-babel-where-is-src-block-result. + (let ((result (org-element-context))) + (and (org-element-type-p result 'macro) + (string= (org-element-property :key result) + "results") + (progn + (goto-char (org-element-end result)) + (skip-chars-backward " \t") + (point)))) + ;; Logic copy/pasted from hide-result + (beginning-of-line) + (let ((case-fold-search t)) + (unless (re-search-forward org-babel-result-regexp nil t) + (error "Not looking at a result line"))) + (org-babel-result-end) + )))))) + +(defun org-babel--async-feedbacks (info handle-result + result-params exec-start-time) + "Flag the result as 'scheduled' and return how to handle feedbacks. + +Use overlays to report progress and status to the user. Do not delete +the existing result unless a new one is available. When the result is +available, remove the async overlays and insert the result as usual, +like for a synchronous result. In case of failure, use an overlay to +report the error. + +The returned function handles 3 types of feedbacks: + - (:success R): Evaluation is successful; result is R. + - (:failure ERR): Evaluation failed; error is ERR. + - (:pending P): Outcome still pending; current progress is P." + ;; FIXME: INFO CMD ... Nothing is used but handle-result here !! + (let (;; copy/pasted from org-babel-insert-result + (inline-elem (let ((context (org-element-context))) + (and (memq (org-element-type context) + '(inline-babel-call inline-src-block)) + context))) + result-indent + ) + (cl-labels + ((eot-point (start) + "Move to End Of Title after START" + (if inline-elem + (org-element-end inline-elem) + (save-excursion (goto-char start) + (forward-line 1) (point)))) + (after-indent (pt) + "Move after indentation, starting at PT." + (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*"))) + (mk-result-overlays () + ;; Make 2 overlays to handle the pending result: one title + ;; (first line) and one for the body. + (pcase-let ((`(,start . ,end) (org-babel--async-result-region + inline-elem info))) + (let ((anchor-end (eot-point start))) + (cons (org-babel--async-make-overlay + (after-indent start) + (1- anchor-end)) + (org-babel--async-make-overlay + anchor-end end))))) + (add-style (status txt) + ;; Add the style matching STATUS over the text TXT. + (propertize txt 'face (org-babel--async-status-face status))) + + (short-version-of (msg) + ;; Compute the short version of MSG, to display in the header. + ;; Must return a string. + (if msg + (car (split-string (format "%s" msg) "\n" :omit-nulls)) + "")) + (update (ovl-title status msg) + ;; Update the title overlay to match STATUS and MSG. + (let (header) + (overlay-put ovl-title + 'face + (org-babel--async-status-face status) + ) + (overlay-put ovl-title + 'before-string (pcase status + (:scheduled "⏱") + (:pending "⏳") + (:failure "❌") + (:success "✔️"))) + + (overlay-put ovl-title + 'after-string + (propertize (format " |%s|" + (if (eq :failure status) + (if (consp msg) (car msg) + (format "%s" msg)) + (short-version-of msg))) + 'face (org-babel--async-status-face status))) + )) + (remove-previous-overlays () + ;; Remove previous title and body overlays. + (mapc (lambda (ovl) + (when (eq 'org-babel--async-note + (overlay-get ovl 'org-babel--async-type)) + (delete-overlay ovl))) + (when-let ((region (org-babel--async-result-region + inline-elem info))) + ;; Not sure why, but we do need to start before + ;; point min, else, in some cases, some overlays + ;; are not found. + (overlays-in (max (1- (car region)) (point-min)) + (cdr region)))))) + + (remove-previous-overlays) + + ;; Ensure there is a non-empty region for the result. + (save-excursion + (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil) + (org-babel-insert-result + ;; Use " " for the empty result. That cannot be nil, else it's interpreted + ;; as a list. We need at least one char, to separate markers if any. + " \n" + result-params + info nil + (nth 0 info) ; lang + exec-start-time + ))) + + ;; Create the overlays that span the result title and its body. + (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays))) + ;; Flag the result as ":scheduled". + (update title-ovl :scheduled nil) + + ;; The callback, that runs in the org buffer at point. + (let ((buf (current-buffer)) + (pt (point-marker))) + (lambda (feedback) + (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback) + (with-current-buffer buf + (save-excursion + (goto-char pt) + (pcase feedback + (`(:success ,r) + ;; Visual beep that the result is available. + (update title-ovl :success r) + (sit-for 0.2) + ;; We remove all overlays and let org insert the result + ;; as it would in the synchronous case. + (delete-overlay title-ovl) + (delete-overlay body-ovl) + (funcall handle-result r)) + + (`(:pending ,r) + ;; Still waiting for the outcome. Update our + ;; overlays with the progress info R. + (message "Updating block at %s@%s" pt buf) + (update title-ovl :pending r)) + + (`(:failure ,err) + ;; We didn't get a result. We update our overlays + ;; to report that failure. And unlock the old + ;; result. + (overlay-put title-ovl 'face nil) + (update title-ovl :failure err) + (delete-overlay body-ovl)) + + (_ (error "Invalid outcome")) + ) + )) + nil)))))) + + +(cl-defun org-babel--async-p (params &key default) + "Return a non-nil value when the execution is asynchronous. +Get the value of the :nasync argument and convert it." + (if-let ((binding (assq :nasync params))) + (pcase (cdr binding) + ((pred (not stringp)) + (error "Invalid value for :nasync argument")) + ((or "no" "n") nil) + ((or "yes" "y") t) + (_ (error "Invalid value for :nasync argument"))) + default)) + + + ;;;###autoload (defun org-babel-execute-src-block (&optional arg info params executor-type) "Execute the current source code block and return the result. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --] [-- Type: text/x-patch, Size: 5958 bytes --] From e8fe2f922a992de48ca80c52f530b0aa5fb45ea3 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:22 +0100 Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5] lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to prepare for the next change: move the part handling the result in its own function `handle-result'. --- lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------ 1 file changed, 53 insertions(+), 52 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index bb44f91cb..f8071a534 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1092,7 +1092,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (make-directory d 'parents) d)))) (cmd (intern (concat "org-babel-execute:" lang))) - result exec-start-time) + (exec-start-time (current-time)) + (handle-result + (lambda (result) + (setq result + (if (and (eq (cdr (assq :result-type params)) 'value) + (or (member "vector" result-params) + (member "table" result-params)) + (not (listp result))) + (list (list result)) + result)) + (let ((file (and (member "file" result-params) + (cdr (assq :file params))))) + ;; If non-empty result and :file then write to :file. + (when file + ;; If `:results' are special types like `link' or + ;; `graphics', don't write result to `:file'. Only + ;; insert a link to `:file'. + (when (and result + (not (or (member "link" result-params) + (member "graphics" result-params)))) + (with-temp-file file + (insert (org-babel-format-result + result + (cdr (assq :sep params))))) + ;; Set file permissions if header argument + ;; `:file-mode' is provided. + (when (assq :file-mode params) + (set-file-modes file (cdr (assq :file-mode params))))) + (setq result file)) + ;; Possibly perform post process provided its + ;; appropriate. Dynamically bind "*this*" to the + ;; actual results of the block. + (let ((post (cdr (assq :post params)))) + (when post + (let ((*this* (if (not file) result + (org-babel-result-to-file + file + (org-babel--file-desc params result) + 'attachment)))) + (setq result (org-babel-ref-resolve post)) + (when file + (setq result-params (remove "file" result-params)))))) + (unless (member "none" result-params) + (org-babel-insert-result + result result-params info + ;; append/prepend cannot handle hash as we accumulate + ;; multiple outputs together. + (when (member "replace" result-params) new-hash) + lang + (time-subtract (current-time) exec-start-time))) + (run-hooks 'org-babel-after-execute-hook) + result)))) (unless (fboundp cmd) (error "No org-babel-execute function for %s!" lang)) (message "Executing %s %s %s..." @@ -1107,57 +1158,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (if name (format "(%s)" name) (format "at position %S" (nth 5 info))))) - (setq exec-start-time (current-time) - result - (let ((r (save-current-buffer (funcall cmd body params)))) - (if (and (eq (cdr (assq :result-type params)) 'value) - (or (member "vector" result-params) - (member "table" result-params)) - (not (listp r))) - (list (list r)) - r))) - (let ((file (and (member "file" result-params) - (cdr (assq :file params))))) - ;; If non-empty result and :file then write to :file. - (when file - ;; If `:results' are special types like `link' or - ;; `graphics', don't write result to `:file'. Only - ;; insert a link to `:file'. - (when (and result - (not (or (member "link" result-params) - (member "graphics" result-params)))) - (with-temp-file file - (insert (org-babel-format-result - result - (cdr (assq :sep params))))) - ;; Set file permissions if header argument - ;; `:file-mode' is provided. - (when (assq :file-mode params) - (set-file-modes file (cdr (assq :file-mode params))))) - (setq result file)) - ;; Possibly perform post process provided its - ;; appropriate. Dynamically bind "*this*" to the - ;; actual results of the block. - (let ((post (cdr (assq :post params)))) - (when post - (let ((*this* (if (not file) result - (org-babel-result-to-file - file - (org-babel--file-desc params result) - 'attachment)))) - (setq result (org-babel-ref-resolve post)) - (when file - (setq result-params (remove "file" result-params)))))) - (unless (member "none" result-params) - (org-babel-insert-result - result result-params info - ;; append/prepend cannot handle hash as we accumulate - ;; multiple outputs together. - (when (member "replace" result-params) new-hash) - lang - (time-subtract (current-time) exec-start-time)))) - (run-hooks 'org-babel-after-execute-hook) - result))))))) + (funcall handle-result (save-current-buffer (funcall cmd body params)))))))))) (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --] [-- Type: text/x-patch, Size: 2620 bytes --] From c9d64263af8d9d9b8efc83585323cc6d876018dc Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:45 +0100 Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5] --- lisp/ob-core.el | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index f8071a534..0f0a36b70 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1091,7 +1091,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (let ((d (file-name-as-directory (expand-file-name dir)))) (make-directory d 'parents) d)))) - (cmd (intern (concat "org-babel-execute:" lang))) + (async (org-babel--async-p params)) + (cmd (intern (concat "org-babel-" + (if async "schedule" "execute") + ":" lang))) (exec-start-time (current-time)) (handle-result (lambda (result) @@ -1145,8 +1148,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (run-hooks 'org-babel-after-execute-hook) result)))) (unless (fboundp cmd) - (error "No org-babel-execute function for %s!" lang)) - (message "Executing %s %s %s..." + (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd))) + (message "Executing %s %s %s %s..." (capitalize lang) (pcase executor-type ('src-block "code block") @@ -1154,11 +1157,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) ('babel-call "call") ('inline-babel-call "inline call") (e (symbol-name e))) + (if async "async" "") (let ((name (nth 4 info))) (if name (format "(%s)" name) (format "at position %S" (nth 5 info))))) - (funcall handle-result (save-current-buffer (funcall cmd body params)))))))))) + (if (not async) + (funcall handle-result (save-current-buffer (funcall cmd body params))) + (let ((handle-feedback + (org-babel--async-feedbacks info handle-result result-params exec-start-time))) + (funcall cmd body params handle-feedback)))))))))) + (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --] [-- Type: text/x-patch, Size: 2458 bytes --] From 8b311516505f693488bfef661905f34e7375539c Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:09 +0100 Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5] --- lisp/ob-core.el | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 0f0a36b70..1de94330a 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1092,9 +1092,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (make-directory d 'parents) d)))) (async (org-babel--async-p params)) - (cmd (intern (concat "org-babel-" - (if async "schedule" "execute") - ":" lang))) + (execute-with (let ((be (cdr (assq :execute-with params)))) + (when (equal be "none") (setq be nil)) + be)) + (cmd (intern (or (and execute-with + (concat execute-with "-" (if async "schedule" "execute"))) + (concat "org-babel-" + (if async "schedule" "execute") + ":" lang)))) + (cmd-args (let ((ps (list body params))) + (when execute-with + (setq ps (cons lang ps))) + ps)) (exec-start-time (current-time)) (handle-result (lambda (result) @@ -1163,10 +1172,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (format "(%s)" name) (format "at position %S" (nth 5 info))))) (if (not async) - (funcall handle-result (save-current-buffer (funcall cmd body params))) + (funcall handle-result (save-current-buffer (apply cmd cmd-args))) (let ((handle-feedback (org-babel--async-feedbacks info handle-result result-params exec-start-time))) - (funcall cmd body params handle-feedback)))))))))) + (apply cmd (nconc cmd-args (list handle-feedback)))))))))))) (defun org-babel-expand-body:generic (body params &optional var-lines) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --] [-- Type: text/x-patch, Size: 14462 bytes --] From 50ecb37089b96cd0fe4107e4d74357c151540e34 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:23 +0100 Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers --- lisp/org-elib-async.el | 323 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 323 insertions(+) create mode 100644 lisp/org-elib-async.el diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el new file mode 100644 index 000000000..ba4e08869 --- /dev/null +++ b/lisp/org-elib-async.el @@ -0,0 +1,323 @@ +;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*- + +;; Copyright (C) 2024 Bruno BARBIER + +;; Author: Bruno BARBIER +;; Version: 0.0.0 +;; Maintainer: Bruno BARBIER +;; Keywords: +;; Status: WORK IN PROGRESS. DO NOT USE. +;; URL: +;; Compatibility: GNU Emacs 30.0.50 +;; +;; This file is NOT (yet) part of GNU Emacs. + +;; This program is free software; you can redistribute it and/or +;; modify it under the terms of the GNU General Public License as +;; published by the Free Software Foundation; either version 2 of +;; the License, or (at your option) any later version. + +;; This program is distributed in the hope that it will be +;; useful, but WITHOUT ANY WARRANTY; without even the implied +;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +;; PURPOSE. See the GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public +;; License along with this program; if not, write to the Free +;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, +;; MA 02111-1307 USA + + +;;; Commentary: +;; Names with "--" are for functions and variables that are meant to be for +;; internal use only. + +;;;; Description +;; Some functions to help dealing with asynchronous tasks. + +;; The prefix 'org-elib' means that this package should evenutally be +;; moved into core Emacs. The functions defined here do NOT depend +;; nor rely on org itself. + +;;; TODOs +;; +;; - Keywords +;; + + +;;; Code: + +;;;; Process +;; +(cl-defun org-elib-async-process (command &key input callback) + "Execute COMMAND. + +A quick naive featureless boggus wrapper around `make-process' to +receive the result when the process is done. + +When INPUT is non-nil, use it as the COMMAND standard input. Let DATA +be the COMMAND output, if COMMAND succeeds, call CALLBACK with +'(:success DATA), else, call CALLBACK with '(:failure DATA)." + (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*")) + (get-outcome + (lambda (process) + (with-current-buffer stdout-buffer + (let* ((exit-code (process-exit-status process)) + (real-end ;; Getting rid of the user message. + (progn (goto-char (point-max)) + (forward-line -1) + (point))) + (txt (string-trim (buffer-substring-no-properties + (point-min) real-end)))) + (list (if (eq 0 exit-code) :success :failure) + (if (not (string-empty-p txt)) txt + (and (not (eq 0 exit-code)) exit-code))))))) + (process (make-process + :name "*org-elib-async-process*" + :buffer stdout-buffer + :command command + :connection-type 'pipe)) + (sentinel + (lambda (&rest _whatever) + (pcase (process-status process) + ('run ) + ('stop) + ((or 'exit 'signal) + (funcall callback (funcall get-outcome process))) + (_ (error "Not a real process")))))) + (add-function :after (process-sentinel process) sentinel) + (when input + (process-send-string process input) + (process-send-eof process)) + process)) +;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o))) + + +;;;; Wait for a process until some condition becomes true. + +(define-error 'org-elib-async-timeout-error + "Timeout waiting for a process.") + +(cl-defun org-elib-async-wait-condition ( cond-p + &key + (tick .3) (message "Waiting") + (nb_secs_between_messages 5) + timeout) + "Wait until the condition COND-P returns non-nil. +Repeatedly call COND-P with no arguments, about every TICK seconds, +until it returns a non-nil value. Return that non-nil value. When +TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if +the COND-P is still nil after TIMEOUT seconds. Assume COND-P calls cost +0s. Do NOT block display updates. Do not block process outputs. Do +not block idle timers. Do block the user, letting him/her know why, but +do not display more messages than one every NB_SECS_BETWEEN_MESSAGES. +Default MESSAGE is \"Waiting\". Use 0.3s as the default for TICK." + ;; FIXME: Still not sure if it's possible to write such a function. + (let ((keep-waiting t) + (result nil) + (start (float-time)) + elapsed + last-elapsed + ) + (while keep-waiting + (setq result (funcall cond-p)) + (if result + (setq keep-waiting nil) + (sleep-for 0.01) + (redisplay :force) + (setq elapsed (- (float-time) start)) + (when (and timeout (> elapsed timeout)) + (signal 'org-timeout-error message elapsed)) + ;; Let the user know, without flooding the message area. + (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages)) + (message (format "%s ...(%.1fs)" message elapsed))) + (unless (sit-for tick :redisplay) + ;; Emacs has something to do; let it process new + ;; sub-processes outputs in case there are some. + (accept-process-output nil 0.01)))) + result)) + + + +;;;; Comint: a FIFO queue of tasks with callbacks +;; org-elib-async-comint-queue executes tasks in a FIFO order. For each +;; task, it identifies the text output for that +;; task. org-elib-async-comint-queue does NOT remove prompts, or other +;; useless texts; this is the responsibility of the user. Currently, +;; org-elib-async-comint-queue assume it has the full control of the +;; session: no user interaction, no other direct modifications. + +(defvar-local org-elib-async-comint-queue--todo :NOT-SET + "A FIFO queue of pending executions.") + + +(defvar-local org-elib-async-comint-queue--unused-output "" + "Process output that has not been used yet.") + +(defvar-local org-elib-async-comint-queue--incoming-text "" + "Newly incoming text, added by the process filter, not yet handled.") + +(defvar-local org-elib-async-comint-queue--current-task nil + "The task that is currently running.") + +(defvar-local org-elib-async-comint-queue--process-filter-running nil + "non-nil when filter is running.") + +(defvar-local org-elib-async-comint-queue--incoming-timer nil + "A timer, when handling incoming text is scheduled or running.") + + +(defvar-local org-elib-async-comint-queue--handle-incoming-running + nil + "True when the incoming text handler is running.") + +(defun org-elib-async-comint-queue--handle-incoming () + (when org-elib-async-comint-queue--handle-incoming-running + (error "Bad filter call detected: Kill buffer %s!" (current-buffer))) + (setq org-elib-async-comint-queue--handle-incoming-running t) + + ;; Take the incoming text. + (setq org-elib-async-comint-queue--unused-output + (concat org-elib-async-comint-queue--unused-output + org-elib-async-comint-queue--incoming-text)) + (setq org-elib-async-comint-queue--incoming-text "") + + ;; Process the unused text with the queued tasks + (unless org-elib-async-comint-queue--current-task + (when org-elib-async-comint-queue--todo + (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo)))) + (when-let ((task org-elib-async-comint-queue--current-task)) + (let ((unused org-elib-async-comint-queue--unused-output) + (session-buffer (current-buffer)) + task-start) + (setq org-elib-async-comint-queue--unused-output + (with-temp-buffer + (insert unused) + (goto-char (point-min)) + (while (and task + (setq task-start (point)) + (search-forward (car task) nil t)) + (when (cdr task) + (let ((txt (buffer-substring-no-properties task-start + (- (point) (length (car task)))))) + (save-excursion (funcall (cdr task) txt)))) + (setq task (and (buffer-live-p session-buffer) + (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo))))) + (buffer-substring (point) (point-max)))) + (setq org-elib-async-comint-queue--current-task task))) + + ;; Signal that we are done. If we already have some new incoming text, + ;; reschedule to run. + (setq org-elib-async-comint-queue--incoming-timer + (if (string-empty-p org-elib-async-comint-queue--incoming-text) + nil + (org-elib-async-comint-queue--wake-up-handle-incoming))) + + ;; We reset it only on success. If it failed for some reason, the + ;; comint buffer is in an unknown state: you'll need to kill that + ;; buffer. + (setq org-elib-async-comint-queue--handle-incoming-running nil)) + + +(defun org-elib-async-comint-queue--wake-up-handle-incoming () + "Wake up the handling of incoming chunks of text. +Assume we are called from the comint buffer." + (setq org-elib-async-comint-queue--incoming-timer + (run-with-timer + 0.01 nil + (let ((comint-buffer (current-buffer))) + (lambda () + (with-local-quit + (with-current-buffer comint-buffer + (org-elib-async-comint-queue--handle-incoming)))))))) + + +(defun org-elib-async-comint-queue--process-filter (chunk) + "Accept the arbitrary CHUNK of text." + (setq org-elib-async-comint-queue--incoming-text + (concat org-elib-async-comint-queue--incoming-text + chunk)) + :; We delegate the real work outside the process filter, as it is + ; not reliable to do anything here. + (unless org-elib-async-comint-queue--incoming-timer + (org-elib-async-comint-queue--wake-up-handle-incoming))) + + + +(define-error 'org-elib-async-comint-queue-task-error + "Task failure.") + +(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback) + "Push the execution of EXEC into the FIFO queue. +When the task completed, call HANDLE-FEEDBACK with its outcome. Return +a function that waits for and return the result on succes, raise on +failure." + (let* ((tid (org-id-uuid)) + (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid)) + (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid)) + (result-sb (make-symbol "result")) + (on-start + (lambda (_) + ;; TODO: Use (point) in session to link back to it. + (when handle-feedback + (funcall handle-feedback '(:pending "running"))))) + (on-result + (lambda (result) + ;; Get the result, and report success using HANDLE-FEEDBACK. + ;; If something fails, report failure using HANDLE-FEEDBACK. + (unwind-protect + (let ((outcome + (condition-case-unless-debug exc + (list :success (funcall exec :post-process result)) + (error (list :failure exc))))) + (when handle-feedback (save-excursion (funcall handle-feedback outcome))) + (set result-sb outcome)) + (funcall exec :finally))))) + + (let ((comint-buffer (funcall exec :get-comint-buffer))) + (with-current-buffer comint-buffer + (setq org-elib-async-comint-queue--todo + (nconc org-elib-async-comint-queue--todo + (list (cons start-tag on-start) + (cons end-tag on-result)))) + (funcall exec :send-instrs-to-session + (funcall exec :instrs-to-enter)) + (funcall exec :send-instrs-to-session + (funcall exec :instr-to-emit-tag start-tag)) + (funcall exec :send-instrs-to-session + (funcall exec :get-code)) + (funcall exec :send-instrs-to-session + (funcall exec :instr-to-emit-tag end-tag)) + (funcall exec :send-instrs-to-session + (funcall exec :instrs-to-exit)) + + (lambda () + (org-elib-async-wait-condition (lambda () + (boundp result-sb))) + (pcase (symbol-value result-sb) + (`(:success ,r) r) + (`(:failure ,err) (signal (car err) (cdr err))))) + )))) + + + +(defun org-elib-async-comint-queue-init-if-needed (buffer) + "Initialize the FIFO queue in BUFFER if needed." + (with-current-buffer buffer + (unless (local-variable-p 'org-elib-async-comint-queue--todo) + (setq-local org-elib-async-comint-queue--todo nil) + (add-hook 'comint-output-filter-functions + #'org-elib-async-comint-queue--process-filter nil :local)))) + + + +;;;; Provide +(provide 'org-elib-async) +;;; org-elib-async.el ends here -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --] [-- Type: text/x-patch, Size: 3038 bytes --] From 8db393a06efc3e8a4a3a101be2e59125d1666817 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:40 +0100 Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails lisp/ob-core.el (org-babel-popup-failure-details): New function. (org-babel-execute-src-block): For synchronous execution, use `org-babel-popup-failure-details' on failure. (org-babel--async-feedbacks): Add call to `org-babel-popup-failure-details' on user request. --- lisp/ob-core.el | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 1de94330a..c0f6a398a 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -923,6 +923,16 @@ (defun org-babel--async-feedbacks (info handle-result (format "%s" msg)) (short-version-of msg))) 'face (org-babel--async-status-face status))) + (when (eq :failure status) + (overlay-put ovl-title + 'keymap + (let ((km (make-sparse-keymap))) + (define-key km (kbd "<mouse-1>") + (lambda () + "Display failure details." + (interactive) + (org-babel-popup-failure-details msg))) + km))) )) (remove-previous-overlays () ;; Remove previous title and body overlays. @@ -1172,11 +1182,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (format "(%s)" name) (format "at position %S" (nth 5 info))))) (if (not async) - (funcall handle-result (save-current-buffer (apply cmd cmd-args))) + (let ((res-sb (make-symbol "result"))) + (condition-case exc + (set res-sb (save-current-buffer (apply cmd cmd-args))) + (error (org-babel-popup-failure-details exc))) + (when (boundp res-sb) + (funcall handle-result (symbol-value res-sb)))) (let ((handle-feedback (org-babel--async-feedbacks info handle-result result-params exec-start-time))) (apply cmd (nconc cmd-args (list handle-feedback)))))))))))) +(defun org-babel-popup-failure-details (exc) + "Notify/display" + (when-let ((buf (get-buffer org-babel-error-buffer-name))) + (with-current-buffer buf (erase-buffer))) + (org-babel-eval-error-notify + 127 ; Don't have exit-code + (if (consp exc) + (format "%s\n%s\n" (car exc) (cdr exc)) + (format "%s\n" exc)))) + (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #9: 0008-scratch-bba-ob-core-async-Some-temporary-test-files.patch --] [-- Type: text/x-patch, Size: 20479 bytes --] From 69db5eef49c69287270b1f171e53ddb5e76625dd Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:33 +0100 Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files --- scratch/bba-ob-core-async/my-async-tests.el | 172 +++++++ scratch/bba-ob-core-async/my-async-tests.org | 484 +++++++++++++++++++ 2 files changed, 656 insertions(+) create mode 100644 scratch/bba-ob-core-async/my-async-tests.el create mode 100644 scratch/bba-ob-core-async/my-async-tests.org diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el new file mode 100644 index 000000000..3550fc88f --- /dev/null +++ b/scratch/bba-ob-core-async/my-async-tests.el @@ -0,0 +1,172 @@ +;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*- + +;; Copyright (C) 2024 Bruno BARBIER + +;; Author: Bruno BARBIER +;; Status: Temporary tests. +;; Compatibility: GNU Emacs 30.0.50 +;; +;; This file is NOT part of GNU Emacs. + +;; This program is free software; you can redistribute it and/or +;; modify it under the terms of the GNU General Public License as +;; published by the Free Software Foundation; either version 2 of +;; the License, or (at your option) any later version. + +;; This program is distributed in the hope that it will be +;; useful, but WITHOUT ANY WARRANTY; without even the implied +;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +;; PURPOSE. See the GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public +;; License along with this program; if not, write to the Free +;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, +;; MA 02111-1307 USA + +(require 'org) +(require 'org-elib-async) + +(defun my-shell-babel-schedule (lang body params handle-feedback) + "Execute the bash script BODY. +Execute the shell script BODY using bash. Use HANDLE-FEEDBACK to report +the outcome (success or failure)." + (unless (equal "bash" lang) + (error "Only for bash")) + (funcall handle-feedback (list :pending "started")) + (org-elib-async-process (list "bash") :input body :callback handle-feedback)) + + +(defun my-org-babel-python-how-to-execute (body params) + "Return how to execute BODY using python. +Return how to execute, as expected by +`org-elib-async-comint-queue--execution'." + ;; Code mostly extracted from ob-python, following + ;; `org-babel-python-evaluate-session'. + ;; Results are expected to differ from ob-python as we follow the + ;; same process for all execution paths: asynchronous or not, with + ;; session or without. + (let* ((org-babel-python-command + (or (cdr (assq :python params)) + org-babel-python-command)) + (session-key (org-babel-python-initiate-session + (cdr (assq :session params)))) + (graphics-file (and (member "graphics" (assq :result-params params)) + (org-babel-graphical-output-file params))) + (result-params (cdr (assq :result-params params))) + (result-type (cdr (assq :result-type params))) + (results-file (when (eq 'value result-type) + (or graphics-file + (org-babel-temp-file "python-")))) + (return-val (when (eq result-type 'value) + (cdr (assq :return params)))) + (preamble (cdr (assq :preamble params))) + (full-body + (concat + (org-babel-expand-body:generic + body params + (org-babel-variable-assignments:python params)) + (when return-val + (format "\n%s" return-val)))) + (post-process + (lambda (r) + (setq r (string-trim r)) + (when (string-prefix-p "Traceback (most recent call last):" r) + (signal 'user-error (list r))) + (when (eq 'value result-type) + (setq r (org-babel-eval-read-file results-file))) + (org-babel-reassemble-table + (org-babel-result-cond result-params + r + (org-babel-python-table-or-string r)) + (org-babel-pick-name (cdr (assq :colname-names params)) + (cdr (assq :colnames params))) + (org-babel-pick-name (cdr (assq :rowname-names params)) + (cdr (assq :rownames params)))))) + (tmp-src-file (org-babel-temp-file "python-")) + (session-body + ;; The real code we evaluate in the session. + (pcase result-type + (`output + (format (string-join + (list "with open('%s') as f:\n" + " exec(compile(f.read(), f.name, 'exec'))\n")) + (org-babel-process-file-name + tmp-src-file 'noquote))) + (`value + ;; FIXME: In this case, any output is an error. + (org-babel-python-format-session-value + tmp-src-file results-file result-params)))) + comint-buffer + finally) + + + (unless session-key + ;; No session. We create a temporary one and use 'finally' to + ;; destroy it once we are done. + ;; + ;; FIXME: This session code should be refactored and moved into + ;; ob-core. + (setq session-key (org-babel-python-initiate-session + ;; We can't use a simple `generate-new-buffer' + ;; due to the earmuffs game. + (org-babel-python-without-earmuffs + (format "*ob-python-no-session-%s*" (org-id-uuid))))) + (setq finally (lambda () + (when-let ((s-buf + (get-buffer (org-babel-python-with-earmuffs session-key)))) + ;; We cannot delete it immediately as we are called from it. + (run-with-idle-timer + 0.1 nil + (lambda () + (when (buffer-live-p s-buf) + (let ((kill-buffer-query-functions nil) + (kill-buffer-hook nil)) + (kill-buffer s-buf))))))))) + + (org-elib-async-comint-queue-init-if-needed session-key) + (setq comint-buffer + (get-buffer (org-babel-python-with-earmuffs session-key))) + (with-temp-file tmp-src-file + (insert (if (and graphics-file (eq result-type 'output)) + (format org-babel-python--output-graphics-wrapper + full-body graphics-file) + full-body))) + + (lambda (&rest q) + (pcase q + (`(:instrs-to-enter) + ;; FIXME: This is wrong. + "import sys; sys.ps1=''; sys.ps2=''") + (`(:instrs-to-exit)) + (`(:finally) (when finally (funcall finally))) + (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag)) + (`(:post-process ,r) (when post-process (funcall post-process r))) + (`(:send-instrs-to-session ,code) + ;; See org-babel-python-send-string + (with-current-buffer comint-buffer + (let ((python-shell-buffer-name + (org-babel-python-without-earmuffs session-key))) + (python-shell-send-string (concat code "\n"))))) + (`(:get-code) session-body) + (`(:get-comint-buffer) comint-buffer) + (_ (error "Unknown query")))))) + + + +(defun my-org-babel-schedule (lang body params handle-feedback) + "Schedule the execution of BODY according to PARAMS. +This function is called by `org-babel-execute-src-block'. Return a +function that waits and returns the result on success, raise on failure." + (cl-assert (equal "python" lang)) + (let ((exec (my-org-babel-python-how-to-execute body params))) + (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback))) + + +(defun my-org-babel-execute (lang body params) + "Execute Python BODY according to PARAMS. +This function is called by `org-babel-execute-src-block'." + ;; We just start the asynchronous execution, wait for it, and return + ;; the result (or raise the exception). No custom code, and, + ;; synchronous and asynchronous should just mix nicely together. + (cl-assert (equal "python" lang)) + (funcall (my-org-babel-schedule lang body params nil))) diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org new file mode 100644 index 000000000..263ca77a2 --- /dev/null +++ b/scratch/bba-ob-core-async/my-async-tests.org @@ -0,0 +1,484 @@ +#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both +* Intro + +An org document with code blocks to help test the proposed patches. + +You need to load: + #+begin_src elisp :results silent + (load-file "my-async-tests.el") + #+end_src + + + +Emacs and org versions: + #+begin_src elisp + (mapcar (lambda (sb) (list sb (symbol-value sb))) + '(emacs-version org-version)) + #+end_src + + #+RESULTS: + | emacs-version | 30.0.50 | + | org-version | 9.7-pre | + +Note that we've disabled eval on export: export doesn't know it needs +to wait for asynchronous results. + +* A simple bash example + :PROPERTIES: + :header-args:bash: :execute-with my-shell-babel :nasync yes + :END: + +The package `my-async-tests.el' contains the function +`my-shell-babel-schedule' to evaluate shell script asynchronously. + +The header-args properties above request asynchronous execution for +bash (:nasync yes), and, tells ob-core to use the prefix +`my-shell-babel' when looking for functions to evaluate a source +block. Thus, org will delegate execution to `my-shell-babel-schedule'. +We don't have `my-shell-babel-execute', so, in this case, :nasync must +be yes. + +A simple execution: + #+begin_src bash + date + #+end_src + + #+RESULTS: + : Fri Feb 16 18:11:08 CET 2024 + +A tricky computation takes some time: + #+begin_src bash + sleep 5; date + #+end_src + + #+RESULTS: + : Fri Feb 16 17:58:23 CET 2024 + +An example of a failure: + #+begin_src bash + sleepdd 1; false + #+end_src + + #+RESULTS: + +* Python + :PROPERTIES: + :header-args:python: :execute-with my-org-babel :nasync yes + :header-args:python+: :session py-async + :END: + +Used =header-args= properties: + - =:execute-with my-org-babel=: look for functions with the prefix `my-org-babel' to execute + blocks (for the asynchronous case use + `my-org-babel-schedule', and, for the synchronous case + `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]]. + + - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule'). + + - =:session py-async= by default, use a session named "py-async". + +** basic examples +*** async with a session +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + + #+RESULTS: + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708103472.9s | 1708103473.9s | 1.0s | + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +*** async with no session + :PROPERTIES: + :header-args:python+: :session none + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + + #+RESULTS: + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708083470.9s | 1708083471.9s | 1.0s | + +Yes, it failed, as expected. "import time" was done in its own +temporary session. The old result is preserved; the error is display +as an overlay. Click on it to get more info about the error. + + +Let's fix it, adding the import line: + #+begin_src python + import time + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708102948.9s | 1708102949.9s | 1.0s | + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + + +*** sync with a session + :PROPERTIES: + :header-args:python+: :session py-sync-session :nasync no + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708102997.5s | 1708102998.5s | 1.0s | + + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +*** sync with no session + :PROPERTIES: + :header-args:python+: :session none :nasync no + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708083470.9s | 1708083471.9s | 1.0s | + +Yes, that fails (no session), displaying the details in a popup. Let's +fix it: + #+begin_src python + import time + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708103039.0s | 1708103040.0s | 1.0s | + + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +** worg examples + +Let's import matplotlib in our session. + + #+begin_src python + import matplotlib + import matplotlib.pyplot as plt + #+end_src + + #+RESULTS: + : None + +A figure in a PDF, asynchronous case. + #+begin_src python :results file link + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + + fname = 'myfig-async.pdf' + plt.savefig(fname) + fname # return this to org-mode + #+end_src + + #+RESULTS: + [[file:myfig-async.pdf]] + + +A figure in a PDF, synchronous case. + #+begin_src python :results file link :nasync no + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + + fname = 'myfig-sync.pdf' + plt.savefig(fname) + fname # return this to org-mode + #+end_src + + #+RESULTS: + [[file:myfig-sync.pdf]] + + + +A PNG figure, asynchronous case. + #+begin_src python :results graphics file output :file boxplot.png + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + fig + #+end_src + + #+RESULTS: + [[file:boxplot.png]] + +Same, but using the =:return= keyword. + #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + #+end_src + + #+RESULTS: + [[file:boxplot.png]] + +Same, asynchronous but without a session this time. + #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none + import matplotlib + import matplotlib.pyplot as plt + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + #+end_src + + #+RESULTS: + [[file:boxplot-no-sess-a-y.png]] + + +Lists are table, + #+begin_src python + [1,2,3] + #+end_src + + #+RESULTS: + | 1 | 2 | 3 | + +unless requested otherwise. + #+begin_src python :results verbatim + [1,2,3] + #+end_src + + #+RESULTS: + : [1, 2, 3] + + +Dictionaries are tables too. + #+begin_src python :results table + {"a": 1, "b": 2} + #+end_src + + #+RESULTS: + | a | 1 | + | b | 2 | + + +Let's try the example with Panda. + #+begin_src python :results none + import pandas as pd + import numpy as np + #+end_src + + #+RESULTS: + : None + + #+begin_src python :results table + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + +And the synchronous case? + + #+begin_src python :results table :nasync no + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + + + +Without session ? + + #+begin_src python :results table :session none + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + +Right, we need to import the libraries (no session). + + #+begin_src python :results table :session none + import pandas as pd + import numpy as np + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + + +** inline examples + + A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}. + + An other one containing a mistake src_python{2/0} {{{results(=6=)}}} + (click on the error to see the details). + + + Some very slow inline asynchronous computations that all run in + the same session. You need to execute the 3 of them at once. Here + is the first one src_python[:return "\"OK1\""]{import time; + time.sleep(5)} {{{results(=OK1=)}}} and a second one + src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} + {{{results(=OK1 bis=)}}} and the third one src_python[:return + "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}. + + Yes, the previous paragraph is unreadable; it's on purpose, to + check that ob-core can figure it out. + + Let's repeat, in a more readable way, and making the last one + synchronous. + + Some very slow inline computations that all run in the same + session. Here is the first asynchronous one + src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}} + and a second one, asynchronous too: + src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}} + and finally, a third one, synchronous this one: + src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}. + + Note that, once the user executes the last synchronous block, the + user is blocked until the synchronous execution can start + (i.e. all previous asynchronous executions are done) and until + it's done. The display is updated though, to see the asynchronous + progress. -- 2.43.0 ^ permalink raw reply related [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-16 17:52 ` Bruno Barbier @ 2024-02-18 21:14 ` Matt 2024-02-19 0:31 ` Jack Kamm ` (3 more replies) 2024-02-19 0:15 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm 2024-02-19 9:06 ` Ihor Radchenko 2 siblings, 4 replies; 55+ messages in thread From: Matt @ 2024-02-18 21:14 UTC (permalink / raw) To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 7215 bytes --] ---- On Fri, 16 Feb 2024 18:52:22 +0100 Bruno Barbier > Sorry for the late reply. Cleaning the code took me longer than expected. No need to apologize, we're all volunteers here :) > Feedbacks, corrections, critiques, etc are most welcome! Thank you for sharing! If I understand correctly, there are several independent topics the code addresses: | topic | manner addressed | |------------------+------------------------------------------------| | execution status | using overlays to communicate execution status | | locating results | using overlays to locate results | | blocking | making all execution asynchronous | |------------------+------------------------------------------------| I suggest these be discussed in separate threads. > > The use of the overlay is a really cool idea! > > > > I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. > > I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document). The backend is free to transform a failure into a success to make that result available though. You can disregard my hesitation on this point. I had not run your code yet and had misunderstood how it worked. Since this thread is dedicated to blocking, let me share my thoughts on that subject. > To execute Python blocks, using the proposed async API: > > - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python). > > - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...). > > - The "no session" case is just about creating a new unique session and throwing it away immediately. This is an interesting idea, feeding all processes through the same mechanism. Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]]. Processes are synchronous or asynchronous. Three primitives exist in Emacs for making processes: 1. make-process (asynchronous) 2. call-process (synchronous) 3. call-process-region (synchronous) There exist several convenience wrappers for these. AFAIK, everything reduces to these three primitives. For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal. This background-foreground distinction is called "job control". Output from a process typically goes to a buffer. This may be changed and instead handle output with a filter function. =call-process= has an option to directly send output to a file. Subprocesses inherent the =default-directory= and the environment from Emacs. The environment may be changed using =process-environment=. There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe". The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.). In my previous message, I divided evaluation into 4 types: - non-persistent vs. persistent - synchronous vs. asynchronous I find the approach of feeding everything through, fundamentally, =make-process= interesting because if we make a chart of the 4 types, we see some ambiguities: | | non-persistent | persistent | |--------------+----------------+--------------| | synchronous | call-process | ??? | |--------------+----------------+--------------| | asynchronous | ??? | make-process | |--------------+----------------+--------------| To make a non-persistent asynchronous process, the first thing that comes to mind is =async-shell-command=. However, as the code shows, another option is to use =make-process= and throw away the state (the process buffer). I'm not sure how we could make a persistent, synchronous process. Persistence is achieved, currently, by a process buffer. Is there another way persistence may be achieved? Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. All that is mainly academic. The idea I think most interesting is using a single primitive to handle all evaluation. It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached). It works as follows. I defined a new Babel "language" called "blub". Blub could be a shell, python, ruby, whatever. I wanted to test that the implementation could work with different interpreters or compilers. Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out. Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. Curious to hear other people's thoughts! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode [-- Attachment #2: ob-blub-test.org --] [-- Type: application/octet-stream, Size: 2532 bytes --] #+begin_src emacs-lisp :results silent :var HERE=(buffer-file-name) ;; load blub (add-to-list 'load-path (file-name-directory HERE)) (require 'ob-blub) (org-babel-do-load-languages 'org-babel-load-languages '((blub . t))) ;; reset uuid on failed block (setq my-org-babel-comint--async-uuid nil) ;; configure shell (setq org-babel-blub-interpreter shell-file-name) (setq org-babel-blub-remove-prompt-command "PROMPT_COMMAND=;PS1=;PS2=;") (setq org-babel-blub-output-start-delimiter "echo \"start_%s\"") (setq org-babel-blub-output-end-delimiter "echo \"end_%s\"") (setq org-babel-blub-interpreter-args '()) ;; configure python ;; (setq org-babel-blub-interpreter "python3") ;; (setq org-babel-blub-remove-prompt-command "import sys;sys.ps1='';sys.ps2='';") ;; (setq org-babel-blub-output-start-delimiter "print(\"start_%s\")") ;; (setq org-babel-blub-output-end-delimiter "print(\"end_%s\")") ;; (setq org-babel-blub-interpreter-args '()) ;; configure ruby ;; (setq org-babel-blub-interpreter "ruby") ; for non-sessions ;; (setq org-babel-blub-interpreter "irb") ; for sessions ;; (setq org-babel-blub-remove-prompt-command nil) ;; (setq org-babel-blub-output-start-delimiter "puts \"start_%s\"") ;; (setq org-babel-blub-output-end-delimiter "puts \"end_%s\"") ;; (setq org-babel-blub-interpreter-args '("--noprompt" "--noreadline" "--nomultiline")) #+end_src * Non-persistent ** shell #+begin_src blub echo "hello" sleep 3 echo "world!" #+end_src #+RESULTS: : hello : world! #+begin_src blub ==echo "hello" sleep 3 echo "world!" #+end_src ** python #+begin_src blub import time print("hello without session") time.sleep(3) print("world") #+end_src ** ruby #+begin_src blub puts "cruel world" sleep(3) puts "good-bye" #+end_src * Persistent ** shell #+begin_src blub :session *shell-blubber* echo "hello" sleep 3 echo "world!" #+end_src #+begin_src blub :session *shell-blubber* ==echo "hello" sleep 3 echo "world!" #+end_src ** python #+begin_src blub :session *python-blubber* import time print("hello") time.sleep(5) print("world") #+end_src #+RESULTS: #+begin_src blub :session *python-blubber* import time print("good-bye") time.sleep(5) print("cruel world") #+end_src ** ruby #+begin_src blub :session *ruby-blubber* puts "good-bye" sleep(3) puts "cruel world" #+end_src * Failures #+begin_src blub :session *bash-blubber* ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt #+end_src #+begin_src blub :session *shell-blubber* :epilogue echo "bye" ssh $USER@localhost echo "hi" #+end_src [-- Attachment #3: ob-blub.el --] [-- Type: application/octet-stream, Size: 10821 bytes --] ;; -*- lexical-binding: t -*- ;;; ob-blub.el --- org-babel functions for blub evaluation ;; Copyright (C) Matt Trzcinski ;; Author: Matt Trzcinski ;; Keywords: literate programming, reproducible research ;; Homepage: https://orgmode.org ;; Version: 0.01 ;;; License: ;; This program is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation; either version 3, or (at your option) ;; any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs; see the file COPYING. If not, write to the ;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, ;; Boston, MA 02110-1301, USA. ;;; Commentary: ;; ;; A new language, blub, is defined using the Org Babel API ;; (`org-babel-execute:blub'). "Blub" is a dummy language. If blub ;; is interpreted, set the interpreter using ;; `org-babel-blub-interpreter'. Compiled languages are not ;; demonstrated here although the implementation may be extended to ;; support compiled languages. ;; ;; The code is split into three sections: ;; ;; 1. Eval ;; 2. Comint ;; 3. Blub ;; ;; Sections are based on their current implements in Org Babel. It's ;; not clear if these are the "right" categorizations. For example, ;; should comint evaluation be in ob-comint or ob-eval? ;; ;; Set the following: ;; ;; `org-babel-blub-interpreter' ;; `org-babel-blub-interpreter-args' ;; `org-babel-blub-remove-prompt-command' ;; `org-babel-blub-output-start-delimiter' ;; `org-babel-blub-output-end-delimiter' ;; ;; Sessions run in a dedicated process buffer. Output is captured ;; from between delimiters. This is similar to the current Babel ;; implementation. However, unlike the current Babel implementation, ;; both delimiters in this proof of concept are custom. The current ;; Babel implementation uses the prompt for the start delimiter which ;; causes problems when it changes. ;; ;; For sessions, only a single process may run at a time. The UUID is ;; stored in `my-org-babel-comint--async-uuid' and the associated Org ;; buffer is stored in `my-org-babel-comint--async-org-buffer'. If an ;; error occurs that prevents the end delimiter from printing, ;; manually the clear `my-org-babel-comint--async-uuid' to run blocks. ;; ;; (setq my-org-babel-comint--async-uuid nil) ;; ;; Results are extracted from standard output using a regexp. ;; ;; Noted problems: ;; - running sudo ;;; Requirements: (require 'ob) \f ;;; Eval: (defun my-org-babel-eval-async (command body) "Start process with COMMAND, send BODY to process, get results. Since results execute asynchronously, a UUID is returned. The UUID is later replaced when the async process finishes." (setq my-org-babel-comint--async-uuid (org-id-uuid)) (setq my-org-babel-comint--async-org-buffer (current-buffer)) (let* ((my-process (make-process :name "my-org-babel-eval-async" :buffer "*my-org-babel-eval-async*" :command `(,command) :connection-type 'pipe :sentinel '(lambda (process msg) (cond ((string= msg "finished\n") (my-org-babel-comint-replace-uuid-with-results my-org-babel-comint--async-org-buffer my-org-babel-comint--async-uuid (with-current-buffer (process-buffer process) (buffer-substring-no-properties (point-min) (point-max)))) (setq my-org-babel-comint--async-uuid nil) (let ((kill-buffer-query-functions nil)) (if (kill-buffer "*my-org-babel-eval-async*") (setq my-org-babel-comint--async-org-buffer nil))) )))))) (process-send-string my-process body) (process-send-eof my-process)) my-org-babel-comint--async-uuid) \f ;;; Comint: (defvar my-org-babel-comint--async-uuid nil "Placeholder for results.") (defvar my-org-babel-comint--async-org-buffer nil "Buffer containing UUID.") (defun my-org-babel-comint-send-string (session string) "Send STRING to comint SESSION." (with-current-buffer session (goto-char (process-mark (get-buffer-process session))) (insert string) (comint-send-input))) (defun my-org-babel-comint-replace-uuid-with-results (buffer uuid results) "Replace UUID string in BUFFER with RESULTS string." (with-current-buffer buffer (save-excursion (goto-char (point-min)) (when (search-forward uuid nil t) (org-babel-previous-src-block) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (result-params (cdr (assq :result-params params)))) (org-babel-insert-result results result-params info)))))) (defun my-org-babel-comint-send-to-session-async (process-buffer &rest body) "Send BODY to PROCESS-BUFFER asynchronously." (setq my-org-babel-comint--async-uuid (org-id-uuid)) (setq my-org-babel-comint--async-org-buffer (current-buffer)) (defun my-org-babel-comint--async-filter (text) "Check TEXT for ending delimiter and replace results held by `my-org-babel-comint--async-uuid' placeholder." (cond ((string-match-p (format "end_%s" my-org-babel-comint--async-uuid) text) (remove-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter) ;; replace my-org-babel-comint--async-uuid in Org buffer (let ((results (with-current-buffer process-buffer ; e.g. "*blubber*" (goto-char (point-min)) (re-search-forward ;; Of course, all the problems with regexp happen here. The goal is getting ;; the text between the delimiters. ;; ;; Some programs (guix shell?) may reset PS1. So, we can't always match on ;; start_uuid being at the very start of the line. Match on the one that ;; doesn't have the quote (that is, the result of the echo). (format "[^\"]start_%s[\r\n]*\\(\\(.*[\r\n]+\\)*.*\\)end_%s$" my-org-babel-comint--async-uuid ; start my-org-babel-comint--async-uuid) ; end nil nil 1) (let ((match (match-string 1))) (substring-no-properties match))))) (my-org-babel-comint-replace-uuid-with-results my-org-babel-comint--async-org-buffer my-org-babel-comint--async-uuid results) (setq my-org-babel-comint--async-uuid nil) (setq my-org-babel-comint--async-org-buffer nil))))) (let* ((proc (get-buffer-process process-buffer))) (with-current-buffer process-buffer (add-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter) (goto-char (process-mark proc)) ;; TODO need better abstraction (insert (format org-babel-blub-output-start-delimiter my-org-babel-comint--async-uuid)) (comint-send-input nil t) (goto-char (process-mark proc)) (insert (car body)) (comint-send-input nil t) (goto-char (process-mark proc)) ;; TODO need better abstraction (insert (format org-babel-blub-output-end-delimiter my-org-babel-comint--async-uuid)) (comint-send-input nil t))) my-org-babel-comint--async-uuid) \f ;;; Blub: (defvar org-babel-blub-interpreter shell-file-name ; bash/sh ;; "irb" ; ruby ;; "python3" ; python "Blub interpreter command or executable.") (defvar org-babel-blub-interpreter-args '() ; bash ;; '("--noprompt" "--noreadline" "--nomultiline") ; ruby "Blub interpreter command or executable arguments.") (defvar org-babel-blub-remove-prompt-command "PROMPT_COMMAND=;PS1=;PS2=;" ; bash ;; "import sys;sys.ps1='';sys.ps2='';" ; python "Command(s) to remove interpreter prompt.") (defvar org-babel-blub-output-start-delimiter "echo \"start_%s\"" ; bash ;; "puts \"start_%s\"" ; ruby ;; "print(\"start_%s\")" ; python "Format expression for writing the start delimiter to standard output in blub.") (defvar org-babel-blub-output-end-delimiter "echo \"end_%s\"" ; bash ;; "puts \"end_%s\"" ; ruby ;; "print(\"end_%s\")" ; python "Format expression for writing the end delimiter to standard output in blub.") ;; this removes the need for `org-babel-comint-buffer-livep' (defun org-babel-blub-get-session (session) "Return SESSION buffer, making process buffer if none exists." (cond ((not (get-buffer-process session)) (apply #'make-comint-in-buffer `(,session ; process name ,session ; buffer name ,org-babel-blub-interpreter ; program nil ; start file ,@org-babel-blub-interpreter-args)) (org-babel-comint-wait-for-output session) (if org-babel-blub-remove-prompt-command (my-org-babel-comint-send-string session org-babel-blub-remove-prompt-command)) ;; Needed for Emacs 23 since the marker is initially ;; undefined and the filter functions try to use it without ;; checking. (with-current-buffer session (set-marker comint-last-output-start (point))) ;; return shell buffer (get-buffer session)) (t session))) (defun org-babel-execute:blub (body params) "Execute BODY of Blub code with org-babel." (let* ((session (cdr (assq :session params)))) (cond ((and session (not (string= session "none")) (org-babel-blub-get-session session)) (if (not my-org-babel-comint--async-uuid) (my-org-babel-comint-send-to-session-async session body) ;; TODO prompt user to just go ahead with it (clobbering ;; the "existing" process) (error "Block already running. Call `(setq my-org-babel-comint--async-uuid nil)' to run a new process"))) (t (my-org-babel-eval-async org-babel-blub-interpreter (org-trim body)))))) (provide 'ob-blub) ;;; ob-blub.el ends here ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-18 21:14 ` Matt @ 2024-02-19 0:31 ` Jack Kamm 2024-02-20 10:28 ` Ihor Radchenko ` (2 subsequent siblings) 3 siblings, 0 replies; 55+ messages in thread From: Jack Kamm @ 2024-02-19 0:31 UTC (permalink / raw) To: Matt, Bruno Barbier; +Cc: Ihor Radchenko, emacs-orgmode Matt <matt@excalamus.com> writes: > The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. > > Curious to hear other people's thoughts! I don't remember the details, but my past self [1] thought it would be good to find a way to replace `process-file' with `make-process' in `org-babel--shell-command-on-region' or `org-babel-eval', and it seems you are thinking along those lines with `my-org-babel-eval-async'. Hope you're able to make progress on this and get the improvements into ob-eval.el eventually. [1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-18 21:14 ` Matt 2024-02-19 0:31 ` Jack Kamm @ 2024-02-20 10:28 ` Ihor Radchenko 2024-02-20 10:46 ` tomas 2024-02-21 15:27 ` Bruno Barbier [not found] ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac> 3 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-20 10:28 UTC (permalink / raw) To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode Matt <matt@excalamus.com> writes: > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). Doing everything asynchronously is not always desired. Consider, for example, #+begin_src bash echo "Contents" > /tmp/tmpfile #+end_src bash #+begin_src bash cat /tmp/tmpfile # I must run after /tmp/tmpfile is created! #+end_src Also, using "print" statements to create output delimiters might sometimes be tricky. If I remember correctly Ruby repl is asynchronous and sometimes produces unexpected artefacts when you send multiple commands in quick succession. The commands may be executed in different order than you may expect. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-20 10:28 ` Ihor Radchenko @ 2024-02-20 10:46 ` tomas 2024-02-20 11:00 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: tomas @ 2024-02-20 10:46 UTC (permalink / raw) To: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1267 bytes --] On Tue, Feb 20, 2024 at 10:28:06AM +0000, Ihor Radchenko wrote: > Matt <matt@excalamus.com> writes: > > > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). > > Doing everything asynchronously is not always desired. > Consider, for example, > > #+begin_src bash > echo "Contents" > /tmp/tmpfile > #+end_src bash > > #+begin_src bash > cat /tmp/tmpfile # I must run after /tmp/tmpfile is created! > #+end_src You "just" [1] need a way of stating dependencies :-) Cheers [1] In quotes, because this opens a vast space of interesting and strange worlds. The functional folks have their monads, the compiler backend builders have their dependency graphs. Definitely doable, but treading with care will be helpful to not step into a mess :-) Cheers -- t [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-20 10:46 ` tomas @ 2024-02-20 11:00 ` Ihor Radchenko 2024-02-20 11:03 ` tomas 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-20 11:00 UTC (permalink / raw) To: tomas; +Cc: emacs-orgmode <tomas@tuxteam.de> writes: >> Doing everything asynchronously is not always desired. >> Consider, for example, >> >> #+begin_src bash >> echo "Contents" > /tmp/tmpfile >> #+end_src bash >> >> #+begin_src bash >> cat /tmp/tmpfile # I must run after /tmp/tmpfile is created! >> #+end_src > > You "just" [1] need a way of stating dependencies :-) > > Cheers > > [1] In quotes, because this opens a vast space of interesting > and strange worlds. The functional folks have their monads, > the compiler backend builders have their dependency graphs. > Definitely doable, but treading with care will be helpful > to not step into a mess :-) Let's not jump into this rabbit hole yet before we have async code working for less complicated scenarios. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-20 11:00 ` Ihor Radchenko @ 2024-02-20 11:03 ` tomas 0 siblings, 0 replies; 55+ messages in thread From: tomas @ 2024-02-20 11:03 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 536 bytes --] On Tue, Feb 20, 2024 at 11:00:28AM +0000, Ihor Radchenko wrote: > <tomas@tuxteam.de> writes: [...] > > You "just" [1] need a way of stating dependencies :-) [...] > Let's not jump into this rabbit hole yet before we have async code > working for less complicated scenarios. :-) My take is: it's never too early to *think* about it (and to have a look what others are doing), but it might yet be too early to design something. Other than that... rabbit holes is what makes things interesting. Cheers -- t [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-18 21:14 ` Matt 2024-02-19 0:31 ` Jack Kamm 2024-02-20 10:28 ` Ihor Radchenko @ 2024-02-21 15:27 ` Bruno Barbier [not found] ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac> 3 siblings, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-21 15:27 UTC (permalink / raw) To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode Hi Matt, Thanks for your answer. Matt <matt@excalamus.com> writes: > ... > If I understand correctly, there are several independent topics the code addresses: > > | topic | manner addressed | > |------------------+------------------------------------------------| > | execution status | using overlays to communicate execution status | > | locating results | using overlays to locate results | > | blocking | making all execution asynchronous | > |------------------+------------------------------------------------| > > I suggest these be discussed in separate threads. Actually my patch is only about reporting execution status and inserting results, in a generic way, in ob-core. I provided some demo code on top of them, just to show how it could be used. Sorry about the confusion. > ... > > Since this thread is dedicated to blocking, let me share my thoughts on that subject. I guess I should start a new thread then :) > ... > > > Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]]. > > Processes are synchronous or asynchronous. > > Three primitives exist in Emacs for making processes: > > 1. make-process (asynchronous) > 2. call-process (synchronous) > 3. call-process-region (synchronous) > > There exist several convenience wrappers for these. AFAIK, everything reduces to these three primitives. For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal. This background-foreground distinction is called "job control". > > Output from a process typically goes to a buffer. This may be changed and instead handle output with a filter function. =call-process= has an option to directly send output to a file. > I've reached the same conclusion. As I wanted one simple implementation, I was thinking about using 'make-process', but, as we have to deal with the output manually and use a process sentinel, and, as offering a REPL to the user is nice, my personal pick is to use a plain shell with comint. > Subprocesses inherent the =default-directory= and the environment from Emacs. The environment may be changed using =process-environment=. > One wonderful thing about using =default-directory= is that, if done right, tramp will automatically handle executions on one or more remote computers. > There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe". The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.). I'm personally ignoring the difference between pty and pipe: as I don't use the terminal features, it doesn't really matter to me. > ... > I'm not sure how we could make a persistent, synchronous process. Persistence is achieved, currently, by a process buffer. Is there another way persistence may be achieved? If I understand correcly, "persistent" here means to preserve some state between executions. They are definitely other way to preserve and provide a state. If you focus on processes only, it's probably OK. But, if you use threads or some other kinds of asynchronicity, that might be too limiting to require a buffer and a process. > Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. > I'm not so sure. If the success of an execution is really important, and, intrinsically unreliable (like a network connection), and, it only takes a few seconds, I'll probably choose synchronous execution, just to see that everything is OK, before moving to my next task. > ... > Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. Thanks for sharing. I didn't have time to try it yet. But it looks like you could use the 'execute-with' keyword that my patchs provide. That way, you may redirect evaluation to use your "blub" implementation, and still use the real language name for your code blocks (that way, org knows how to fontify them, edit them, etc). And, you will not have to manually modify and reevaluate your elisp when switching languages. > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). I'll try it. Thanks, > The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. > > Curious to hear other people's thoughts! One think that comes to mind is to check that our testsuite covers everything we might break. If 'execute-with' gets added to org (from my patchs), that may provide an easy way to test/compare several execution engines. I'm thinking about modifying my patch to provide a 'feedbacks-with' keyword so that we can also test/compare different ways to report the execution status and outcome. Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
[parent not found: <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>]
* Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) [not found] ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac> @ 2024-02-28 10:18 ` Bruno Barbier 2024-03-02 10:03 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-02-28 10:18 UTC (permalink / raw) To: emacs-orgmode; +Cc: Ihor Radchenko, Jack Kamm, Matt Hi, Bruno Barbier <brubar.cs@gmail.com> writes: > Hi Matt, [...] >> Since this thread is dedicated to blocking, let me share my thoughts on that subject. > > I guess I should start a new thread then :) I finally got a new version. I've renamed the proposed feature "pending contents", that is, some contents that something will update *later*. With that feature, source code blocks and dynamic blocks can be made asynchronous … and more! I've publish the proposed changes as a branch. You can fecth that branch there: ┌──── │ repo: https://framagit.org/brubar/org-mode-mirror │ branch: bba-pending-contents └──── The file [my-async-tests.org] describes the proposed changes, its implementation and contains examples to play with it (shell, python, ruby, results{append,prepend,silent}, inline blocks, multithreading, dynamic asynchronous blocks, source codes that depend on other blocks {post, stdin}, etc.). I've tried to fully describe the feature in the new section "Pending contents", in the file `lisp/org-macs.el'. Testing it in a bare Emacs should now work (I'm using 30.0.50). *DO NOT TEST* in your production Emacs: this is *alpha* software and it hooks itself deeply into Emacs (kill hooks), and (only when trying the multithread examples) uses thread mutexes&locks. Here is a simple recipe to test the proposed "pending contents" feature: ┌──── │ git clone -b bba-pending-contents https://framagit.org/brubar/org-mode-mirror │ cd org-mode-mirror │ make compile │ cd scratch/bba-pending-contents/ │ emacs -q -L ../../lisp my-async-tests.org └──── Thanks you all for your previous comments. I hope I've addressed most of them. Comments, critiques, ideas, corrections are most welcome. Thanks, Bruno [my-async-tests.org] <https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-02-28 10:18 ` Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) Bruno Barbier @ 2024-03-02 10:03 ` Ihor Radchenko 2024-03-02 10:57 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-03-02 10:03 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: > I've publish the proposed changes as a branch. You can fecth that > branch there: > ┌──── > │ repo: https://framagit.org/brubar/org-mode-mirror > │ branch: bba-pending-contents > └──── Thanks! > I've tried to fully describe the feature in the new section "Pending > contents", in the file `lisp/org-macs.el'. I have one general concern about the implementation. Overlays are not transferred when a new indirect buffer is created (for example, by org-capture, or by user). So, it will be (1) impossible to see pending overlays in indirect buffers; (2) user edits of pending text from indirect buffer will not be prevented. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-02 10:03 ` Ihor Radchenko @ 2024-03-02 10:57 ` Bruno Barbier 2024-03-02 11:13 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-02 10:57 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Hi Ihor, Ihor Radchenko <yantar92@posteo.net> writes: [...] >> I've tried to fully describe the feature in the new section "Pending >> contents", in the file `lisp/org-macs.el'. > > I have one general concern about the implementation. > > Overlays are not transferred when a new indirect buffer is created (for > example, by org-capture, or by user). So, it will be (1) impossible to > see pending overlays in indirect buffers; (2) user edits of pending text > from indirect buffer will not be prevented. If I understand correctly, you would prefer a solution that relies on text properties ? (I didn't leak the overlay details in the API, so, it should not be too hard to switch to text properties, except for possible conflicts with other existing properties). What about markers ? Are they OK ? Thanks, Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-02 10:57 ` Bruno Barbier @ 2024-03-02 11:13 ` Ihor Radchenko 2024-03-02 18:06 ` Bruno Barbier [not found] ` <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b> 0 siblings, 2 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-03-02 11:13 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: >> Overlays are not transferred when a new indirect buffer is created (for >> example, by org-capture, or by user). So, it will be (1) impossible to >> see pending overlays in indirect buffers; (2) user edits of pending text >> from indirect buffer will not be prevented. > > If I understand correctly, you would prefer a solution that relies on > text properties ? (I didn't leak the overlay details in the API, so, it > should not be too hard to switch to text properties, except for possible > conflicts with other existing properties). Yes, I think. Might also try to mirror overlays, but that's problematic in practice. However, you cannot use 'before-string/'after-string for overlays. > What about markers ? Are they OK ? Markers always point to a specific buffer. However, for indirect buffers, the text and text properties are shared. So, editing in one buffer will be automatically reflected in all the clones. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-02 11:13 ` Ihor Radchenko @ 2024-03-02 18:06 ` Bruno Barbier [not found] ` <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b> 1 sibling, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-03-02 18:06 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >>> Overlays are not transferred when a new indirect buffer is created (for >>> example, by org-capture, or by user). So, it will be (1) impossible to >>> see pending overlays in indirect buffers; (2) user edits of pending text >>> from indirect buffer will not be prevented. >> >> If I understand correctly, you would prefer a solution that relies on >> text properties ? (I didn't leak the overlay details in the API, so, it >> should not be too hard to switch to text properties, except for possible >> conflicts with other existing properties). > > Yes, I think. Might also try to mirror overlays, but that's problematic > in practice. I now think that overlays are the right way; the /pending content/ is attached to one buffer: a base or a clone; this is for the user to decide. I will manually add text properties, below the overlay, to mark the text as /pending/, so that pending contents will be visible and read-only in all other buffers, base or indirect ones. Cloning buffers is easy to test. I'm not sure which scenario I should use to test org-capture though. I'll update my branch with that improvement soon. Thanks Ihor! Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
[parent not found: <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b>]
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) [not found] ` <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b> @ 2024-03-07 17:08 ` Bruno Barbier 2024-03-07 18:29 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-07 17:08 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Hi, Bruno Barbier <brubar.cs@gmail.com> writes: > Ihor Radchenko <yantar92@posteo.net> writes: > >> Bruno Barbier <brubar.cs@gmail.com> writes: >> >>>> Overlays are not transferred when a new indirect buffer is created (for >>>> example, by org-capture, or by user). So, it will be (1) impossible to >>>> see pending overlays in indirect buffers; (2) user edits of pending text >>>> from indirect buffer will not be prevented. >>> [...] > I now think that overlays are the right way; the /pending content/ is > attached to one buffer: a base or a clone; this is for the user to > decide. > > I will manually add text properties, below the overlay, to mark the text > as /pending/, so that pending contents will be visible and read-only in > all other buffers, base or indirect ones. > > Cloning buffers is easy to test. I'm not sure which scenario I should > use to test org-capture though. > > I'll update my branch with that improvement soon. > > > Thanks Ihor! > > Bruno Hi, After some work, some bug fixes and a few segfaults (using indirect buffers, see bug#69529), I pushed a new version. The main changes are: • Handle indirect buffers: the pending content belongs to the buffer that started it (using text properties to mirror overlays). • Use the fringe to indicate success or failure. • Describe the pending content (past&present) when the user clicks it (pending, success or failure, time, duration, log, etc.). • Improve the logging API and provide examples (may be used to collect stderr for example). As before, the org file describes how to test it, see the file [scratch/bba-pending-contents/my-async-tests.org] (direct link below). Comments, critiques, ideas, corrections are most welcome. Thanks, Bruno [scratch/bba-pending-contents/my-async-tests.org] <https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-07 17:08 ` Bruno Barbier @ 2024-03-07 18:29 ` Ihor Radchenko 2024-03-08 14:19 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-03-07 18:29 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: >> I now think that overlays are the right way; the /pending content/ is >> attached to one buffer: a base or a clone; this is for the user to >> decide. >> >> I will manually add text properties, below the overlay, to mark the text >> as /pending/, so that pending contents will be visible and read-only in >> all other buffers, base or indirect ones. Thanks! I have some minor concerns about implementation, but you clearly demonstrated the things can be working in general. Let me now zoom out all the way down to org-pending library design. While reading the library header and `org-pending' docstring (btw, it should probably be a separate library, not a part of org-macs), I felt confused about terminology and also had déjà vu's about what org-pending does. More or less, org-pending implements Elisp asynchronous process control, but the processes are associated with region, not the whole buffer. Hence, I feel that we should adopt terminology similar to the existing terminology for processes - process filters, process sentinels, sending signals to processes, etc. And similar API. Also, I am not sure if I like the idea of exposing raw PINFO alist and mutating it. In particular, I have doubts about mutating CONTENT. What we might do instead is implement PINFO as struct with custom accessors/setters. WDYT? -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-07 18:29 ` Ihor Radchenko @ 2024-03-08 14:19 ` Bruno Barbier 2024-03-13 9:48 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-08 14:19 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Hi Ihor, Ihor Radchenko <yantar92@posteo.net> writes: > Thanks! > I have some minor concerns about implementation, but you clearly > demonstrated the things can be working in general. Thanks! > While reading the library header and `org-pending' docstring (btw, it > should probably be a separate library, not a part of org-macs), I was feeling more and more like squatting the wrong file :-) Would "lisp/org-pending.el" be OK ? Or do you see a better place/name ? > I felt confused about terminology and also had déjà vu's about what > org-pending does. > More or less, org-pending implements Elisp asynchronous process control, > but the processes are associated with region, not the whole buffer. > Hence, I feel that we should adopt terminology similar to the existing > terminology for processes - process filters, process sentinels, sending > signals to processes, etc. And similar API. The current `org-pending' doesn't know what a process is, or a thread. The job of org-pending is only to let Emacs know that a given region will be updated at some point later (much like what `org-edit-src-code' does, when a source block is being edited manually). I've re-read the Elisp manual about asynchronous processes. The only thing that seemed obvious to me, was to replace "feedbacks-handler" with "sentinel". I also tried to simplify things and use REGION instead of CONTENT where it make sense. Thanks. Do you see something else ? > Also, I am not sure if I like the idea of exposing raw PINFO alist and > mutating it. In particular, I have doubts about mutating CONTENT. > What we might do instead is implement PINFO as struct with custom > accessors/setters. I went for this simple alist because it's more flexible (easy to add new bindings) and less code than using a struct. And Org already uses lists like this (params, info, etc.). Mutating CONTENT (or other essential data) would be a very bad idea indeed; but it's like that for pretty much everything with elisp. I'm OK to rewrite it using a cl-defstruct (to be honnest, I just flipped a coin to decide between a struct or an alist :) ). I'll push my new version after relocating everything into its own file. Let me know if lisp/org-pending.el is OK. Thanks. Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-08 14:19 ` Bruno Barbier @ 2024-03-13 9:48 ` Ihor Radchenko 2024-03-19 9:33 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-03-13 9:48 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: >> While reading the library header and `org-pending' docstring (btw, it >> should probably be a separate library, not a part of org-macs), > > I was feeling more and more like squatting the wrong file :-) > > Would "lisp/org-pending.el" be OK ? Or do you see a better place/name ? org-pending sounds fine. Maybe org-pending-text to be more explicit. >> More or less, org-pending implements Elisp asynchronous process control, >> but the processes are associated with region, not the whole buffer. >> Hence, I feel that we should adopt terminology similar to the existing >> terminology for processes - process filters, process sentinels, sending >> signals to processes, etc. And similar API. > > The current `org-pending' doesn't know what a process is, or a thread. > The job of org-pending is only to let Emacs know that a given region > will be updated at some point later (much like what > `org-edit-src-code' does, when a source block is being edited > manually). > I've re-read the Elisp manual about asynchronous processes. The only > thing that seemed obvious to me, was to replace "feedbacks-handler" > with "sentinel". I also tried to simplify things and use REGION > instead of CONTENT where it make sense. Thanks. > Do you see something else ? Similar to process API, your library provides means to interact with process associated with the pending text - create them (40.4 Creating an Asynchronous Process), handle feedback (40.9.2 Process Filter Functions), handle on-done/fail/etc (40.10 Sentinels: Detecting Process Status Changes), kill/cancel execution (40.5 Deleting Processes, 40.8 Sending Signals to Processes), insert-log (strerr in Emacs processes) list pending (40.6 Process Information), org-pending-on-kill-buffer (40.11 Querying Before Exit), get information (`process-attributes', 40.12 Accessing Other Processes) I simply went across 40 Processes section of Elisp manual and I see parallels for pretty much every subsection; but the terminology _and API_ are different. >> Also, I am not sure if I like the idea of exposing raw PINFO alist and >> mutating it. In particular, I have doubts about mutating CONTENT. >> What we might do instead is implement PINFO as struct with custom >> accessors/setters. > > I went for this simple alist because it's more flexible (easy to add new > bindings) and less code than using a struct. And Org already uses lists > like this (params, info, etc.). Mutating CONTENT (or other essential > data) would be a very bad idea indeed; but it's like that for pretty > much everything with elisp. > > I'm OK to rewrite it using a cl-defstruct (to be honnest, I just > flipped a coin to decide between a struct or an alist :) ). I think that I need to elaborate. It is not necessarily a matter of alist vs. struct, but more about the API. I do not like the idea of mutating the alist entries directly. I'd rather expose an API to work with "pending" as an opaque object - send signals to it, get status, etc. struct is slightly better here because it automatically defines setters and getters for all the slots without a need to write extra boilerplate code. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-13 9:48 ` Ihor Radchenko @ 2024-03-19 9:33 ` Bruno Barbier 2024-03-20 10:23 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-19 9:33 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: >> Would "lisp/org-pending.el" be OK ? Or do you see a better place/name ? > > org-pending sounds fine. Maybe org-pending-text to be more explicit. I've picked: "org-pending"; it's shorter and can by used as a prefix. Thanks. > Similar to process API, your library provides means to interact with > process associated with the pending text - create them (40.4 Creating an > Asynchronous Process), handle feedback (40.9.2 Process Filter > Functions), handle on-done/fail/etc (40.10 Sentinels: Detecting Process > Status Changes), kill/cancel execution (40.5 Deleting Processes, 40.8 > Sending Signals to Processes), insert-log (strerr in Emacs processes) > list pending (40.6 Process Information), org-pending-on-kill-buffer > (40.11 Querying Before Exit), get information (`process-attributes', > 40.12 Accessing Other Processes) > > I simply went across 40 Processes section of Elisp manual and I see > parallels for pretty much every subsection; but the terminology _and > API_ are different. > I rewrote the API, rename many things, moved the code around and sorted everything into heading/subheading sections. This is hopefully less confusing and a lot simpler; and the documentation hopefully explains better how to use it. The updated section "Commentary", in org-pending, describes the 3 steps that are needed to mark and use a pending region: a PENREG (I've renamed "PINFO" to "PENREG", for PENding REGion, more specific). I tried to stick with the process terminology when relevant (mostly kill-query and sentinel). It doesn't really match the process API, but, now, it should be easier to understand why. >> I'm OK to rewrite it using a cl-defstruct (to be honnest, I just >> flipped a coin to decide between a struct or an alist :) ). > > I think that I need to elaborate. > It is not necessarily a matter of alist vs. struct, but more about the > API. I do not like the idea of mutating the alist entries directly. I'd > rather expose an API to work with "pending" as an opaque object - send > signals to it, get status, etc. > > struct is slightly better here because it automatically defines setters > and getters for all the slots without a need to write extra boilerplate > code. Done. Thanks! I added a function `org-pending-user-edit': it allows to edit a region using org-pending and `string-edit' (see org-pending.el). I also added an other execution engine (DEMO_ONLY), that allows to execute any elisp asynchronously, using callbacks (see `my-use-callbacks-schedule', in my-async-tests.el), and in my-async-tests.org, near the multithreading examples. To test, follow the org file [my-async-tests.org] (see direct link below). WDYT of this version ? Thanks! Bruno [I've forced push the new version, squashing everything into simple commits, if you need the old version, just ask and I'll push a copy there.] [my-async-tests.org]: https://framagit.org/brubar/org-mode-mirror/-/blob/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org?ref_type=heads [repo & branch]: https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents?ref_type=heads bba-pending-contents ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-19 9:33 ` Bruno Barbier @ 2024-03-20 10:23 ` Ihor Radchenko 2024-03-21 10:06 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-03-20 10:23 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: > I rewrote the API, rename many things, moved the code around and > sorted everything into heading/subheading sections. This is hopefully > less confusing and a lot simpler; and the documentation hopefully > explains better how to use it. Thanks! It does look more clear. > The updated section "Commentary", in org-pending, describes the 3 steps > that are needed to mark and use a pending region: a PENREG (I've renamed > "PINFO" to "PENREG", for PENding REGion, more specific). I feel that org-pending-penreg (org-pending-<pending region>) is redundant. Maybe better use org-pending-region? > WDYT of this version ? Comments on org-pending-pendreg struct: 1. It is not clear why you need a separate ~virtual~ field. When ~region~ is nil it already implies that the pending region is virtual. 2. ~id~ field is semi-internal and assumed to be a number. Maybe we can do something more similar to Emacs process API: (make-process &rest ARGS) ... :name NAME -- NAME is name for process. It is modified if necessary to make it unique. We can replace id with human-readable name that can also be supplied when creating a pending region 3. ~source~ field must match the ~region~ marker buffer. Then, why do we need this field at all? May as well just use (marker-buffer (car region)) On the design of ~org-pending~ and ~org-pending-task-connect~: 1. I feel confused about the overall design of the interaction between pending region and the associated task. Functions like ~org-pending-task-send-update~ imply that pending region is rather decoupled from from associated task and the task should arrange manually for sending updates to the pending region object. On the other hand, there is ~task-connection~ that is used to kill associated task/process or to "await" for it. This time, pending region is strongly coupled with the task, killing it upon deleting the pending region. I think that we need more (optional) coupling between pending region and the associated task. We should be able to get more information about the task from pending region side - get logs, current status, exit status, etc. More specifically, I think that we need (1) allow to pass task as an argument for ~org-pending~. (2) In ~org-pending-task-connect~, we should allow the task to be a process or timer object. Then, we can automatically arrange retrieving process/timer status from the task: Use process sentinel (maybe, modifying existing via ~add-function~) to arrange process status changes to be automatically submitted to the pending region; Get log updates via process filter Kill process via ~kill-process~ Similar for timers. If the argument to ~org-pending-task-connect~ is a lambda, we can use the current approach you implemented on the branch. 2. ~org-pending-task-send-update~ name is confusing - it reads as if we send an update _to_ the task. Maybe better ~org-pending-region-update~? Then, we might even drop ~-sentinel~ field in org-pending-penreg object and instead implement that hard-coded ~update~ lambda from ~org-pending~ as a part of ~org-pending-region-update~. 3. I feel that different handling of "owner" and indirect buffers is not right. From the user perspective, it does not matter whether we run an src block from one or another indirect buffers - it makes sense to see the status in all the indirect org-mode buffers. Maybe we can hook into org-mode's fontification and display pending overlays in all the indirect buffers. Further, it is very confusing that running src block twice from the same buffer is not the same as running the same src block from one buffer and then from another indirect buffer. The current implementation of ~remove-previous-overlays~ makes such distinction for reasons I do not understand. 4. I have questions about ~handle-result~ argument in ~org-pending~. It is only called on success, but I can easily see that we need to handle things specially on failure as well. For example, insert stderr or perform other actions like displaying some buffer. Or we may even hook some special action to clicking on status overlay. For example, clicking on "failure" status overlay may raise stderr log. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-20 10:23 ` Ihor Radchenko @ 2024-03-21 10:06 ` Bruno Barbier 2024-03-21 12:15 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-21 10:06 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Ihor Radchenko <yantar92@posteo.net> writes: Thanks for your review Ihor! > Bruno Barbier <brubar.cs@gmail.com> writes: > >> I rewrote the API, rename many things, moved the code around and >> sorted everything into heading/subheading sections. This is hopefully >> less confusing and a lot simpler; and the documentation hopefully >> explains better how to use it. > > Thanks! It does look more clear. > >> The updated section "Commentary", in org-pending, describes the 3 steps >> that are needed to mark and use a pending region: a PENREG (I've renamed >> "PINFO" to "PENREG", for PENding REGion, more specific). > > I feel that org-pending-penreg (org-pending-<pending region>) is > redundant. Maybe better use org-pending-region? PENREG is the name of the structure; the "org-pending" is the mandatory library prefix; this mechanically gives the name. A PENDREG object is not a "region" in Emacs sense. Do you see a better name for the structure PENREG, so that it doesn't sound like a "region" ? >> WDYT of this version ? > > Comments on org-pending-pendreg struct: > > 1. It is not clear why you need a separate ~virtual~ field. When > ~region~ is nil it already implies that the pending region is > virtual. It's a constant. Calling a function looks more like we need to recompute it each time, and, we could even change it. And cl-defstruct writes the function for us. Do you prefer a manually written function ? > 2. ~id~ field is semi-internal and assumed to be a number. > Maybe we can do something more similar to Emacs process API: > > (make-process &rest ARGS) > ... > :name NAME -- NAME is name for process. It is modified if necessary > to make it unique. > > We can replace id with human-readable name that can also be supplied > when creating a pending region Good idea. Done. > 3. ~source~ field must match the ~region~ marker buffer. Then, why do we > need this field at all? May as well just use (marker-buffer (car region)) The "source" is the region requesting the update. The pending region is the "target" of the update, i.e. the region that will be updated. For example, in DEMO_ONLY, with org-babel, these 2 regions are never the same: 1. the source is the source code block, 2. the target (pending region) is the result region. I tried to improve the documentation of `org-pending-penreg' (source & region). > On the design of ~org-pending~ and ~org-pending-task-connect~: > > 1. I feel confused about the overall design of the interaction between > pending region and the associated task. > > Functions like ~org-pending-task-send-update~ imply that pending > region is rather decoupled from from associated task and the task > should arrange manually for sending updates to the pending region > object. Exactly: the task implementation must use these ~org-pending-task-XXX~ functions to send updates to one (or more) pending region(s). > On the other hand, there is ~task-connection~ that is used to kill > associated task/process or to "await" for it. This time, pending > region is strongly coupled with the task, killing it upon deleting > the pending region. These are optional features; and only ~org-pending~ will know if and when those might be useful. That's why the task needs to provide callbacks here. 1. cancel: Emacs may, in exceptional cases only, send a "cancel" to the task, meaning, "The user destroyed the pending region, and thus, Emacs will not use any update for it". 2. insert-details: If, and only if, the user decides to investigate what happened, Emacs will ask the task if it has any details to add, that might help the user (like exit-code for an OS process, stderr for an OS process or link to a log file, etc.) 3. get (await): It's an (unofficial) way, (in the degenerate case where the task implementation gives up on asynchronicity) to block until the outcome is available. `org-pending' itself doesn't use it; DEMO_ONLY uses it with org-babel to define the synchronous executions. > I think that we need more (optional) coupling between pending region > and the associated task. We should be able to get more information > about the task from pending region side - get logs, current status, > exit status, etc. > More specifically, I think that we need (1) allow to pass task as an > argument for ~org-pending~. That's actually what I started with, but, I couldn't make it work. Breaking it like this is what allowed me to get the most generic and simplest API that works for anything: threads, callbacks, OS processes, timers. If org-pending takes a "task" as an argument, then, we have to define a universal API for whatever a "task" might be: threads, processes, callbacks, timers, etc. and any combination of them. It looks simpler to say that the "task" (whatever "task" means), MAY call: - org-pending-task-send-update (:progress xxx1) - org-pending-task-send-update (:progress xxx2) - org-pending-task-send-update (:progress xxx3) then, finally MUST either call: - org-pending-task-send-update (:success RESULT) or: org-pending-task-send-update (:failure RESULT) > (2) In ~org-pending-task-connect~, we > should allow the task to be a process or timer object. Then, we can > automatically arrange retrieving process/timer status from the task: > Use process sentinel (maybe, modifying existing via ~add-function~) > to arrange process status changes to be automatically submitted to > the pending region; > > Get log updates via process filter > > Kill process via ~kill-process~ It looks to me like a very specific case (one OS process for one pending region); and I'm not sure how useful it would be in practice. But this could easily be implemented on top of the existing API. > Similar for timers. Same, it could easily be defined on top of the existing API. > If the argument to ~org-pending-task-connect~ is a lambda, we can use > the current approach you implemented on the branch. > 2. ~org-pending-task-send-update~ name is confusing - it reads as if we > send an update _to_ the task. Maybe better ~org-pending-region-update~? Yes ... I wanted a common prefix for the 3 functions that a "task" implementation is allowed to use: - org-pending-task-connect, - org-pending-task-send-update, - org-pending-task-not-implemented. It's not confusing if one ignores the common prefix :-) I've renamed all these functions from "org-pending-task-" to "org-pending-ti-" where "ti" stands for "task implementation". > Then, we might even drop ~-sentinel~ field in org-pending-penreg > object and instead implement that hard-coded ~update~ lambda from > ~org-pending~ as a part of ~org-pending-region-update~. That would require to manually capture (dump/load) the context that the sentinel closure is automatically capturing. Why would it be better ? Debugging purposes ? In this case, the current context is (currently) very small, and there is an obvious place where to dump/load, so, just tell me if you want me to eliminate that closure. > 3. I feel that different handling of "owner" and indirect buffers is not > right. > From the user perspective, it does not matter whether we run an src > block from one or another indirect buffers - it makes sense to see > the status in all the indirect org-mode buffers. I just tried to followe Emacs: a buffer owns its overlays; a pending region is (kind of) an overlay. Thus, a buffer owns its pending region. > Maybe we can hook into org-mode's fontification and display > pending overlays in all the indirect buffers. Well ... "adding overlays in indirect buffers using font-lock" looks like a very bumpy road to me ... (being very positive, assuming there is even a road there ... :-) ). As jit-lock is explicitly disabled in indirect buffers, I'm not even sure what it would technically mean. > Further, it is very confusing that running src block twice from the > same buffer is not the same as running the same src block from one > buffer and then from another indirect buffer. The current > implementation of ~remove-previous-overlays~ makes such distinction > for reasons I do not understand. Technically, the outcomes are overlays too; thus, they belong to one buffer. If a user created an indirect buffer to focus on some source blocks, he should expect to manage everything about them from that buffer. ... that looks to me like a plausible explanation that matches the technical limitations :-) We might be able to add some other workarounds for indirect buffers, to provide seamless switches between buffers. But would that really be worth it? In summary, about indirect buffers, I'm not sure it's a good idea to try to handle them. I didn't do much with them, but, I'll already was able to segfault Emacs. I would prefer to put the no-clone-indirect property to the org-mode personally :-) Couldn't we just to forbid "pending regions" in indirect buffers ? (pending regions don't exist today, so, that doesn't look that bad, at least for now) > 4. I have questions about ~handle-result~ argument in ~org-pending~. > It is only called on success, Yes. It means Org needs to handle the exact same result as in the synchronous case, and, it must handle it in exactly the same way. That's a design choice. That's probably why it was easy to write the org-babel examples in DEMO_ONLY; and they already handle most of the standard Org parameters: async & sync, session & no session, append/prepend/post/stdin/var, etc. > but I can easily see that we need to > handle things specially on failure as well. For example, insert > stderr or perform other actions like displaying some buffer. Or we > may even hook some special action to clicking on status overlay. For > example, clicking on "failure" status overlay may raise stderr log. It's already there, no? If you click on any result (success or failure, inline block or not, even dynamic blocks), Emacs pops up a buffer with all the details (source, start, end, duration, stderr, etc.). The function `org-pending-describe-penreg' defines what is inserted. A given task is free to insert log, links, widgets, images, diffs, etc. (by providing the relevant :insert-details method). These are design choices that are relevant here: 1. Do not differ from the synchronous case. 2. Do not delete a valid result until you know that you have something better (where a progress or a failure is not "better"). 3. Do not interrupt the user with popups. If, in the synchronous case, org-babel writes some logs, then the task must report a success to org-pending so that Org just behaves the same. If the task reports a failure, the user keeps the previous content. We could modify how ob-core uses org-pending, adding some options if some users would like errors to be written down using some Org syntax. I've pushed my update to the public repo (sorry, forced push again due to some mistakes). Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-21 10:06 ` Bruno Barbier @ 2024-03-21 12:15 ` Ihor Radchenko 2024-03-25 17:46 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-03-21 12:15 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: >> I feel that org-pending-penreg (org-pending-<pending region>) is >> redundant. Maybe better use org-pending-region? > > PENREG is the name of the structure; the "org-pending" is the > mandatory library prefix; this mechanically gives the name. A PENDREG > object is not a "region" in Emacs sense. > > Do you see a better name for the structure PENREG, so that it doesn't > sound like a "region" ? Library prefix is also a part of the name and delivers useful information. "org-pending-region" and "region" and not the same names. We make use of prefix semantic in various places: - org-export-backend, implying not just "backend", but also "export" - org-cite-processor, implying not just "processor", but also "cite" - org-lint-checker - "org-lint" + "checker" - org-element-deferred - "org-element" + "deferred" So, there is no need to duplicate information from the prefix - it is an integral part of the struct name. Doing otherwise would go again the existing naming in Org code base. >> 1. It is not clear why you need a separate ~virtual~ field. When >> ~region~ is nil it already implies that the pending region is >> virtual. > > It's a constant. Calling a function looks more like we need to > recompute it each time, and, we could even change it. And > cl-defstruct writes the function for us. > > Do you prefer a manually written function ? Either a function or a clear indication in the docstring that ~virtual~ and ~region~ are connected and both read-only. Also, ~virtual~ field is unused. So, maybe we can even drop it completely. We can always add new fields in future, if a need arises. >> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we >> need this field at all? May as well just use (marker-buffer (car region)) > > The "source" is the region requesting the update. The docstring of `org-pending' states that it is a buffer position: The SOURCE is the buffer position that requested this pending region. > ... The pending region > is the "target" of the update, i.e. the region that will be updated. > > > For example, in DEMO_ONLY, with org-babel, these 2 regions are never > the same: > 1. the source is the source code block, > 2. the target (pending region) is the result region. I am wondering why source must be a buffer position. What if we want to mark a region pending for some task not associated with a source? And why do we need to know the source at all? > 2. insert-details: If, and only if, the user decides to > investigate what happened, Emacs will ask the task if it has any > details to add, that might help the user (like exit-code for an > OS process, stderr for an OS process or link to a log file, etc.) I have to say that I am confused about "insert-details" part. Mostly because it is not per se connected to the associated task. It is rather an additional handler used to provide debug information about the task status and outcome. AFAIU, it is conceptually very similar to HANDLE-RESULT function. I think that rather than handing HANDLE-RESULT and also TASK-CONTROL, we may reduce everything to a single "handler" object that will serve as a way for PENREG to communicate back to Elisp. That way, we do not need to have a concept of a "task". Instead, it will be a familiar async API with ability to (1) create (2) send signals to (3) receive signals from PENREG object. `org-pending' will be the entry point to create PENREG object. `org-pending-ti-send-update' (or maybe simply `org-pending-send-update'?) will be a way to send data to PENREG object. HANLDER will be another object we may expose via something like (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...) Then, PENREG will call appropriate handler function as needed. >> If the argument to ~org-pending-task-connect~ is a lambda, we can use >> the current approach you implemented on the branch. > >> 2. ~org-pending-task-send-update~ name is confusing - it reads as if we >> send an update _to_ the task. Maybe better ~org-pending-region-update~? > > Yes ... I wanted a common prefix for the 3 functions that a "task" > implementation is allowed to use: > - org-pending-task-connect, > - org-pending-task-send-update, > - org-pending-task-not-implemented. > > It's not confusing if one ignores the common prefix :-) > > I've renamed all these functions from "org-pending-task-" to > "org-pending-ti-" where "ti" stands for "task implementation". I still feel confused. As stated above, it might be a good idea to get rid of the concept of "task" completely. >> Then, we might even drop ~-sentinel~ field in org-pending-penreg >> object and instead implement that hard-coded ~update~ lambda from >> ~org-pending~ as a part of ~org-pending-region-update~. > > That would require to manually capture (dump/load) the context that > the sentinel closure is automatically capturing. > > Why would it be better ? Debugging purposes ? Yes. Lexical context is implicit and harder to debug, while storing necessary data explicitly in the struct slots is more robust as we are very clear which context is intended to be captured. >> 3. I feel that different handling of "owner" and indirect buffers is not >> right. >> From the user perspective, it does not matter whether we run an src >> block from one or another indirect buffers - it makes sense to see >> the status in all the indirect org-mode buffers. > > I just tried to followe Emacs: a buffer owns its overlays; a pending > region is (kind of) an overlay. Thus, a buffer owns its pending > region. I do not think that it is a good analogy. Not when we also mark the text read-only in all the indirect buffers as well. Let me state my idea differently - if some text in buffer is "pending", it should be visible in all indirect buffers. Otherwise, as a user, I may be confused why some parts of the buffer are read-only. >> Maybe we can hook into org-mode's fontification and display >> pending overlays in all the indirect buffers. > > Well ... "adding overlays in indirect buffers using font-lock" looks > like a very bumpy road to me ... (being very positive, assuming there > is even a road there ... :-) ). As jit-lock is explicitly disabled in > indirect buffers, I'm not even sure what it would technically mean. Right. font-lock is linked to text properties, so font-lock in indirect buffers is finicky (it relies upon text properties to function). We might try to brew something with `pre-redisplay-functions', but let's not dive into that rabbit hole for now. >> Further, it is very confusing that running src block twice from the >> same buffer is not the same as running the same src block from one >> buffer and then from another indirect buffer. The current >> implementation of ~remove-previous-overlays~ makes such distinction >> for reasons I do not understand. > > Technically, the outcomes are overlays too; thus, they belong to > one buffer. > > If a user created an indirect buffer to focus on some source blocks, > he should expect to manage everything about them from that buffer. > ... that looks to me like a plausible explanation that matches the > technical limitations :-) This sounds like trying to fit (by force) expectations to technical limitations. As a user, I do not really manage everything from the same buffer and do not expect that Org mode expects me to do so :) But let's postpone indirect buffer discussion to later and focus on more high-level design first. > ... would prefer to put the no-clone-indirect > property to the org-mode personally :-) We cannot do it. People use indirect buffers with Org mode extensively. > Couldn't we just to forbid "pending regions" in indirect buffers ? > (pending regions don't exist today, so, that doesn't look that bad, at > least for now) This might be ok. But we should be prepared that "read-only" on the pending regions is not going to be reliable - the regions marked pending can be changed out of sight. >> but I can easily see that we need to >> handle things specially on failure as well. For example, insert >> stderr or perform other actions like displaying some buffer. Or we >> may even hook some special action to clicking on status overlay. For >> example, clicking on "failure" status overlay may raise stderr log. > > It's already there, no? > > If you click on any result (success or failure, inline block or not, > even dynamic blocks), Emacs pops up a buffer with all the details > (source, start, end, duration, stderr, etc.). The function > `org-pending-describe-penreg' defines what is inserted. A given task is > free to insert log, links, widgets, images, diffs, etc. (by providing > the relevant :insert-details method). Your thinking also makes sense, if I use a different definition of "failure" (in the context of PENREG, not in the context of exit code of the attached process) -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-21 12:15 ` Ihor Radchenko @ 2024-03-25 17:46 ` Bruno Barbier 2024-03-27 11:29 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-03-25 17:46 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Hi Ihor, Thanks for your review and detailed explanations. I've pushed an update that should address most of your comments. Let me answer point by point below. Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >>> I feel that org-pending-penreg (org-pending-<pending region>) is >>> redundant. Maybe better use org-pending-region? >> >> PENREG is the name of the structure; the "org-pending" is the >> mandatory library prefix; this mechanically gives the name. A PENDREG >> object is not a "region" in Emacs sense. >> >> Do you see a better name for the structure PENREG, so that it doesn't >> sound like a "region" ? > > Library prefix is also a part of the name and delivers useful > information. "org-pending-region" and "region" and not the same names. > > We make use of prefix semantic in various places: > - org-export-backend, implying not just "backend", but also "export" > - org-cite-processor, implying not just "processor", but also "cite" > - org-lint-checker - "org-lint" + "checker" > - org-element-deferred - "org-element" + "deferred" > > So, there is no need to duplicate information from the prefix - it is an > integral part of the struct name. Doing otherwise would go again the > existing naming in Org code base. Got it. Thanks for the explanation. I've found a better name: I'm now calling it a "lock". So I renamed "PENREG" into "REGLOCK" as "region lock". The structure is now `org-pending-reglock'. >>> 1. It is not clear why you need a separate ~virtual~ field. When >>> ~region~ is nil it already implies that the pending region is >>> virtual. >> [...] > Also, ~virtual~ field is unused. So, maybe we can even drop it > completely. We can always add new fields in future, if a need arises. The region is not allowed to be nil anymore. All "virtual" issues are solved! :) >>> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we >>> need this field at all? May as well just use (marker-buffer (car region)) >> >> The "source" is the region requesting the update. > > The docstring of `org-pending' states that it is a buffer position: > > The SOURCE is the buffer position that requested this pending region. Sorry. It was, yes. >> ... The pending region >> is the "target" of the update, i.e. the region that will be updated. >> >> >> For example, in DEMO_ONLY, with org-babel, these 2 regions are never >> the same: >> 1. the source is the source code block, >> 2. the target (pending region) is the result region. > > I am wondering why source must be a buffer position. > What if we want to mark a region pending for some task not associated > with a source? And why do we need to know the source at all? Good point. It was to allow the user to quickly jump to the source code block; I removed it from org-pending. >> 2. insert-details: If, and only if, the user decides to >> investigate what happened, Emacs will ask the task if it has any >> details to add, that might help the user (like exit-code for an >> OS process, stderr for an OS process or link to a log file, etc.) > > I have to say that I am confused about "insert-details" part. Mostly > because it is not per se connected to the associated task. It is rather > an additional handler used to provide debug information about the task > status and outcome. > AFAIU, it is conceptually very similar to HANDLE-RESULT function. You're right: it was confusing. It's now like a hook, that org-pending-describe-reglock will use. It should now be clearer that it's for "information purposes" only. > I think that rather than handing HANDLE-RESULT and also TASK-CONTROL, we > may reduce everything to a single "handler" object that will serve as a > way for PENREG to communicate back to Elisp. That way, we do not need to > have a concept of a "task". I removed the task-control, and, the concept of "task". HANDLE-RESULT is gone from org-pending. `org-pending' has a new keyword: :on-outcome that will allow to do anything, both on success and on failure. > Instead, it will be a familiar async API > with ability to (1) create (2) send signals to (3) receive signals from > PENREG object. > `org-pending' will be the entry point to create PENREG object. > > `org-pending-ti-send-update' (or maybe simply > `org-pending-send-update'?) will be a way to send data to PENREG object. > I've renamed org-pending-ti-send-update to org-pending-send-update (now that the task control is gone, the prefix becomes useless). > HANLDER will be another object we may expose via something like > (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...) > Then, PENREG will call appropriate handler function as needed. As the task-control is now gone: - get/await is gone, - cancel is now a hook/function of REGLOCK, - insert-details is now a hook/function too of REGLOCK. >>> If the argument to ~org-pending-task-connect~ is a lambda, we can use >>> the current approach you implemented on the branch. >> >>> 2. ~org-pending-task-send-update~ name is confusing - it reads as if we >>> send an update _to_ the task. Maybe better ~org-pending-region-update~? >> >> Yes ... I wanted a common prefix for the 3 functions that a "task" >> implementation is allowed to use: >> - org-pending-task-connect, >> - org-pending-task-send-update, >> - org-pending-task-not-implemented. >> >> It's not confusing if one ignores the common prefix :-) >> >> I've renamed all these functions from "org-pending-task-" to >> "org-pending-ti-" where "ti" stands for "task implementation". > > I still feel confused. As stated above, it might be a good idea to get > rid of the concept of "task" completely. Done. >>> Then, we might even drop ~-sentinel~ field in org-pending-penreg >>> object and instead implement that hard-coded ~update~ lambda from >>> ~org-pending~ as a part of ~org-pending-region-update~. >> >> That would require to manually capture (dump/load) the context that >> the sentinel closure is automatically capturing. >> >> Why would it be better ? Debugging purposes ? > > Yes. Lexical context is implicit and harder to debug, while storing > necessary data explicitly in the struct slots is more robust as we are > very clear which context is intended to be captured. Done. >>> 3. I feel that different handling of " [[]] owner" and indirect buffers is not >>> right. >>> From the user perspective, it does not matter whether we run an src >>> block from one or another indirect buffers - it makes sense to see >>> the status in all the indirect org-mode buffers. >> >> I just tried to followe Emacs: a buffer owns its overlays; a pending >> region is (kind of) an overlay. Thus, a buffer owns its pending >> region. > > I do not think that it is a good analogy. > Not when we also mark the text read-only in all the indirect buffers > as well. > Let me state my idea differently - if some text in buffer is "pending", > it should be visible in all indirect buffers. Otherwise, as a user, I > may be confused why some parts of the buffer are read-only. You're describing the current implementation ... at least, what I tried to do :) With the current implementation, they should be clearly visible in all buffers, using the secondary-selection face. If they are not, it's because Org is explicitly removing some text properties. Oh, I see ... sorry: don't test indirect buffers with empty results, as Org is almost always removing custom faces on "#+RESULT:" (IIUC, Org doesn't comply with font-lock-face). [...] > But let's postpone indirect buffer discussion to later and focus on more > high-level design first. > ok. Thanks. [...] >>> but I can easily see that we need to >>> handle things specially on failure as well. For example, insert >>> stderr or perform other actions like displaying some buffer. Or we >>> may even hook some special action to clicking on status overlay. For >>> example, clicking on "failure" status overlay may raise stderr log. >> >> It's already there, no? >> >> If you click on any result (success or failure, inline block or not, >> even dynamic blocks), Emacs pops up a buffer with all the details >> (source, start, end, duration, stderr, etc.). The function >> `org-pending-describe-penreg' defines what is inserted. A given task is >> free to insert log, links, widgets, images, diffs, etc. (by providing >> the relevant :insert-details method). > > Your thinking also makes sense, if I use a different definition of > "failure" (in the context of PENREG, not in the context of exit code of > the attached process) org-pending now takes an :on-outcome callback: we can now decide what to do both on success and on failure, and insert things in the org document in all cases if needed. Let me know what you think about this new version, Thanks again for your help! Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-25 17:46 ` Bruno Barbier @ 2024-03-27 11:29 ` Ihor Radchenko 2024-03-30 22:53 ` Rudolf Adamkovič 2024-04-04 16:33 ` Bruno Barbier 0 siblings, 2 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-03-27 11:29 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: > I've pushed an update that should address most of your comments. Thanks! > I've found a better name: I'm now calling it a "lock". So I renamed > "PENREG" into "REGLOCK" as "region lock". The structure is now > `org-pending-reglock'. I slightly dislike short names and would prefer region-lock, but not a big deal - it is just my personal style preference. > I've renamed org-pending-ti-send-update to org-pending-send-update > (now that the task control is gone, the prefix becomes useless). I have a further request on interaction with penreg objects. I feel that it is not ideal that overlays associated with penreg objects cannot be fully controlled by the callers. In particular, I'd like to see some way to 1. Create penreg object without locking the region, so that scheduled-at time is not set immediately and status overlay is not displayed. Then, `org-pending-send-update' could send :schedule signal to perform actual lock. 2. Act on the outcome overlays - there is currently no way to remove them using penreg object. Maybe :cancel signal? Canceled penreg objects can then be garbage-collected from the manager. Also, the top-level commentary is getting incomplete and out-of-sync at this point. May you work towards more detailed top-level description of the library? This will make your ideas more explicit and make life easier for me during the further review (now, I have to guess what you meant by some parts of the code). >> HANLDER will be another object we may expose via something like >> (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...) >> Then, PENREG will call appropriate handler function as needed. > > As the task-control is now gone: > - get/await is gone, > - cancel is now a hook/function of REGLOCK, > - insert-details is now a hook/function too of REGLOCK. >> ... >> Yes. Lexical context is implicit and harder to debug, while storing >> necessary data explicitly in the struct slots is more robust as we are >> very clear which context is intended to be captured. > > Done. Is there any reason why you hide the extra information behind :-alist filed? Why not directly adding extra fields with proper documentation? -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-27 11:29 ` Ihor Radchenko @ 2024-03-30 22:53 ` Rudolf Adamkovič 2024-04-04 16:35 ` Bruno Barbier 2024-04-04 16:33 ` Bruno Barbier 1 sibling, 1 reply; 55+ messages in thread From: Rudolf Adamkovič @ 2024-03-30 22:53 UTC (permalink / raw) To: Ihor Radchenko, Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Ihor Radchenko <yantar92@posteo.net> writes: >> I've found a better name: I'm now calling it a "lock". So I renamed >> "PENREG" into "REGLOCK" as "region lock". The structure is now >> `org-pending-reglock'. > > I slightly dislike short names and would prefer region-lock, but not a > big deal - it is just my personal style preference. +1 for the full name. Searching 'M-x' for 'region' gives 229 results on my Emacs, so there is a precedent. In fact, when I first read the name 'reglock', I took 'reg' for *not* a region, but register or registry, precisely because Emacs consistently spells the word out in full. Rudy -- "I love deadlines. I love the whooshing noise they make as they go by." --- Douglas Adams, The Salmon of Doubt, 2002 Rudolf Adamkovič <rudolf@adamkovic.org> [he/him] Studenohorská 25, 84103 Bratislava, Slovakia, European Union ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-30 22:53 ` Rudolf Adamkovič @ 2024-04-04 16:35 ` Bruno Barbier 0 siblings, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-04-04 16:35 UTC (permalink / raw) To: Rudolf Adamkovič, Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Rudolf Adamkovič <rudolf@adamkovic.org> writes: > > +1 for the full name. > > Searching 'M-x' for 'region' gives 229 results on my Emacs, so there is > a precedent. In fact, when I first read the name 'reglock', I took > 'reg' for *not* a region, but register or registry, precisely because > Emacs consistently spells the word out in full. > Thanks for your input. Note that "reglock" is the current technical name, an implementation detail, and for developer only. The only current command using that term (that shows up using M-x) is a debugging internal command (org-pending--describe-reglock-at-point). I'm thinking about using just "lock" but I'll keep using "reglock" for now, as it's easier to search/replace, until the public API is finalized. Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-03-27 11:29 ` Ihor Radchenko 2024-03-30 22:53 ` Rudolf Adamkovič @ 2024-04-04 16:33 ` Bruno Barbier 2024-04-11 11:44 ` Ihor Radchenko 1 sibling, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-04-04 16:33 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> I've pushed an update that should address most of your comments. > > Thanks! > >> I've found a better name: I'm now calling it a "lock". So I renamed >> "PENREG" into "REGLOCK" as "region lock". The structure is now >> `org-pending-reglock'. > > I slightly dislike short names and would prefer region-lock, but not a > big deal - it is just my personal style preference. It's more about being one word than being short, and being a new and opaque word. REGLOCK is a new technical term; it's definition is the cl-defstruct. For normal use, this is just a black box to pass around. I'll probably switch to using just "lock". >> I've renamed org-pending-ti-send-update to org-pending-send-update >> (now that the task control is gone, the prefix becomes useless). > > I have a further request on interaction with penreg objects. > I feel that it is not ideal that overlays associated with penreg objects > cannot be fully controlled by the callers. I'm trying to limit the public API surface. I don't think we should leak that we are currently using a mix of overlays and text properties. > In particular, I'd like to see some way to > > 1. Create penreg object without locking the region, so that scheduled-at > time is not set immediately and status overlay is not displayed. > Then, `org-pending-send-update' could send :schedule signal to > perform actual lock. Using the term "region" was confusing, sorry. That's why I switched to region "lock". I don't think there is a use to create a lock that doesn't lock. Also, that might be tricky to implement: `org-pending-send-update' is called asynchronously, from the user point of view. Having regions that suddenly become locked, independently of what the user is currently doing (if we implement the :schedule message), might be difficult. What use do you have in mind ? > 2. Act on the outcome overlays - there is currently no way to remove > them using penreg object. I've added a funcion `org-pending-delete-outcome-marks' to manually delete outcome marks that are in a given region. Else, everything is handled automatically. Once the outcome is known, the reglock is dead (not live-p). org-pending may leave outcome marks about the outcomes (outcome marks are optional). The outcome marks automatically disappear if the user remove the section, or, if a new lock is created for the same region. > Maybe :cancel signal? Canceled penreg > objects can then be garbage-collected from the manager. Cancel is handled by sending a failure message (see `org-pending-cancel'). It's customizable using the reglock field ~org-pending-reglock-user-cancel-function~, which can decide what to do (like kill a process) and which can send a better outcome. Standard 'cancel' leaves a failure outcome mark. I've added garbage collections of useless reglocks (success or failure): see `org-pending--mgr-garbage-collect'. > Also, the top-level commentary is getting incomplete and out-of-sync at > this point. May you work towards more detailed top-level description of > the library? Sorry. I tried hard to keep it in sync with all the modifications. I found it too much work, and, possibly overwhelming for the reader, to explain everything in the top-level "Commentary" section. That's why I deleted everything that wasn't mandatory to understand the core features. Everything should be documented as elisp documentation strings, following the documentation of `org-pending' and `org-pending-send-update', and, from code comments. > May you work towards more detailed top-level description of > the library? > This will make your ideas more explicit and make life > easier for me during the further review (now, I have to guess what you > meant by some parts of the code). Sorry, and thank you again for your time. I tried to improve the overall documentation. I hope it's going to be easier for you, and others. >>> HANLDER will be another object we may expose via something like >>> (org-pending-handler (&key on-success-function on-cancel-function on-await on-insert-logs) ...) >>> Then, PENREG will call appropriate handler function as needed. >> >> As the task-control is now gone: >> - get/await is gone, >> - cancel is now a hook/function of REGLOCK, >> - insert-details is now a hook/function too of REGLOCK. >>> ... >>> Yes. Lexical context is implicit and harder to debug, while storing >>> necessary data explicitly in the struct slots is more robust as we are >>> very clear which context is intended to be captured. >> >> Done. > > Is there any reason why you hide the extra information behind :-alist > filed? Why not directly adding extra fields with proper documentation? To hide them, indeed :) The API for 'get-status and 'get-live-p are `org-pending-reglock-status' and `org-pending-reglock-live-p' (they are read-only). The API for the new `useless-p' is `org-pending-reglock-useless-p' (it's read-only too). The fields anchor-ovl, region-ovl, on-outcome, set-status and creation-point are the dump of the closure context, so that org-pending doesn't rely anymore on a closure to handle updates; I've rewritten that recently. Nobody is supposed to use or change those values, except the update process. IMHO, dumping those as fields in the lock structure would be more confusing and fragile than keeping those out of sight. I could add comments when they are created/used in the code to help understand how they are used. I've added a new macro `org-pending-updating-region' that locks a region while executing its body. I've pushed a new version. The documentation in org-pending should now be in sync, sorry. Let me know what you think, Thanks, Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-04-04 16:33 ` Bruno Barbier @ 2024-04-11 11:44 ` Ihor Radchenko 2024-04-19 11:23 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-04-11 11:44 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: >> I have a further request on interaction with penreg objects. >> I feel that it is not ideal that overlays associated with penreg objects >> cannot be fully controlled by the callers. > > I'm trying to limit the public API surface. I don't think we should > leak that we are currently using a mix of overlays and text > properties. Let me rephrase my concern - I do not like that after reglock is no longer live (got success/failure signal), there is no way to clean up the visual hints associated with this particular reglock. >> In particular, I'd like to see some way to >> >> 1. Create penreg object without locking the region, so that scheduled-at >> time is not set immediately and status overlay is not displayed. >> Then, `org-pending-send-update' could send :schedule signal to >> perform actual lock. > > Using the term "region" was confusing, sorry. That's why I switched > to region "lock". I don't think there is a use to create a lock that > doesn't lock. > > Also, that might be tricky to implement: `org-pending-send-update' is > called asynchronously, from the user point of view. Having regions > that suddenly become locked, independently of what the user is > currently doing (if we implement the :schedule message), might be > difficult. > > What use do you have in mind ? I was mostly confused about linkage between "visual hints" in buffer and how they are connected with reglock object. You may discard this comment of mine. >> 2. Act on the outcome overlays - there is currently no way to remove >> them using penreg object. > > I've added a funcion `org-pending-delete-outcome-marks' to manually > delete outcome marks that are in a given region. > > Else, everything is handled automatically. Once the outcome is known, > the reglock is dead (not live-p). org-pending may leave outcome marks > about the outcomes (outcome marks are optional). The outcome marks > automatically disappear if the user remove the section, or, if a new > lock is created for the same region. I do not like this. I'd like the Elisp program that creates the reglock to be able to clean up any visual hints associated with it. A function doing it for a given region cannot do this AFAIU. >> Maybe :cancel signal? Canceled penreg >> objects can then be garbage-collected from the manager. > > Cancel is handled by sending a failure message (see > `org-pending-cancel'). It's customizable using the reglock field > ~org-pending-reglock-user-cancel-function~, which can decide what to > do (like kill a process) and which can send a better outcome. > Standard 'cancel' leaves a failure outcome mark. Note that this function is not documented anywhere other than in reglock class documentation. In general, I am confused about your overall design of the user interaction with the locks. The updated top commentary explains well how Elisp programs can send data to the locks, but it does not say anything about how Elisp programs can receive the data. Also, I'd like to see more information in the top commentary about what are the "visual hints" displayed to the user and how to configure them. >> Also, the top-level commentary is getting incomplete and out-of-sync at >> this point. May you work towards more detailed top-level description of >> the library? > > Sorry. I tried hard to keep it in sync with all the modifications. > > I found it too much work, and, possibly overwhelming for the reader, > to explain everything in the top-level "Commentary" section. > > That's why I deleted everything that wasn't mandatory to understand > the core features. > > Everything should be documented as elisp documentation strings, > following the documentation of `org-pending' and > `org-pending-send-update', and, from code comments. I agree, but I'd like to see the core concepts explained on top: 1. How to create region lock (done) 2. What the region lock does (prohibit modifications, done) 3. How the lock is presented to the user and how to control the presentation (not done) 4. What can user do with the lock and how it is reflected in Elisp level (not done) 5. What can Elisp do with the lock (done) > I tried to improve the overall documentation. I hope it's going to be > easier for you, and others. Yes! Thanks! >> Is there any reason why you hide the extra information behind :-alist >> filed? Why not directly adding extra fields with proper documentation? > > To hide them, indeed :) > The API for 'get-status and 'get-live-p are > `org-pending-reglock-status' and `org-pending-reglock-live-p' (they > are read-only). The API for the new `useless-p' is > `org-pending-reglock-useless-p' (it's read-only too). We usually "hide" fields by declaring them private. Hiding them from the type docs is not a good idea because it defeats the purpose of type documentation in general. > The fields anchor-ovl, region-ovl, on-outcome, set-status and > creation-point are the dump of the closure context, so that > org-pending doesn't rely anymore on a closure to handle updates; I've > rewritten that recently. Nobody is supposed to use or change those > values, except the update process. > > IMHO, dumping those as fields in the lock structure would be more > confusing and fragile than keeping those out of sight. I could add > comments when they are created/used in the code to help understand how > they are used. I disagree. In particular, I dislike the fact that they are not documented anywhere and one has to read the internals of the code to understand their purpose. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-04-11 11:44 ` Ihor Radchenko @ 2024-04-19 11:23 ` Bruno Barbier 2024-04-20 10:07 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-04-19 11:23 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Jack Kamm, Matt Hi Ihor, Thanks for the review. I've pushed a new version, hoping to decrease the number of dislikes ;-) Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >>> I have a further request on interaction with penreg objects. >>> I feel that it is not ideal that overlays associated with penreg objects >>> cannot be fully controlled by the callers. >> >> I'm trying to limit the public API surface. I don't think we should >> leak that we are currently using a mix of overlays and text >> properties. > > Let me rephrase my concern - I do not like that after reglock is no > longer live (got success/failure signal), there is no way to clean up the > visual hints associated with this particular reglock. [....] For the org-pending library, "live" means "locked". Once the outcome is known, it can't be "live" anymore (it's unlocked); as it's not reusable, it's "dead". As the region is not locked anymore, the lock properties/fields can't be trusted anymore. But see below about removing the visual outcome hints of a given reglock. >>> 2. Act on the outcome overlays - there is currently no way to remove >>> them using penreg object. >> >> I've added a funcion `org-pending-delete-outcome-marks' to manually >> delete outcome marks that are in a given region. >> >> Else, everything is handled automatically. Once the outcome is known, >> the reglock is dead (not live-p). org-pending may leave outcome marks >> about the outcomes (outcome marks are optional). The outcome marks >> automatically disappear if the user remove the section, or, if a new >> lock is created for the same region. > > I do not like this. > I'd like the Elisp program that creates the reglock to be able to > clean up any visual hints associated with it. > A function doing it for a > given region cannot do this AFAIU. ok. I've added the function `org-pending-reglock-delete-outcome-marks, that will delete the outcome visual hints for a given reglock, if there are some. I updated how the lock is described to the user (org-pending-describe-reglock): I added a button "Forget" (if the lock is dead, that removes the outcome marks), and I added a "Cancel" button if the lock is still live. >>> Maybe :cancel signal? Canceled penreg >>> objects can then be garbage-collected from the manager. >> >> Cancel is handled by sending a failure message (see >> `org-pending-cancel'). It's customizable using the reglock field >> ~org-pending-reglock-user-cancel-function~, which can decide what to >> do (like kill a process) and which can send a better outcome. >> Standard 'cancel' leaves a failure outcome mark. > > Note that this function is not documented anywhere other than in reglock > class documentation. Thanks. I've improved the documentation of `org-pending' to mention that one may want to customize the following fields of a reglock: before-kill-function, user-cancel-function and insert-details-function. And, also, I added that one can attach custom properties using the "properties" field. > In general, I am confused about your overall design > of the user interaction with the locks. > The updated top commentary explains well how Elisp programs can send > data to the locks, but it does not say anything about how Elisp programs > can receive the data. An elisp program, that uses org-pending, must update the locks using `org-pending-send-update'. That program does not receive any data from the lock; it may customize Emacs behavior using the reglock fields mentioned above: before-kill-function, user-cancel-function and insert-details-function. Hopefully, it's clearer now with the improved documentation of the org-pending function. Just let me know if you still think that the top commentary should explain this. Thanks. > Also, I'd like to see more information in the top commentary about what > are the "visual hints" displayed to the user and how to configure them. If you think the current "visual hints" are good enough and could be shipped as-is, in a first version (indirect buffers, etc.); I could work on documenting them better. What kind of configuration are you thinking about ? just the faces ? or more advanced configurations ? [...] >>> Is there any reason why you hide the extra information behind :-alist >>> filed? Why not directly adding extra fields with proper documentation? >> >> To hide them, indeed :) > >> The API for 'get-status and 'get-live-p are >> `org-pending-reglock-status' and `org-pending-reglock-live-p' (they >> are read-only). The API for the new `useless-p' is >> `org-pending-reglock-useless-p' (it's read-only too). > > We usually "hide" fields by declaring them private. > Hiding them from the type docs is not a good idea because it defeats the > purpose of type documentation in general. > >> The fields anchor-ovl, region-ovl, on-outcome, set-status and >> creation-point are the dump of the closure context, so that >> org-pending doesn't rely anymore on a closure to handle updates; I've >> rewritten that recently. Nobody is supposed to use or change those >> values, except the update process. >> >> IMHO, dumping those as fields in the lock structure would be more >> confusing and fragile than keeping those out of sight. I could add >> comments when they are created/used in the code to help understand how >> they are used. > > I disagree. In particular, I dislike the fact that they are not > documented anywhere and one has to read the internals of the code to > understand their purpose. Done. I hope the minimal documentation is enough. Thanks again for your reviews and your comments, Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) 2024-04-19 11:23 ` Bruno Barbier @ 2024-04-20 10:07 ` Ihor Radchenko 0 siblings, 0 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-04-20 10:07 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode, Jack Kamm, Matt Bruno Barbier <brubar.cs@gmail.com> writes: > Thanks for the review. > > I've pushed a new version, hoping to decrease the number of dislikes > ;-) Thanks! >>> Cancel is handled by sending a failure message (see >>> `org-pending-cancel'). It's customizable using the reglock field >>> ~org-pending-reglock-user-cancel-function~, which can decide what to >>> do (like kill a process) and which can send a better outcome. >>> Standard 'cancel' leaves a failure outcome mark. >> >> Note that this function is not documented anywhere other than in reglock >> class documentation. > > Thanks. I've improved the documentation of `org-pending' to mention > that one may want to customize the following fields of a reglock: > before-kill-function, user-cancel-function and > insert-details-function. And, also, I added that one can attach > custom properties using the "properties" field. Thanks, but I still feel confused. May you: 1. Explain what does "kill" and "cancel" mean in the context of the REGLOCK. 2. Add information about visual hints, "kill", and "cancel" to the top-level commentary. For now, when reading the top commentary: ;; This library contains an API to lock a region while it is "being ;; updated"; the content of the region is "pending" and cannot be ;; modified. It will be updated, later, when the new content is ;; available. I have an impression that the only side effect of "locking" is that the region becomes read-only. It is not clear at all that any other visual indication will be provided. >> In general, I am confused about your overall design >> of the user interaction with the locks. >> The updated top commentary explains well how Elisp programs can send >> data to the locks, but it does not say anything about how Elisp programs >> can receive the data. > > An elisp program, that uses org-pending, must update the locks using > `org-pending-send-update'. That program does not receive any data > from the lock; it may customize Emacs behavior using the reglock > fields mentioned above: before-kill-function, user-cancel-function and > insert-details-function. > > Hopefully, it's clearer now with the improved documentation of the > org-pending function. > > Just let me know if you still think that the top commentary should > explain this. Thanks. Yes, I do think that top commentary should explain this. The very idea that lock may be "canceled" or "killed" is important when designing Elisp code that uses the org-pending library. >> Also, I'd like to see more information in the top commentary about what >> are the "visual hints" displayed to the user and how to configure them. > > If you think the current "visual hints" are good enough and could be > shipped as-is, in a first version (indirect buffers, etc.); I could > work on documenting them better. What kind of configuration are you > thinking about ? just the faces ? or more advanced configurations ? I am not thinking about details yet. I just want insert-details-function to be described in the top commentary. At high level. Something like * Visual indication of the "pending" regions While the region is locked, it is visually highlighted in the buffer, providing information about the lock status as an overlay. The status can be: 1. :scheduled - the region is "being updated", but the computation has not yet started. 2. :progress - the computation is in progress After unlocking the region, the visual indication is not necessarily removed. Instead, the outcome is indicated in an overlay. The outcome may be: 1. :success - the computation has been successful and the region text has been updated 2. :failure - the computation failed The lock status is updated according to the data submitted to REGLOCK object via `org-pending-send-update'. * User interaction with pending regions For any pending region, users may request detailed description to be displayed. The description includes the pending region status, creation time, outcome, duration, results, errors, etc. Elisp code using this library may also supply additional details about any given reglock, via `insert-details-function' field of REGLOCK object. For example, process logs can be displayed this way. -------- I hope that the above clarifies what I am looking for. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-16 17:52 ` Bruno Barbier 2024-02-18 21:14 ` Matt @ 2024-02-19 0:15 ` Jack Kamm 2024-02-21 15:43 ` Bruno Barbier 2024-02-19 9:06 ` Ihor Radchenko 2 siblings, 1 reply; 55+ messages in thread From: Jack Kamm @ 2024-02-19 0:15 UTC (permalink / raw) To: Bruno Barbier, Matt; +Cc: Ihor Radchenko, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > I'm not using it with official org backends (yet). I'm using it with > several custom backends that I'm working on. One of the backend > delegate the block executions to emacs subprocesses: so I have a kind of > asynchronous executions for free for any language, including elisp > itself. For sessions, wouldn't running in a subprocess prevent the user from directly interacting with the REPL outside of Org? If so, that's a problem. Org-babel sessions need to play nicely with inferior Python, inferior ESS, and other interactive comint modes. > So, here we go. You'll find attach a set of patchs. It works for me with > Emacsc 30.50 and 9.7-pre (from today). I suggest to keep these patches on a public branch somewhere, see: https://orgmode.org/worg/org-contribute.html#patches "When discussing important changes, it is sometimes not so useful to send long and/or numerous patches. In this case, you can maintain your changes on a public branch of a public clone of Org and send a link to the diff between your changes and the latest Org commit that sits in your clone." I tried running your example on emacs29 using emacs -q -L /path/to/org-mode/lisp my-async-tests.org but it fails with the error below. Also "make" gives a bunch of compilation warnings (which I've put at the bottom). > +You need to load: > + #+begin_src elisp :results silent > + (load-file "my-async-tests.el") > + #+end_src This raises the following error in *Org-Babel Error Output* void-variable (org-elib-async-process) [ Babel evaluation exited with code 127 ] All the subsequent blocks don't work because of that, for example: > +A simple execution: > + #+begin_src bash > + date > + #+end_src yields: org-babel-execute-src-block: No org-babel-execute function for bash: my-shell-babel-schedule! Finally here are the warnings when running "make": Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/ob-core.el... In org-babel--async-feedbacks: ob-core.el:851:2: Warning: docstring has wrong usage of unescaped single quotes (use \= or different quoting) ob-core.el:871:9: Warning: Unused lexical variable `result-indent' ob-core.el:906:18: Warning: Unused lexical variable `header' Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/org-elib-async.el... In toplevel form: org-elib-async.el:52:11: Warning: reference to free variable ‘org-elib-async-process’ org-elib-async.el:52:43: Warning: reference to free variable ‘&key’ org-elib-async.el:52:48: Warning: reference to free variable ‘input’ org-elib-async.el:52:54: Warning: reference to free variable ‘callback’ org-elib-async.el:78:29: Warning: reference to free variable ‘command’ org-elib-async.el:127:9: Warning: Variable ‘last-elapsed’ left uninitialized In org-elib-async-wait-condition: org-elib-async.el:137:12: Warning: ‘signal’ called with 3 arguments, but accepts only 2 In end of data: org-elib-async.el:262:16: Warning: the function ‘org-id-uuid’ is not known to be defined. org-elib-async.el:52:35: Warning: the function ‘command’ is not known to be defined. org-elib-async.el:52:2: Warning: the function ‘cl-defun’ might not be defined at runtime. ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 0:15 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm @ 2024-02-21 15:43 ` Bruno Barbier 0 siblings, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-21 15:43 UTC (permalink / raw) To: Jack Kamm, Matt; +Cc: Ihor Radchenko, emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 2427 bytes --] Hi Jack, Jack Kamm <jackkamm@gmail.com> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> I'm not using it with official org backends (yet). I'm using it with >> several custom backends that I'm working on. One of the backend >> delegate the block executions to emacs subprocesses: so I have a kind of >> asynchronous executions for free for any language, including elisp >> itself. > > For sessions, wouldn't running in a subprocess prevent the user from > directly interacting with the REPL outside of Org? Good point. The REPL should be created in the same subprocess; the REPL display and interaction must happen in the user main emacs. If the REPL is based on comint, it should be relatively easy to implement. > If so, that's a problem. Org-babel sessions need to play nicely with > inferior Python, inferior ESS, and other interactive comint modes. With this solution, the user and the REPL/execution will be in separate processes; so there will be disavantages. For basic interactions, mostly based on text input/output, it should work well. >> So, here we go. You'll find attach a set of patchs. It works for me with >> Emacsc 30.50 and 9.7-pre (from today). > > I suggest to keep these patches on a public branch somewhere, see: > https://orgmode.org/worg/org-contribute.html#patches > > "When discussing important changes, it is sometimes not so useful to > send long and/or numerous patches. > > In this case, you can maintain your changes on a public branch of a > public clone of Org and send a link to the diff between your changes > and the latest Org commit that sits in your clone." Good point. I'll switch to such a solution as soon as possible. > I tried running your example on emacs29 using > > emacs -q -L /path/to/org-mode/lisp my-async-tests.org > > but it fails with the error below. Also "make" gives a bunch of > compilation warnings (which I've put at the bottom). > ... My bad: I should have compiled the demo code in a standalone emacs. I forgot to require some libraries: cl-lib and org-id. I've now tested with your command line (thanks). It should now work. Sorry about that. > Finally here are the warnings when running "make": I should have fixed everything; no more (new) warnings. Thanks! Please find attached the new set of patchs. I'll switch to using a clone and a branch soon, in case if you prefer to wait. Thanks again! Bruno [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --] [-- Type: text/x-patch, Size: 1367 bytes --] From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:31:36 +0100 Subject: [PATCH 1/8] ob-core async: Add faces [1/5] lisp/org-faces.el (org-async-scheduled, org-async-pending, org-async-failure): new faces --- lisp/org-faces.el | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lisp/org-faces.el b/lisp/org-faces.el index 0e20de51a..5a8a8fd51 100644 --- a/lisp/org-faces.el +++ b/lisp/org-faces.el @@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun "Face used for clock display for overrun tasks in mode line." :group 'org-faces) +(defface org-async-scheduled '((t :inherit org-tag :background "gray")) + "Face for babel results for code blocks that are scheduled for execution." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + +(defface org-async-pending '((t :inherit org-checkbox :background "dark orange")) + "Face for babel results for code blocks that are running." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + +(defface org-async-failure '((t :inherit org-warning)) + "Face for babel results for code blocks that have failed." + :group 'org-faces + :version "27.2" + :package-version '(Org . "9.5")) + (provide 'org-faces) ;;; org-faces.el ends here -- 2.43.0 [-- Attachment #3: 0001-ob-core-async-Add-faces-1-5.patch --] [-- Type: text/x-patch, Size: 10343 bytes --] From 9f135bd5e8e153323bed5a3274851fa78f246b83 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:00 +0100 Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5] --- lisp/ob-core.el | 213 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index bfeac257b..d98626fe8 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -792,6 +792,219 @@ (defun org-babel-session-buffer (&optional info) (when (org-babel-comint-buffer-livep buffer-name) buffer-name))) +(defun org-babel--async-status-face (status) + (pcase status + (:scheduled 'org-async-scheduled) + (:pending 'org-async-pending) + (:failure 'org-async-failure) + (:success nil) + (_ (error "Not a status")) + )) + +(defun org-babel--async-make-overlay (beg end) + "Create an overlay between positions BEG and END and return it." + (let ((overlay (make-overlay beg end)) + (read-only + (list + (lambda (&rest _) + (user-error + "Cannot modify an area being updated")))) + ) + (cl-flet ((make-read-only + (ovl) + (overlay-put ovl 'modification-hooks read-only) + (overlay-put ovl 'insert-in-front-hooks read-only) + (overlay-put ovl 'insert-behind-hooks read-only)) + ) + (overlay-put overlay 'org-babel--async-type 'org-babel--async-note) + (overlay-put overlay 'face 'secondary-selection) + (overlay-put overlay 'help-echo "Pending src block result...") + (make-read-only overlay) + overlay))) + +(defun org-babel--async-result-region (inline-elem &optional info) + "Return the region of the results, for the source block at point." + (unless info (setq info (org-babel-get-src-block-info))) + (save-excursion + (when-let ((res-begin (org-babel-where-is-src-block-result nil info))) + (cons res-begin + (save-excursion + (goto-char res-begin) + (if inline-elem + ;; Logic copy/pasted from org-babel-where-is-src-block-result. + (let ((result (org-element-context))) + (and (org-element-type-p result 'macro) + (string= (org-element-property :key result) + "results") + (progn + (goto-char (org-element-end result)) + (skip-chars-backward " \t") + (point)))) + ;; Logic copy/pasted from hide-result + (beginning-of-line) + (let ((case-fold-search t)) + (unless (re-search-forward org-babel-result-regexp nil t) + (error "Not looking at a result line"))) + (org-babel-result-end) + )))))) + +(defun org-babel--async-feedbacks (info handle-result + result-params exec-start-time) + "Flag the result as \='scheduled\=' and return how to handle feedbacks. + +Use overlays to report progress and status to the user. Do not delete +the existing result unless a new one is available. When the result is +available, remove the async overlays and insert the result as usual, +like for a synchronous result. In case of failure, use an overlay to +report the error. + +The returned function handles 3 types of feedbacks: + - (:success R): Evaluation is successful; result is R. + - (:failure ERR): Evaluation failed; error is ERR. + - (:pending P): Outcome still pending; current progress is P." + ;; FIXME: INFO CMD ... Nothing is used but handle-result here !! + (let (;; copy/pasted from org-babel-insert-result + (inline-elem (let ((context (org-element-context))) + (and (memq (org-element-type context) + '(inline-babel-call inline-src-block)) + context)))) + (cl-labels + ((eot-point (start) + "Move to End Of Title after START" + (if inline-elem + (org-element-end inline-elem) + (save-excursion (goto-char start) + (forward-line 1) (point)))) + (after-indent (pt) + "Move after indentation, starting at PT." + (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*"))) + (mk-result-overlays () + ;; Make 2 overlays to handle the pending result: one title + ;; (first line) and one for the body. + (pcase-let ((`(,start . ,end) (org-babel--async-result-region + inline-elem info))) + (let ((anchor-end (eot-point start))) + (cons (org-babel--async-make-overlay + (after-indent start) + (1- anchor-end)) + (org-babel--async-make-overlay + anchor-end end))))) + (add-style (status txt) + ;; Add the style matching STATUS over the text TXT. + (propertize txt 'face (org-babel--async-status-face status))) + + (short-version-of (msg) + ;; Compute the short version of MSG, to display in the header. + ;; Must return a string. + (if msg + (car (split-string (format "%s" msg) "\n" :omit-nulls)) + "")) + (update (ovl-title status msg) + ;; Update the title overlay to match STATUS and MSG. + (overlay-put ovl-title + 'face + (org-babel--async-status-face status)) + (overlay-put ovl-title + 'before-string (pcase status + (:scheduled "⏱") + (:pending "⏳") + (:failure "❌") + (:success "✔️"))) + (overlay-put ovl-title + 'after-string + (propertize (format " |%s|" + (if (eq :failure status) + (if (consp msg) (car msg) + (format "%s" msg)) + (short-version-of msg))) + 'face (org-babel--async-status-face status)))) + (remove-previous-overlays () + ;; Remove previous title and body overlays. + (mapc (lambda (ovl) + (when (eq 'org-babel--async-note + (overlay-get ovl 'org-babel--async-type)) + (delete-overlay ovl))) + (when-let ((region (org-babel--async-result-region + inline-elem info))) + ;; Not sure why, but we do need to start before + ;; point min, else, in some cases, some overlays + ;; are not found. + (overlays-in (max (1- (car region)) (point-min)) + (cdr region)))))) + + (remove-previous-overlays) + + ;; Ensure there is a non-empty region for the result. + (save-excursion + (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil) + (org-babel-insert-result + ;; Use " " for the empty result. That cannot be nil, else it's interpreted + ;; as a list. We need at least one char, to separate markers if any. + " \n" + result-params + info nil + (nth 0 info) ; lang + exec-start-time + ))) + + ;; Create the overlays that span the result title and its body. + (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays))) + ;; Flag the result as ":scheduled". + (update title-ovl :scheduled nil) + + ;; The callback, that runs in the org buffer at point. + (let ((buf (current-buffer)) + (pt (point-marker))) + (lambda (feedback) + (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback) + (with-current-buffer buf + (save-excursion + (goto-char pt) + (pcase feedback + (`(:success ,r) + ;; Visual beep that the result is available. + (update title-ovl :success r) + (sit-for 0.2) + ;; We remove all overlays and let org insert the result + ;; as it would in the synchronous case. + (delete-overlay title-ovl) + (delete-overlay body-ovl) + (funcall handle-result r)) + + (`(:pending ,r) + ;; Still waiting for the outcome. Update our + ;; overlays with the progress info R. + (message "Updating block at %s@%s" pt buf) + (update title-ovl :pending r)) + + (`(:failure ,err) + ;; We didn't get a result. We update our overlays + ;; to report that failure. And unlock the old + ;; result. + (overlay-put title-ovl 'face nil) + (update title-ovl :failure err) + (delete-overlay body-ovl)) + + (_ (error "Invalid outcome")) + ) + )) + nil)))))) + + +(cl-defun org-babel--async-p (params &key default) + "Return a non-nil value when the execution is asynchronous. +Get the value of the :nasync argument and convert it." + (if-let ((binding (assq :nasync params))) + (pcase (cdr binding) + ((pred (not stringp)) + (error "Invalid value for :nasync argument")) + ((or "no" "n") nil) + ((or "yes" "y") t) + (_ (error "Invalid value for :nasync argument"))) + default)) + + + ;;;###autoload (defun org-babel-execute-src-block (&optional arg info params executor-type) "Execute the current source code block and return the result. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --] [-- Type: text/x-patch, Size: 5958 bytes --] From b0cdd3f5a9bd6e4e72adeac91f968741da95f98f Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:22 +0100 Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5] lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to prepare for the next change: move the part handling the result in its own function `handle-result'. --- lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------ 1 file changed, 53 insertions(+), 52 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index d98626fe8..d1adba61c 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1086,7 +1086,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (make-directory d 'parents) d)))) (cmd (intern (concat "org-babel-execute:" lang))) - result exec-start-time) + (exec-start-time (current-time)) + (handle-result + (lambda (result) + (setq result + (if (and (eq (cdr (assq :result-type params)) 'value) + (or (member "vector" result-params) + (member "table" result-params)) + (not (listp result))) + (list (list result)) + result)) + (let ((file (and (member "file" result-params) + (cdr (assq :file params))))) + ;; If non-empty result and :file then write to :file. + (when file + ;; If `:results' are special types like `link' or + ;; `graphics', don't write result to `:file'. Only + ;; insert a link to `:file'. + (when (and result + (not (or (member "link" result-params) + (member "graphics" result-params)))) + (with-temp-file file + (insert (org-babel-format-result + result + (cdr (assq :sep params))))) + ;; Set file permissions if header argument + ;; `:file-mode' is provided. + (when (assq :file-mode params) + (set-file-modes file (cdr (assq :file-mode params))))) + (setq result file)) + ;; Possibly perform post process provided its + ;; appropriate. Dynamically bind "*this*" to the + ;; actual results of the block. + (let ((post (cdr (assq :post params)))) + (when post + (let ((*this* (if (not file) result + (org-babel-result-to-file + file + (org-babel--file-desc params result) + 'attachment)))) + (setq result (org-babel-ref-resolve post)) + (when file + (setq result-params (remove "file" result-params)))))) + (unless (member "none" result-params) + (org-babel-insert-result + result result-params info + ;; append/prepend cannot handle hash as we accumulate + ;; multiple outputs together. + (when (member "replace" result-params) new-hash) + lang + (time-subtract (current-time) exec-start-time))) + (run-hooks 'org-babel-after-execute-hook) + result)))) (unless (fboundp cmd) (error "No org-babel-execute function for %s!" lang)) (message "Executing %s %s %s..." @@ -1101,57 +1152,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (if name (format "(%s)" name) (format "at position %S" (nth 5 info))))) - (setq exec-start-time (current-time) - result - (let ((r (save-current-buffer (funcall cmd body params)))) - (if (and (eq (cdr (assq :result-type params)) 'value) - (or (member "vector" result-params) - (member "table" result-params)) - (not (listp r))) - (list (list r)) - r))) - (let ((file (and (member "file" result-params) - (cdr (assq :file params))))) - ;; If non-empty result and :file then write to :file. - (when file - ;; If `:results' are special types like `link' or - ;; `graphics', don't write result to `:file'. Only - ;; insert a link to `:file'. - (when (and result - (not (or (member "link" result-params) - (member "graphics" result-params)))) - (with-temp-file file - (insert (org-babel-format-result - result - (cdr (assq :sep params))))) - ;; Set file permissions if header argument - ;; `:file-mode' is provided. - (when (assq :file-mode params) - (set-file-modes file (cdr (assq :file-mode params))))) - (setq result file)) - ;; Possibly perform post process provided its - ;; appropriate. Dynamically bind "*this*" to the - ;; actual results of the block. - (let ((post (cdr (assq :post params)))) - (when post - (let ((*this* (if (not file) result - (org-babel-result-to-file - file - (org-babel--file-desc params result) - 'attachment)))) - (setq result (org-babel-ref-resolve post)) - (when file - (setq result-params (remove "file" result-params)))))) - (unless (member "none" result-params) - (org-babel-insert-result - result result-params info - ;; append/prepend cannot handle hash as we accumulate - ;; multiple outputs together. - (when (member "replace" result-params) new-hash) - lang - (time-subtract (current-time) exec-start-time)))) - (run-hooks 'org-babel-after-execute-hook) - result))))))) + (funcall handle-result (save-current-buffer (funcall cmd body params)))))))))) (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --] [-- Type: text/x-patch, Size: 2620 bytes --] From 019a0d2d8ba042606632e13976f8dfaeb37a8e74 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:32:45 +0100 Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5] --- lisp/ob-core.el | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index d1adba61c..262218923 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1085,7 +1085,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (let ((d (file-name-as-directory (expand-file-name dir)))) (make-directory d 'parents) d)))) - (cmd (intern (concat "org-babel-execute:" lang))) + (async (org-babel--async-p params)) + (cmd (intern (concat "org-babel-" + (if async "schedule" "execute") + ":" lang))) (exec-start-time (current-time)) (handle-result (lambda (result) @@ -1139,8 +1142,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (run-hooks 'org-babel-after-execute-hook) result)))) (unless (fboundp cmd) - (error "No org-babel-execute function for %s!" lang)) - (message "Executing %s %s %s..." + (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd))) + (message "Executing %s %s %s %s..." (capitalize lang) (pcase executor-type ('src-block "code block") @@ -1148,11 +1151,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) ('babel-call "call") ('inline-babel-call "inline call") (e (symbol-name e))) + (if async "async" "") (let ((name (nth 4 info))) (if name (format "(%s)" name) (format "at position %S" (nth 5 info))))) - (funcall handle-result (save-current-buffer (funcall cmd body params)))))))))) + (if (not async) + (funcall handle-result (save-current-buffer (funcall cmd body params))) + (let ((handle-feedback + (org-babel--async-feedbacks info handle-result result-params exec-start-time))) + (funcall cmd body params handle-feedback)))))))))) + (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --] [-- Type: text/x-patch, Size: 2458 bytes --] From c17aaea885e3aa6087563d55045e74ce71557dbf Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:09 +0100 Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5] --- lisp/ob-core.el | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 262218923..64f434f71 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -1086,9 +1086,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (make-directory d 'parents) d)))) (async (org-babel--async-p params)) - (cmd (intern (concat "org-babel-" - (if async "schedule" "execute") - ":" lang))) + (execute-with (let ((be (cdr (assq :execute-with params)))) + (when (equal be "none") (setq be nil)) + be)) + (cmd (intern (or (and execute-with + (concat execute-with "-" (if async "schedule" "execute"))) + (concat "org-babel-" + (if async "schedule" "execute") + ":" lang)))) + (cmd-args (let ((ps (list body params))) + (when execute-with + (setq ps (cons lang ps))) + ps)) (exec-start-time (current-time)) (handle-result (lambda (result) @@ -1157,10 +1166,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (format "(%s)" name) (format "at position %S" (nth 5 info))))) (if (not async) - (funcall handle-result (save-current-buffer (funcall cmd body params))) + (funcall handle-result (save-current-buffer (apply cmd cmd-args))) (let ((handle-feedback (org-babel--async-feedbacks info handle-result result-params exec-start-time))) - (funcall cmd body params handle-feedback)))))))))) + (apply cmd (nconc cmd-args (list handle-feedback)))))))))))) (defun org-babel-expand-body:generic (body params &optional var-lines) -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --] [-- Type: text/x-patch, Size: 14628 bytes --] From d5766eada2c31d0886f514f5ac6b38ad342e158f Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:23 +0100 Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers --- lisp/org-elib-async.el | 327 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 327 insertions(+) create mode 100644 lisp/org-elib-async.el diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el new file mode 100644 index 000000000..f0a1e4432 --- /dev/null +++ b/lisp/org-elib-async.el @@ -0,0 +1,327 @@ +;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*- + +;; Copyright (C) 2024 Bruno BARBIER + +;; Author: Bruno BARBIER +;; Version: 0.0.0 +;; Maintainer: Bruno BARBIER +;; Keywords: +;; Status: WORK IN PROGRESS. DO NOT USE. +;; URL: +;; Compatibility: GNU Emacs 30.0.50 +;; +;; This file is NOT (yet) part of GNU Emacs. + +;; This program is free software; you can redistribute it and/or +;; modify it under the terms of the GNU General Public License as +;; published by the Free Software Foundation; either version 2 of +;; the License, or (at your option) any later version. + +;; This program is distributed in the hope that it will be +;; useful, but WITHOUT ANY WARRANTY; without even the implied +;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +;; PURPOSE. See the GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public +;; License along with this program; if not, write to the Free +;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, +;; MA 02111-1307 USA + + +;;; Commentary: +;; Names with "--" are for functions and variables that are meant to be for +;; internal use only. + +;;;; Description +;; Some functions to help dealing with asynchronous tasks. + +;; The prefix 'org-elib' means that this package should evenutally be +;; moved into core Emacs. The functions defined here do NOT depend +;; nor rely on org itself. + +;;; TODOs +;; +;; - Keywords +;; + + +;;; Code: +;; +(require 'cl-lib) +(require 'org-id) + +;;;; Process +;; +(cl-defun org-elib-async-process (command &key input callback) + "Execute COMMAND. + +A quick naive featureless boggus wrapper around `make-process' to +receive the result when the process is done. + +When INPUT is non-nil, use it as the COMMAND standard input. Let DATA +be the COMMAND output, if COMMAND succeeds, call CALLBACK with +(:success DATA), else, call CALLBACK with (:failure DATA)." + (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*")) + (get-outcome + (lambda (process) + (with-current-buffer stdout-buffer + (let* ((exit-code (process-exit-status process)) + (real-end ;; Getting rid of the user message. + (progn (goto-char (point-max)) + (forward-line -1) + (point))) + (txt (string-trim (buffer-substring-no-properties + (point-min) real-end)))) + (list (if (eq 0 exit-code) :success :failure) + (if (not (string-empty-p txt)) txt + (and (not (eq 0 exit-code)) exit-code))))))) + (process (make-process + :name "*org-elib-async-process*" + :buffer stdout-buffer + :command command + :connection-type 'pipe)) + (sentinel + (lambda (&rest _whatever) + (pcase (process-status process) + ('run ) + ('stop) + ((or 'exit 'signal) + (funcall callback (funcall get-outcome process))) + (_ (error "Not a real process")))))) + (add-function :after (process-sentinel process) sentinel) + (when input + (process-send-string process input) + (process-send-eof process)) + process)) +;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o))) +;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o))) + + +;;;; Wait for a process until some condition becomes true. + +(define-error 'org-elib-async-timeout-error + "Timeout waiting for a process.") + +(cl-defun org-elib-async-wait-condition ( cond-p + &key + (tick .3) (message "Waiting") + (nb_secs_between_messages 5) + timeout) + "Wait until the condition COND-P returns non-nil. +Repeatedly call COND-P with no arguments, about every TICK seconds, +until it returns a non-nil value. Return that non-nil value. When +TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if +the COND-P is still nil after TIMEOUT seconds. Assume COND-P calls cost +0s. Do NOT block display updates. Do not block process outputs. Do +not block idle timers. Do block the user, letting him/her know why, but +do not display more messages than one every NB_SECS_BETWEEN_MESSAGES. +Default MESSAGE is \"Waiting\". Use 0.3s as the default for TICK." + ;; FIXME: Still not sure if it's possible to write such a function. + (let ((keep-waiting t) + (result nil) + (start (float-time)) + elapsed + last-elapsed) + (while keep-waiting + (setq result (funcall cond-p)) + (if result + (setq keep-waiting nil) + (sleep-for 0.01) + (redisplay :force) + (setq elapsed (- (float-time) start)) + (when (and timeout (> elapsed timeout)) + (signal 'org-timeout-error (list message elapsed))) + ;; Let the user know, without flooding the message area. + (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages)) + (message (format "%s ...(%.1fs)" message elapsed))) + (unless (sit-for tick :redisplay) + ;; Emacs has something to do; let it process new + ;; sub-processes outputs in case there are some. + (accept-process-output nil 0.01)) + (setq last-elapsed elapsed))) + result)) + + + +;;;; Comint: a FIFO queue of tasks with callbacks +;; org-elib-async-comint-queue executes tasks in a FIFO order. For each +;; task, it identifies the text output for that +;; task. org-elib-async-comint-queue does NOT remove prompts, or other +;; useless texts; this is the responsibility of the user. Currently, +;; org-elib-async-comint-queue assume it has the full control of the +;; session: no user interaction, no other direct modifications. + +(defvar-local org-elib-async-comint-queue--todo :NOT-SET + "A FIFO queue of pending executions.") + + +(defvar-local org-elib-async-comint-queue--unused-output "" + "Process output that has not been used yet.") + +(defvar-local org-elib-async-comint-queue--incoming-text "" + "Newly incoming text, added by the process filter, not yet handled.") + +(defvar-local org-elib-async-comint-queue--current-task nil + "The task that is currently running.") + +(defvar-local org-elib-async-comint-queue--process-filter-running nil + "non-nil when filter is running.") + +(defvar-local org-elib-async-comint-queue--incoming-timer nil + "A timer, when handling incoming text is scheduled or running.") + + +(defvar-local org-elib-async-comint-queue--handle-incoming-running + nil + "True when the incoming text handler is running.") + +(defun org-elib-async-comint-queue--handle-incoming () + (when org-elib-async-comint-queue--handle-incoming-running + (error "Bad call to handle-incoming: kill buffer %s!" (current-buffer))) + (setq org-elib-async-comint-queue--handle-incoming-running t) + + ;; Take the incoming text. + (setq org-elib-async-comint-queue--unused-output + (concat org-elib-async-comint-queue--unused-output + org-elib-async-comint-queue--incoming-text)) + (setq org-elib-async-comint-queue--incoming-text "") + + ;; Process the unused text with the queued tasks + (unless org-elib-async-comint-queue--current-task + (when org-elib-async-comint-queue--todo + (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo)))) + (when-let ((task org-elib-async-comint-queue--current-task)) + (let ((unused org-elib-async-comint-queue--unused-output) + (session-buffer (current-buffer)) + task-start) + (setq org-elib-async-comint-queue--unused-output + (with-temp-buffer + (insert unused) + (goto-char (point-min)) + (while (and task + (setq task-start (point)) + (search-forward (car task) nil t)) + (when (cdr task) + (let ((txt (buffer-substring-no-properties task-start + (- (point) (length (car task)))))) + (save-excursion (funcall (cdr task) txt)))) + (setq task (and (buffer-live-p session-buffer) + (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo))))) + (buffer-substring (point) (point-max)))) + (setq org-elib-async-comint-queue--current-task task))) + + ;; Signal that we are done. If we already have some new incoming text, + ;; reschedule to run. + (setq org-elib-async-comint-queue--incoming-timer + (if (string-empty-p org-elib-async-comint-queue--incoming-text) + nil + (org-elib-async-comint-queue--wake-up-handle-incoming))) + + ;; We reset it only on success. If it failed for some reason, the + ;; comint buffer is in an unknown state: you'll need to kill that + ;; buffer. + (setq org-elib-async-comint-queue--handle-incoming-running nil)) + + +(defun org-elib-async-comint-queue--wake-up-handle-incoming () + "Wake up the handling of incoming chunks of text. +Assume we are called from the comint buffer." + (setq org-elib-async-comint-queue--incoming-timer + (run-with-timer + 0.01 nil + (let ((comint-buffer (current-buffer))) + (lambda () + (with-local-quit + (with-current-buffer comint-buffer + (org-elib-async-comint-queue--handle-incoming)))))))) + + +(defun org-elib-async-comint-queue--process-filter (chunk) + "Accept the arbitrary CHUNK of text." + (setq org-elib-async-comint-queue--incoming-text + (concat org-elib-async-comint-queue--incoming-text + chunk)) + :; We delegate the real work outside the process filter, as it is + ; not reliable to do anything here. + (unless org-elib-async-comint-queue--incoming-timer + (org-elib-async-comint-queue--wake-up-handle-incoming))) + + + +(define-error 'org-elib-async-comint-queue-task-error + "Task failure.") + +(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback) + "Push the execution of EXEC into the FIFO queue. +When the task completed, call HANDLE-FEEDBACK with its outcome. Return +a function that waits for and return the result on succes, raise on +failure." + (let* ((tid (org-id-uuid)) + (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid)) + (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid)) + (result-sb (make-symbol "result")) + (on-start + (lambda (_) + ;; TODO: Use (point) in session to link back to it. + (when handle-feedback + (funcall handle-feedback '(:pending "running"))))) + (on-result + (lambda (result) + ;; Get the result, and report success using HANDLE-FEEDBACK. + ;; If something fails, report failure using HANDLE-FEEDBACK. + (unwind-protect + (let ((outcome + (condition-case-unless-debug exc + (list :success (funcall exec :post-process result)) + (error (list :failure exc))))) + (when handle-feedback (save-excursion (funcall handle-feedback outcome))) + (set result-sb outcome)) + (funcall exec :finally))))) + + ;; TODO: Add detect-properties => alist of properties that can be used: PS1 and PS2 + (let ((comint-buffer (funcall exec :get-comint-buffer))) + (with-current-buffer comint-buffer + (setq org-elib-async-comint-queue--todo + (nconc org-elib-async-comint-queue--todo + (list (cons start-tag on-start) + (cons end-tag on-result)))) + (funcall exec :send-instrs-to-session + (funcall exec :instrs-to-enter)) + (funcall exec :send-instrs-to-session + (funcall exec :instr-to-emit-tag start-tag)) + (funcall exec :send-instrs-to-session + (funcall exec :get-code)) + (funcall exec :send-instrs-to-session + (funcall exec :instr-to-emit-tag end-tag)) + (funcall exec :send-instrs-to-session + (funcall exec :instrs-to-exit)) + + (lambda () + (org-elib-async-wait-condition (lambda () + (boundp result-sb))) + (pcase (symbol-value result-sb) + (`(:success ,r) r) + (`(:failure ,err) (signal (car err) (cdr err))))) + )))) + + + +(defun org-elib-async-comint-queue-init-if-needed (buffer) + "Initialize the FIFO queue in BUFFER if needed." + (with-current-buffer buffer + (unless (local-variable-p 'org-elib-async-comint-queue--todo) + (setq-local org-elib-async-comint-queue--todo nil) + (add-hook 'comint-output-filter-functions + #'org-elib-async-comint-queue--process-filter nil :local)))) + + + +;;;; Provide +(provide 'org-elib-async) +;;; org-elib-async.el ends here -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --] [-- Type: text/x-patch, Size: 3220 bytes --] From 1129770cc62b0580c71ba2a3f6a94f6fd18574b3 Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Fri, 16 Feb 2024 14:33:40 +0100 Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails lisp/ob-core.el (org-babel-popup-failure-details): New function. (org-babel-execute-src-block): For synchronous execution, use `org-babel-popup-failure-details' on failure. (org-babel--async-feedbacks): Add call to `org-babel-popup-failure-details' on user request. --- lisp/ob-core.el | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 64f434f71..e8f9e9ad9 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -917,7 +917,17 @@ (defun org-babel--async-feedbacks (info handle-result (if (consp msg) (car msg) (format "%s" msg)) (short-version-of msg))) - 'face (org-babel--async-status-face status)))) + 'face (org-babel--async-status-face status))) + (when (eq :failure status) + (overlay-put ovl-title + 'keymap + (let ((km (make-sparse-keymap))) + (define-key km (kbd "<mouse-1>") + (lambda () + "Display failure details." + (interactive) + (org-babel-popup-failure-details msg))) + km)))) (remove-previous-overlays () ;; Remove previous title and body overlays. (mapc (lambda (ovl) @@ -1166,11 +1176,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type) (format "(%s)" name) (format "at position %S" (nth 5 info))))) (if (not async) - (funcall handle-result (save-current-buffer (apply cmd cmd-args))) + (let ((res-sb (make-symbol "result"))) + (condition-case exc + (set res-sb (save-current-buffer (apply cmd cmd-args))) + (error (org-babel-popup-failure-details exc))) + (when (boundp res-sb) + (funcall handle-result (symbol-value res-sb)))) (let ((handle-feedback (org-babel--async-feedbacks info handle-result result-params exec-start-time))) (apply cmd (nconc cmd-args (list handle-feedback)))))))))))) +(defun org-babel-popup-failure-details (exc) + "Notify/display" + (when-let ((buf (get-buffer org-babel-error-buffer-name))) + (with-current-buffer buf (erase-buffer))) + (org-babel-eval-error-notify + 127 ; Don't have exit-code + (if (consp exc) + (format "%s\n%s\n" (car exc) (cdr exc)) + (format "%s\n" exc)))) + (defun org-babel-expand-body:generic (body params &optional var-lines) "Expand BODY with PARAMS. -- 2.43.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #9: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --] [-- Type: text/x-patch, Size: 33567 bytes --] From 2078be0ebd741127c383a386f672dbc3d741206d Mon Sep 17 00:00:00 2001 From: Bruno BARBIER <brubar.cs@gmail.com> Date: Wed, 21 Feb 2024 15:54:05 +0100 Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files --- scratch/bba-ob-core-async/my-async-tests.el | 339 ++++++++ scratch/bba-ob-core-async/my-async-tests.org | 820 +++++++++++++++++++ 2 files changed, 1159 insertions(+) create mode 100644 scratch/bba-ob-core-async/my-async-tests.el create mode 100644 scratch/bba-ob-core-async/my-async-tests.org diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el new file mode 100644 index 000000000..918a4b142 --- /dev/null +++ b/scratch/bba-ob-core-async/my-async-tests.el @@ -0,0 +1,339 @@ +;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*- + +;; Copyright (C) 2024 Bruno BARBIER + +;; Author: Bruno BARBIER +;; Status: Temporary tests. +;; Compatibility: GNU Emacs 30.0.50 +;; +;; This file is NOT part of GNU Emacs. + +;; This program is free software; you can redistribute it and/or +;; modify it under the terms of the GNU General Public License as +;; published by the Free Software Foundation; either version 2 of +;; the License, or (at your option) any later version. + +;; This program is distributed in the hope that it will be +;; useful, but WITHOUT ANY WARRANTY; without even the implied +;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +;; PURPOSE. See the GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public +;; License along with this program; if not, write to the Free +;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, +;; MA 02111-1307 USA + +(require 'cl-lib) +(require 'org) +(require 'org-elib-async) + +(require 'ob-shell) +(require 'ob-python) + +;;; Shells +;; + +;;;; One shell script + +;; Standalone direct asynchronous execution. + +(defun my-shell-babel-schedule (lang body _params handle-feedback) + "Execute the bash script BODY. +Execute the shell script BODY using bash. Use HANDLE-FEEDBACK to report +the outcome (success or failure)." + (unless (equal "bash" lang) + (error "Only for bash")) + (funcall handle-feedback (list :pending "started")) + (org-elib-async-process (list "bash") :input body :callback handle-feedback)) + + +;;;; Asynchronous using ob-shell + +(defun my-org-babel-shell-how-to-execute (body params) + "Return how to execute BODY using a POSIX shell. +Return how to execute, as expected by +`org-elib-async-comint-queue--execution'." + ;; Code mostly extracted from ob-shell, following + ;; `org-babel-execute:shell' and `org-babel-sh-evaluate'. + ;; Results are expected to differ from ob-shell as we follow the + ;; same process for all execution paths: asynchronous or not, with + ;; session or without. + (let* ((session (org-babel-sh-initiate-session + (cdr (assq :session params)))) + (stdin (let ((stdin (cdr (assq :stdin params)))) + (when stdin (org-babel-sh-var-to-string + (org-babel-ref-resolve stdin))))) + (result-params (cdr (assq :result-params params))) + (value-is-exit-status + (or (and + (equal '("replace") result-params) + (not org-babel-shell-results-defaults-to-output)) + (member "value" result-params))) + (cmdline (cdr (assq :cmdline params))) + (shebang (cdr (assq :shebang params))) + (full-body (concat + (org-babel-expand-body:generic + body params (org-babel-variable-assignments:shell params)) + (when value-is-exit-status "\necho $?"))) + (post-process + (lambda (r) + (setq r (org-trim r)) + (org-babel-reassemble-table + (org-babel-result-cond result-params + r + (let ((tmp-file (org-babel-temp-file "sh-"))) + (with-temp-file tmp-file (insert r)) + (org-babel-import-elisp-from-file tmp-file))) + (org-babel-pick-name + (cdr (assq :colname-names params)) (cdr (assq :colnames params))) + (org-babel-pick-name + (cdr (assq :rowname-names params)) (cdr (assq :rownames params)))))) + comint-buffer + finally + to-run) + + (setq comint-buffer + (if session session + ;; No session. We create a temporary one and use 'finally' to + ;; destroy it once we are done. + ;; + ;; FIXME: This session code should be refactored and moved into + ;; ob-core. + (let ((s-buf (org-babel-sh-initiate-session + (generate-new-buffer-name (format "*ob-shell-no-session*"))))) + (setq finally (lambda () + ;; We cannot delete it immediately as we are called from it. + (run-with-idle-timer + 0.1 nil + (lambda () + (when (buffer-live-p s-buf) + (let ((kill-buffer-query-functions nil) + (kill-buffer-hook nil)) + (kill-buffer s-buf))))))) + s-buf))) + + (org-elib-async-comint-queue-init-if-needed comint-buffer) + + (setq to-run + (cond + ((or stdin cmdline) ; external shell script w/STDIN + (let ((script-file (org-babel-temp-file "sh-script-")) + (stdin-file (org-babel-temp-file "sh-stdin-")) + (padline (not (string= "no" (cdr (assq :padline params)))))) + (with-temp-file script-file + (when shebang (insert shebang "\n")) + (when padline (insert "\n")) + (insert full-body)) + (set-file-modes script-file #o755) + (with-temp-file stdin-file (insert (or stdin ""))) + (with-temp-buffer + (with-connection-local-variables + (concat + (mapconcat #'shell-quote-argument + (cons (if shebang (file-local-name script-file) + shell-file-name) + (if shebang (when cmdline (list cmdline)) + (list shell-command-switch + (concat (file-local-name script-file) " " cmdline)))) + " ") + "<" (shell-quote-argument stdin-file)))))) + (session ; session evaluation + full-body) + ;; External shell script, with or without a predefined + ;; shebang. + ((org-string-nw-p shebang) + (let ((script-file (org-babel-temp-file "sh-script-")) + (padline (not (equal "no" (cdr (assq :padline params)))))) + (with-temp-file script-file + (insert shebang "\n") + (when padline (insert "\n")) + (insert full-body)) + (set-file-modes script-file #o755) + (if (file-remote-p script-file) + ;; Run remote script using its local path as COMMAND. + ;; The remote execution is ensured by setting + ;; correct `default-directory'. + (let ((default-directory (file-name-directory script-file))) + (file-local-name script-file) + script-file "")))) + (t + (let ((script-file (org-babel-temp-file "sh-script-"))) + (with-temp-file script-file + (insert full-body)) + (set-file-modes script-file #o755) + (mapconcat #'shell-quote-argument + (list shell-file-name + shell-command-switch + (if (file-remote-p script-file) + (file-local-name script-file) + script-file)) + " "))))) + ;; TODO: How to handle `value-is-exit-status'? + (lambda (&rest q) + (pcase q + (`(:instrs-to-enter) + ;; FIXME: This is wrong. + "export PS1=''; export PS2='';") + (`(:instrs-to-exit)) + (`(:finally) (when finally (funcall finally))) + (`(:instr-to-emit-tag ,tag) (format "printf '%s\\n'" tag)) + (`(:post-process ,r) (when post-process (funcall post-process r))) + (`(:send-instrs-to-session ,code) + (with-current-buffer comint-buffer + (when code + (goto-char (point-max)) + (insert code) (insert "\n") + (comint-send-input nil t)))) + (`(:get-code) to-run) + (`(:get-comint-buffer) comint-buffer) + (_ (error "Unknown query")))))) + + + + + +;;; Python +;; + +;;;; Asynchronous using ob-python + +(defun my-org-babel-python-how-to-execute (body params) + "Return how to execute BODY using python. +Return how to execute, as expected by +`org-elib-async-comint-queue--execution'." + ;; Code mostly extracted from ob-python, following + ;; `org-babel-python-evaluate-session'. + ;; Results are expected to differ from ob-python as we follow the + ;; same process for all execution paths: asynchronous or not, with + ;; session or without. + (let* ((org-babel-python-command + (or (cdr (assq :python params)) + org-babel-python-command)) + (session-key (org-babel-python-initiate-session + (cdr (assq :session params)))) + (graphics-file (and (member "graphics" (assq :result-params params)) + (org-babel-graphical-output-file params))) + (result-params (cdr (assq :result-params params))) + (result-type (cdr (assq :result-type params))) + (results-file (when (eq 'value result-type) + (or graphics-file + (org-babel-temp-file "python-")))) + (return-val (when (eq result-type 'value) + (cdr (assq :return params)))) + (full-body + (concat + (org-babel-expand-body:generic + body params + (org-babel-variable-assignments:python params)) + (when return-val + (format "\n%s" return-val)))) + (post-process + (lambda (r) + (setq r (string-trim r)) + (when (string-prefix-p "Traceback (most recent call last):" r) + (signal 'user-error (list r))) + (when (eq 'value result-type) + (setq r (org-babel-eval-read-file results-file))) + (org-babel-reassemble-table + (org-babel-result-cond result-params + r + (org-babel-python-table-or-string r)) + (org-babel-pick-name (cdr (assq :colname-names params)) + (cdr (assq :colnames params))) + (org-babel-pick-name (cdr (assq :rowname-names params)) + (cdr (assq :rownames params)))))) + (tmp-src-file (org-babel-temp-file "python-")) + (session-body + ;; The real code we evaluate in the session. + (pcase result-type + (`output + (format (string-join + (list "with open('%s') as f:\n" + " exec(compile(f.read(), f.name, 'exec'))\n")) + (org-babel-process-file-name + tmp-src-file 'noquote))) + (`value + ;; FIXME: In this case, any output is an error. + (org-babel-python-format-session-value + tmp-src-file results-file result-params)))) + comint-buffer + finally) + + + (unless session-key + ;; No session. We create a temporary one and use 'finally' to + ;; destroy it once we are done. + ;; + ;; FIXME: This session code should be refactored and moved into + ;; ob-core. + (setq session-key (org-babel-python-initiate-session + ;; We can't use a simple `generate-new-buffer' + ;; due to the earmuffs game. + (org-babel-python-without-earmuffs + (format "*ob-python-no-session-%s*" (org-id-uuid))))) + (setq finally (lambda () + (when-let ((s-buf + (get-buffer (org-babel-python-with-earmuffs session-key)))) + ;; We cannot delete it immediately as we are called from it. + (run-with-idle-timer + 0.1 nil + (lambda () + (when (buffer-live-p s-buf) + (let ((kill-buffer-query-functions nil) + (kill-buffer-hook nil)) + (kill-buffer s-buf))))))))) + + (setq comint-buffer + (get-buffer (org-babel-python-with-earmuffs session-key))) + (org-elib-async-comint-queue-init-if-needed comint-buffer) + (with-temp-file tmp-src-file + (insert (if (and graphics-file (eq result-type 'output)) + (format org-babel-python--output-graphics-wrapper + full-body graphics-file) + full-body))) + + (lambda (&rest q) + (pcase q + (`(:instrs-to-enter) + ;; FIXME: This is wrong. + "import sys; sys.ps1=''; sys.ps2=''") + (`(:instrs-to-exit)) + (`(:finally) (when finally (funcall finally))) + (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag)) + (`(:post-process ,r) (when post-process (funcall post-process r))) + (`(:send-instrs-to-session ,code) + ;; See org-babel-python-send-string + (with-current-buffer comint-buffer + (let ((python-shell-buffer-name + (org-babel-python-without-earmuffs session-key))) + (python-shell-send-string (concat code "\n"))))) + (`(:get-code) session-body) + (`(:get-comint-buffer) comint-buffer) + (_ (error "Unknown query")))))) + + +;;; Org babel 'execute-with'. + + +;;;; Asynchronous +;; +(defun my-org-babel-schedule (lang body params handle-feedback) + "Schedule the execution of BODY according to PARAMS. +This function is called by `org-babel-execute-src-block'. Return a +function that waits and returns the result on success, raise on failure." + (let ((exec (pcase lang + ("python" (my-org-babel-python-how-to-execute body params)) + ("bash" (my-org-babel-shell-how-to-execute body params)) + (_ (error "Not handled (yet): %s" lang))))) + (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback))) + + +;;;; Synchronous +;; +(defun my-org-babel-execute (lang body params) + "Execute Python BODY according to PARAMS. +This function is called by `org-babel-execute-src-block'." + ;; We just start the asynchronous execution, wait for it, and return + ;; the result (or raise the exception). No custom code, and, + ;; synchronous and asynchronous should just mix nicely together. + (funcall (my-org-babel-schedule lang body params nil))) diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org new file mode 100644 index 000000000..08284c470 --- /dev/null +++ b/scratch/bba-ob-core-async/my-async-tests.org @@ -0,0 +1,820 @@ +#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both +* Intro + +An org document with code blocks to help test the proposed patchs. + +See [[*On top of ob-shell][On top of ob-shell]] for asynchronous shell scripts using ob-shell. + +See [[*On top of ob-python][On top of ob-python]] for asynchronous python scripts using ob-python. + +You need to load: + #+begin_src elisp :results silent + (setq-local org-confirm-babel-evaluate nil) + (load-file "my-async-tests.el") + #+end_src + + +Emacs and org versions: + #+begin_src elisp + (mapcar (lambda (sb) (list sb (symbol-value sb))) + '(emacs-version org-version)) + #+end_src + + #+RESULTS: + | emacs-version | 30.0.50 | + | org-version | 9.7-pre | + +Note that we've disabled eval on export: export doesn't know it needs +to wait for asynchronous results. + +* POSIX shells +** A simple bash example + :PROPERTIES: + :header-args:bash: :execute-with my-shell-babel :nasync yes + :END: + +The package `my-async-tests.el' contains the function +`my-shell-babel-schedule' to evaluate shell script asynchronously. + +The header-args properties above request asynchronous execution for +bash (:nasync yes), and, tells ob-core to use the prefix +`my-shell-babel' when looking for functions to evaluate a source +block. Thus, org will delegate execution to `my-shell-babel-schedule'. +We don't have `my-shell-babel-execute', so, in this case, :nasync must +be yes. + +Examples taken from the org mailing list and from worg. + +A simple execution: + #+begin_src bash + date + #+end_src + + #+RESULTS: + : Wed Feb 21 15:56:23 CET 2024 + +A tricky computation takes some time: + #+begin_src bash + sleep 1; date + #+end_src + + #+RESULTS: + : Wed Feb 21 15:56:25 CET 2024 + +An example of a failure: + #+begin_src bash + sleepdd 1; false + #+end_src + + #+RESULTS: + +** On top of ob-shell + :PROPERTIES: + :header-args:bash: :execute-with my-org-babel :nasync yes + :header-args:bash+: :session sh-async + :END: + + #+begin_src bash + sleep 1; date + #+end_src + + #+RESULTS: + : Wed Feb 21 15:56:42 CET 2024 + + + #+begin_src bash :results output :session *test* :nasync yes + cd /tmp + echo "hello world" + #+end_src + + #+RESULTS: + : hello world + + #+begin_src bash :results output + # comment + # comment + #+end_src + + #+RESULTS: + + #+begin_src bash :results output + # print message + echo \"hello world\" + #+end_src + + #+RESULTS: + : "hello world" + + #+begin_src bash :results output + echo "hello" + echo "world" + #+end_src + + #+RESULTS: + : hello + : world + + + #+begin_src bash :results output + echo PID: "$$" + #+end_src + + #+RESULTS: + : PID: 22212 + + #+begin_src bash :results output + echo PID: "$$" + #+end_src + + #+RESULTS: + : PID: 22212 + + + #+begin_src bash :results output :session shared + echo PID: "$$" + X=5 + #+end_src + + #+RESULTS: + : PID: 22218 + + #+begin_src bash :results output :session shared + echo PID: "$$" + echo X was set to "$X" + #+end_src + + #+RESULTS: + : PID: 22218 + : X was set to 5 + + #+begin_src bash :nasync yes :results value scalar + echo "Execute session blocks in the background" + sleep 3 + echo "Using the :async header" + #+end_src + + #+RESULTS: + : Execute session blocks in the background + : Using the :async header + : 0 + + + + #+name: their-os + Linux + + + #+begin_src bash :results output :shebang #!/usr/bin/env bash :stdin their-os :cmdline RMS :tangle ask_for_os.sh + + # call as ./ask_for_os.sh NAME, where NAME is who to ask + + if [ -z "$1" ]; then + asked="$USER" + else + asked="$1" + fi + + echo Hi, "$asked"! What operating system are you using? + read my_os + + if [ "$asked" = "RMS" ]; then + echo You\'re using GNU/"$my_os"! + elif [ "$asked" = "Linus" ]; then + echo You\'re using "$my_os"! + else + echo You\'re using `uname -o`! + fi + #+end_src + + #+RESULTS: + : Hi, RMS! What operating system are you using? + : You're using GNU/Linux! + + + + #+begin_src bash + declare -a array + + m=4 + n=3 + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + a[${i},${j}]=$RANDOM + done + done + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + echo -ne "${a[${i},${j}]}\t" + done + echo + done + #+end_src + + #+RESULTS: + | 7592 | 13920 | 4911 | + | 7592 | 13920 | 4911 | + | 7592 | 13920 | 4911 | + | 7592 | 13920 | 4911 | + + + #+begin_src bash :results list + declare -a array + + m=4 + n=3 + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + a[${i},${j}]=$RANDOM + done + done + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + echo -ne "${a[${i},${j}]}\t" + done + echo + done + #+end_src + + #+RESULTS: + - 17593 + 384 + 1439 + - 17593 + 384 + 1439 + - 17593 + 384 + 1439 + - 17593 + 384 + 1439 + + #+begin_src bash :results file :file my_output.txt + declare -a array + + m=4 + n=3 + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + a[${i},${j}]=$RANDOM + done + done + for ((i=0; i<m; i++)) + do + for ((j=0; j<n; j++)) + do + echo -ne "${a[${i},${j}]}\t" + done + echo + done + #+end_src + + #+RESULTS: + [[file:my_output.txt]] + + + #+begin_src bash :results output + cat my_output.txt + #+end_src + + #+RESULTS: + : 30132 16194 19934 + : 30132 16194 19934 + : 30132 16194 19934 + : 30132 16194 19934 + + + #+begin_src bash :results output :dir /ssh:phone: :session none + if [ ! -e "foo_file" ]; + then + echo "foo" > foo_file + echo "Created foo_file" + else + echo "foo_file already exists!" + fi + #+end_src + + #+RESULTS: + : foo_file already exists! + + + #+begin_src bash :results output :dir /ssh:phone: :session *remote* + if [ ! -e "foo_file" ]; + then + echo "foo" > foo_file + echo "Created foo_file" + else + echo "foo_file already exists!" + fi + #+end_src + + #+RESULTS: + : Created foo_file + + + #+RESULTS: + + + #+begin_src bash :results none :session *my-session* + X=1 + #+end_src + + #+RESULTS: + + #+begin_src bash :results output :session *my-session* + echo X was set to "$X" + #+end_src + + #+RESULTS: + : X was set to 1 + + #+begin_src bash :results output :session *another-session* + echo X was set to "$X" + #+end_src + + #+RESULTS: + : X was set to + + #+RESULTS: + + + #+begin_src bash :results output + echo "Hello, world!" + sleep 3 + echo "Good-bye, cruel World..." + #+end_src + + #+RESULTS: + : Hello, world! + : Good-bye, cruel World... + + + + #+begin_src bash :var by_two=0 x=3 :session none + if [ "$by_two" = "0" ]; then + echo $(($x * 2)) + else + echo $(($x * 3)) + fi + #+end_src + + #+RESULTS: + : 6 + + + #+begin_src bash :results output :var arr='("apple" "banana" "cherry") + echo The first element is... + echo \"${arr[1]}\" + #+end_src + + #+RESULTS: + : The first element is... + : "banana" + +*** TODO Doesn't work yet: asynchronous that depends from other blocks + #+name: multiply_by_2 + #+begin_src bash :var data="" :results output + echo $(($data * 2)) + #+end_src + + #+RESULTS: multiply_by_2 + : bash: * 2: syntax error: operand expected (error token is "* 2") + + #+begin_src bash :post multiply_by_2(data=*this*) + echo 3 + #+end_src + + #+results: + + +* On top of ob-python + :PROPERTIES: + :header-args:python: :execute-with my-org-babel :nasync yes + :header-args:python+: :session py-async + :END: + +Used =header-args= properties: + - =:execute-with my-org-babel=: look for functions with the prefix `my-org-babel' to execute + blocks (for the asynchronous case use + `my-org-babel-schedule', and, for the synchronous case + `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]]. + + - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule'). + + - =:session py-async= by default, use a session named "py-async". + +** basic examples +*** async with a session +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + + #+RESULTS: + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708515666.8s | 1708515667.8s | 1.0s | + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +*** async with no session + :PROPERTIES: + :header-args:python+: :session none + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + + #+RESULTS: + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708083470.9s | 1708083471.9s | 1.0s | + +Yes, it failed, as expected. "import time" was done in its own +temporary session. The old result is preserved; the error is display +as an overlay. Click on it to get more info about the error. + + +Let's fix it, adding the import line: + #+begin_src python + import time + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708515915.0s | 1708515916.0s | 1.0s | + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + + +*** sync with a session + :PROPERTIES: + :header-args:python+: :session py-sync-session :nasync no + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708102997.5s | 1708102998.5s | 1.0s | + + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +*** sync with no session + :PROPERTIES: + :header-args:python+: :session none :nasync no + :END: + +A very simple test: + #+begin_src python + 2+3 + #+end_src + + #+RESULTS: + : 5 + +Let's import the module time in our session. + #+begin_src python :results silent + import time + #+end_src + +(Yes, =:results silent= needs some work.) + + + +A table that requires some time to compute: + #+begin_src python + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708083470.9s | 1708083471.9s | 1.0s | + +Yes, that fails (no session), displaying the details in a popup. Let's +fix it: + #+begin_src python + import time + start = time.time() + time.sleep(1) + end = time.time() + ["%.1fs" % t for t in [start, end, end-start]] + #+end_src + + #+RESULTS: + | 1708103039.0s | 1708103040.0s | 1.0s | + + + +An error (click on the error , <mouse-1>, to see the details): + #+begin_src python + 2/0 + #+end_src + + #+RESULTS: + + +** worg examples + +Let's import matplotlib in our session. + + #+begin_src python + import matplotlib + import matplotlib.pyplot as plt + #+end_src + + #+RESULTS: + : None + +A figure in a PDF, asynchronous case. + #+begin_src python :results file link + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + + fname = 'myfig-async.pdf' + plt.savefig(fname) + fname # return this to org-mode + #+end_src + + #+RESULTS: + [[file:myfig-async.pdf]] + + +A figure in a PDF, synchronous case. + #+begin_src python :results file link :nasync no + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + + fname = 'myfig-sync.pdf' + plt.savefig(fname) + fname # return this to org-mode + #+end_src + + #+RESULTS: + [[file:myfig-sync.pdf]] + + + +A PNG figure, asynchronous case. + #+begin_src python :results graphics file output :file boxplot.png + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + fig + #+end_src + + #+RESULTS: + [[file:boxplot.png]] + +Same, but using the =:return= keyword. + #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + #+end_src + + #+RESULTS: + [[file:boxplot.png]] + +Same, asynchronous but without a session this time. + #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none + import matplotlib + import matplotlib.pyplot as plt + fig=plt.figure(figsize=(3,2)) + plt.plot([1,3,2]) + fig.tight_layout() + #+end_src + + #+RESULTS: + [[file:boxplot-no-sess-a-y.png]] + + +Lists are table, + #+begin_src python + [1,2,3] + #+end_src + + #+RESULTS: + | 1 | 2 | 3 | + +unless requested otherwise. + #+begin_src python :results verbatim + [1,2,3] + #+end_src + + #+RESULTS: + : [1, 2, 3] + + +Dictionaries are tables too. + #+begin_src python :results table + {"a": 1, "b": 2} + #+end_src + + #+RESULTS: + | a | 1 | + | b | 2 | + + +Let's try the example with Panda. + #+begin_src python :results none + import pandas as pd + import numpy as np + #+end_src + + #+RESULTS: + : None + + #+begin_src python :results table + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + +And the synchronous case? + + #+begin_src python :results table :nasync no + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + + + +Without session ? + + #+begin_src python :results table :session none + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + +Right, we need to import the libraries (no session). + + #+begin_src python :results table :session none + import pandas as pd + import numpy as np + pd.DataFrame(np.array([[1,2,3],[4,5,6]]), + columns=['a','b','c']) + #+end_src + + #+RESULTS: + | | a | b | c | + |---+---+---+---| + | 0 | 1 | 2 | 3 | + | 1 | 4 | 5 | 6 | + + +** inline examples + + A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}. + + An other one containing a mistake src_python{2/0} {{{results(=6=)}}} + (click on the error to see the details). + + + Some very slow inline asynchronous computations that all run in + the same session. You need to execute the 3 of them at once. Here + is the first one src_python[:return "\"OK1\""]{import time; + time.sleep(5)} {{{results(=OK1=)}}} and a second one + src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} + {{{results(=OK1 bis=)}}} and the third one src_python[:return + "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}. + + Yes, the previous paragraph is unreadable; it's on purpose, to + check that ob-core can figure it out. + + Let's repeat, in a more readable way, and making the last one + synchronous. + + Some very slow inline computations that all run in the same + session. Here is the first asynchronous one + src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}} + and a second one, asynchronous too: + src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}} + and finally, a third one, synchronous this one: + src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}. + + Note that, once the user executes the last synchronous block, the + user is blocked until the synchronous execution can start + (i.e. all previous asynchronous executions are done) and until + it's done. The display is updated though, to see the asynchronous + progress. -- 2.43.0 ^ permalink raw reply related [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-16 17:52 ` Bruno Barbier 2024-02-18 21:14 ` Matt 2024-02-19 0:15 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm @ 2024-02-19 9:06 ` Ihor Radchenko 2024-02-19 19:47 ` Matt 2024-02-21 16:03 ` Bruno Barbier 2 siblings, 2 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-02-19 9:06 UTC (permalink / raw) To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > Hi Matt, Jack, Ihor, > > Sorry for the late reply. Cleaning the code took me longer than > expected. Thanks for the code! It is a lot more that I expected. I have many questions ;) > The new API itself is more about how to wait for and display one block > result. So, it's not really aware of sessions. I think that it is important to think about sessions. For non-sessions, we may execute the code in parallel, disregarding already running execution. In contrast, for sessions, we need to maintain some queue and wait until previous execution finishes before running next (if multiple session blocks are executed in quick succession). It may also be necessary to provide an UI listing the queued and running execution. Users should be able to stop "stale" processes if they are defunc (consider infinite loop). >> Also interesting, I think it's worth exploring/testing this overlay idea >> out. Does that mean that output is asynchronously printing into the Org >> buffer? It sounds cool but I wonder if it might cause problems while >> trying to edit another part of the buffer. > > Currently, I've limited the progress feedback to fit on one line to > avoid anoying vertical display jumps. When the computation is > successful, the result is inserted as usual, and, that may be annoying; > when it updates a previous result of the same height, it's ok. Else, we > could fold it to stay on one line too (using the overlay), until the > user explicitly request to see it. Let's not worry about "jumps" yet. It is a minor issue that can be easily solved. What is more important is when users, for example, remove the whole subtree containing the place where the execution result is to be inserted. Or when users perform edits at or around that place where the result is to be inserted. Or consider subtree with pending result refiled elsewhere (different file or different place in the same file); or archived. Or consider user running an src block, quickly editing it, and then running again. What should we do then? Should we stop the first running process? > So, here we go. You'll find attach a set of patchs. It works for me with > Emacsc 30.50 and 9.7-pre (from today). I have several general questions: - what if append/prepend to result asynchronously? - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)? - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? Example: try running #+begin_src bash idontexist #+end_src > I didn't check yet that the code and the commits follow all the > guidelines. This is just a preliminary version for feedbacks. > Corrections/critiques are welcome, but don't check the details until I > check them myself. > So, I decided to rewrite the whole thing, taking code from the > synchronous case (following `org-babel-python-evaluate-session'). I > also created a package that contains all the functions that should be > reusable for any language. The patch [1] adds some generic functions to > help dealing with asynchronicity (process, comint, etc.). The patch [2] > shows a new possible way to execute python code blocks, both > synchronously and asynchronously, with or without sessions. You should > just need to open [3] and follow what's written there, and execute the > existing bash and python code blocks. Note that we already have a WIP an asynchronous API in the works. Check out `org-async-call' in https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 If possible, we should reuse it. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 9:06 ` Ihor Radchenko @ 2024-02-19 19:47 ` Matt 2024-02-19 20:10 ` Ihor Radchenko ` (2 more replies) 2024-02-21 16:03 ` Bruno Barbier 1 sibling, 3 replies; 55+ messages in thread From: Matt @ 2024-02-19 19:47 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode ---- On Mon, 19 Feb 2024 01:31:42 +0100 Jack Kamm wrote --- > I don't remember the details, but my past self [1] thought it would be > good to find a way to replace `process-file' with `make-process' in > `org-babel--shell-command-on-region' or `org-babel-eval', and it seems > you are thinking along those lines with `my-org-babel-eval-async'. Hope > you're able to make progress on this and get the improvements into > ob-eval.el eventually. > > [1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t AFAIK, aside from appending "&", =make-process= is the only way. It would probably make sense to continue using =shell= though. If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer? ---- On Mon, 19 Feb 2024 10:03:36 +0100 Ihor Radchenko wrote --- > Note that we already have a WIP an asynchronous API in the works. > Check out `org-async-call' in > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > If possible, we should reuse it. :O - What's the status? - Anything I could help with? - Are there any notes or supplements that go along with it? > I have several general questions: > > - what if append/prepend to result asynchronously? > - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)? > - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? > Example: try running > #+begin_src bash > idontexist > #+end_src Are these open questions for the `org-async-call' implementation? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 19:47 ` Matt @ 2024-02-19 20:10 ` Ihor Radchenko 2024-02-20 8:32 ` Ihor Radchenko 2024-02-20 17:04 ` Jack Kamm 2 siblings, 0 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-02-19 20:10 UTC (permalink / raw) To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode Matt <matt@excalamus.com> writes: > > Note that we already have a WIP an asynchronous API in the works. > > Check out `org-async-call' in > > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > > If possible, we should reuse it. > > :O > > - What's the status? Finalizing. > - Anything I could help with? Check https://github.com/tecosaur/org-latex-preview-todos/issues/ > - Are there any notes or supplements that go along with it? Docstring should detail the API. >> I have several general questions: >> >> - what if append/prepend to result asynchronously? >> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)? >> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? >> Example: try running >> #+begin_src bash >> idontexist >> #+end_src > > Are these open questions for the `org-async-call' implementation? No. These are about the patch. And have nothing to do with `org-async-call', which is just a way to call and queue async processes. `org-async-call' was written for async latex preview, but its interface should be generic enough to be reused for babel. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 19:47 ` Matt 2024-02-19 20:10 ` Ihor Radchenko @ 2024-02-20 8:32 ` Ihor Radchenko 2024-02-20 17:04 ` Jack Kamm 2 siblings, 0 replies; 55+ messages in thread From: Ihor Radchenko @ 2024-02-20 8:32 UTC (permalink / raw) To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode Matt <matt@excalamus.com> writes: > - Anything I could help with? https://github.com/tecosaur/org-latex-preview-todos/issues/21 It looks like the main remaining blocker is the lack of LaTeX snippet export for ox-odt. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 19:47 ` Matt 2024-02-19 20:10 ` Ihor Radchenko 2024-02-20 8:32 ` Ihor Radchenko @ 2024-02-20 17:04 ` Jack Kamm 2 siblings, 0 replies; 55+ messages in thread From: Jack Kamm @ 2024-02-20 17:04 UTC (permalink / raw) To: Matt, Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode Matt <matt@excalamus.com> writes: > AFAIK, aside from appending "&", =make-process= is the only way. It would probably make sense to continue using =shell= though. If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer? Yes, I typically use ob-R and ob-python in this way. Aside from the convenience of interacting directly with the REPL, the comint mode may provide other useful facilities, for example in ob-R the inferior ESS mode provides completion that is aware of the objects in the current session. However I think modifying `org-babel-eval' will mainly be useful to provide async to the nonsession users. Comint-based sessions mainly use functionality from ob-comint.el, not ob-eval.el, and I think that the use cases are different enough that they shouldn't be merged into a single implementation. ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-19 9:06 ` Ihor Radchenko 2024-02-19 19:47 ` Matt @ 2024-02-21 16:03 ` Bruno Barbier 2024-02-23 12:11 ` Ihor Radchenko 1 sibling, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-02-21 16:03 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode Hi Ihor, Ihor Radchenko <yantar92@posteo.net> writes: > > Thanks for the code! > It is a lot more that I expected. Note that only the first 5 patchs are real patchs. The remaining things are just a demo how it could be used. The current async (ob-comint) depends on writing UUIDs in org files, and, that's why I couldn't use it as a demo. I'm thinking about dropping the patch: 'ob-core async: Add org-babel--async tools [2/5]' and just provide an other new keyword (feedbacks-with) so that anybody may plug its own feedback process. > I have many questions ;) Thanks! They are welcome :-) >> The new API itself is more about how to wait for and display one block >> result. So, it's not really aware of sessions. > > I think that it is important to think about sessions. For non-sessions, > we may execute the code in parallel, disregarding already running > execution. In contrast, for sessions, we need to maintain some queue and > wait until previous execution finishes before running next (if multiple > session blocks are executed in quick succession). I agree. My proposed patch leaves this open to further work. My proposed patch allows to have one way, in ob-core, to display progress of asynchronous executions: like not started, 10%, success, failed; and a way to plug in asynchronous executions in ob-core. I've included some demo code just to show how that API could be used. > It may also be necessary to provide an UI listing the queued and running > execution. Users should be able to stop "stale" processes if they are > defunc (consider infinite loop). That looks nice but that may be too limiting. Like it will probably forbid to execute something asynchronously using elisp threads. Anyway, that outside the intended scope of my set of patchs. > What is more important is when users, for example, remove the whole > subtree containing the place where the execution result is to be > inserted. Or when users perform edits at or around that place where the > result is to be inserted. Or consider subtree with pending result > refiled elsewhere (different file or different place in the same file); > or archived. My patch definitely needs to address those, even if only by raising an error. > Or consider user running an src block, quickly editing it, and then > running again. What should we do then? Should we stop the first running > process? To keep things simple and generic, I would just forbid the user to reschedule the task until the previous outcome is available. > I have several general questions: > > - what if append/prepend to result asynchronously? You mean if org is executing several times the same code concurrently? I think we should forbid it. > - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)? "One execution at a given time" will solve this too :-) > - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? In the asynchronous case, I've decided to leave the overlay in place in case of errors, with the error description. Clicking on that error pops up the same kind of popups as in the synchronous case. You can give it a try using examples of errors my demo file: scratch/bba-ob-core-async/my-async-tests.org (Please use the updated set of patchs that I've sent today, else, you will have to fix the required libraries). > Note that we already have a WIP an asynchronous API in the works. > Check out `org-async-call' in > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > If possible, we should reuse it. Thanks. I will have a look at it. Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-21 16:03 ` Bruno Barbier @ 2024-02-23 12:11 ` Ihor Radchenko 2024-02-23 13:24 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-23 12:11 UTC (permalink / raw) To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: > Note that only the first 5 patchs are real patchs. The remaining things > are just a demo how it could be used. The current async (ob-comint) > depends on writing UUIDs in org files, and, that's why I couldn't use it > as a demo. I'm thinking about dropping the patch: > > 'ob-core async: Add org-babel--async tools [2/5]' > > and just provide an other new keyword (feedbacks-with) so that anybody > may plug its own feedback process. May you please clarify if adding the new code block parameter that defines custom execute function is something you want to add to Org mode or just a helper code to demo you main patch? >> Or consider user running an src block, quickly editing it, and then >> running again. What should we do then? Should we stop the first running >> process? > > To keep things simple and generic, I would just forbid the user to > reschedule the task until the previous outcome is available. This may not be as trivial as one might think. Consider #+name: foo #+begin_src bash :async yes ... #+end_src #+results: foo <pending results> Another code block with the same name will write results under #+results: foo #+name: foo #+begin_src bash :async yes ... #+end_src We can currently have multiple code blocks writing to the same place. And async execution should not only check is the current src block is scheduled, but also whether we are attempting to write to a place where no other running block is scheduled to write. >> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? > > In the asynchronous case, I've decided to leave the overlay in place > in case of errors, with the error description. Clicking on that error > pops up the same kind of popups as in the synchronous case. Error buffer does not necessarily appear on failure. When the code leads to process writing to stderr, we also display error buffer. Even if the process exits with 0 code. Also, your code currently creates overlays unconditionally. However, when we have :results silent or :results none, Org babel must not modify buffer. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-23 12:11 ` Ihor Radchenko @ 2024-02-23 13:24 ` Bruno Barbier 2024-02-24 11:59 ` Ihor Radchenko 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-02-23 13:24 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> Note that only the first 5 patchs are real patchs. The remaining things >> are just a demo how it could be used. The current async (ob-comint) >> depends on writing UUIDs in org files, and, that's why I couldn't use it >> as a demo. I'm thinking about dropping the patch: >> >> 'ob-core async: Add org-babel--async tools [2/5]' >> >> and just provide an other new keyword (feedbacks-with) so that anybody >> may plug its own feedback process. > > May you please clarify if adding the new code block parameter that > defines custom execute function is something you want to add to Org mode > or just a helper code to demo you main patch? Yes. (I thought about adding it in a separate request; but that would be simpler to include it.) >>> Or consider user running an src block, quickly editing it, and then >>> running again. What should we do then? Should we stop the first running >>> process? >> >> To keep things simple and generic, I would just forbid the user to >> reschedule the task until the previous outcome is available. > > This may not be as trivial as one might think. > Consider > > #+name: foo > #+begin_src bash :async yes > ... > #+end_src > > #+results: foo > <pending results> > > Another code block with the same name will write results under > #+results: foo > #+name: foo > #+begin_src bash :async yes > ... > #+end_src > We can currently have multiple code blocks writing to the same place. > And async execution should not only check is the current src block is > scheduled, but also whether we are attempting to write to a place where > no other running block is scheduled to write. Today, the asynchronous execution is attached to the result; the source code itself is not locked and stays editable. With the current implementation, trying your example, the second execution fails with: Error running timer: (user-error "Cannot modify an area being updated") So, it's already OK I guess. I should improve the behavior (to fail before launching the second execution). But, the current design should work in that case. > >>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications? >> >> In the asynchronous case, I've decided to leave the overlay in place >> in case of errors, with the error description. Clicking on that error >> pops up the same kind of popups as in the synchronous case. > > Error buffer does not necessarily appear on failure. When the code leads > to process writing to stderr, we also display error buffer. Even if the > process exits with 0 code. Got it. The current design adds the popup on failure; I should make it more flexible to allow to configure a popup on success too. > Also, your code currently creates overlays unconditionally. > However, when we have :results silent or :results none, Org babel must > not modify buffer. Yes. I'll fix it (and that one was in my TODO list :-)) But if the execution fails, I guess I'll need to provide some feedbacks anyway, but I don't know yet how (asynchronous popups are not an option, modifying the file neither). Thanks for your review and questions, Bruno > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-23 13:24 ` Bruno Barbier @ 2024-02-24 11:59 ` Ihor Radchenko 2024-02-24 16:42 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Ihor Radchenko @ 2024-02-24 11:59 UTC (permalink / raw) To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode Bruno Barbier <brubar.cs@gmail.com> writes: >> May you please clarify if adding the new code block parameter that >> defines custom execute function is something you want to add to Org mode >> or just a helper code to demo you main patch? > > Yes. (I thought about adding it in a separate request; but that would be > simpler to include it.) I have doubts about how useful such parameter would be for _users_. It is certainly useful for developers of new babel backends, but I do not see how users will use it. I'd prefer to discuss this in a separate thread. >> Error buffer does not necessarily appear on failure. When the code leads >> to process writing to stderr, we also display error buffer. Even if the >> process exits with 0 code. > > Got it. The current design adds the popup on failure; I should make it > more flexible to allow to configure a popup on success too. By popup, do you mean an overlay where clicking on it will raise the usual `org-babel-eval-error-notify' buffer? If yes, what happens if a user executes a code block, it fails, and the user executes it again without looking at the error? (Second execution may be triggered from a different place, indirectly, via noweb ref or similar). > But if the execution fails, I guess I'll need to provide some feedbacks > anyway, but I don't know yet how (asynchronous popups are not an option, > modifying the file neither). We need something more than just an overlay. Maybe some kind of babel execute history (in tabulated-list-mode buffer), keeping information about execution stats and stderr data. We may provide a mode-line or fringe indicator that will warn user if something went wrong with recent executions. Something akin compile-mode mode line indicator listing errors and warnings during compilation. (Handling asynchronous messages is actually a complex topic. It has been previously discussed on emacs-devel as well. See <https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>) -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-24 11:59 ` Ihor Radchenko @ 2024-02-24 16:42 ` Bruno Barbier 2024-02-24 19:54 ` Matt 0 siblings, 1 reply; 55+ messages in thread From: Bruno Barbier @ 2024-02-24 16:42 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >>> May you please clarify if adding the new code block parameter that >>> defines custom execute function is something you want to add to Org mode >>> or just a helper code to demo you main patch? >> >> Yes. (I thought about adding it in a separate request; but that would be >> simpler to include it.) > > I have doubts about how useful such parameter would be for _users_. It > is certainly useful for developers of new babel backends, but I do not > see how users will use it. > > I'd prefer to discuss this in a separate thread. Sure. I'll keep it for an other thread. >>> Error buffer does not necessarily appear on failure. When the code leads >>> to process writing to stderr, we also display error buffer. Even if the >>> process exits with 0 code. >> >> Got it. The current design adds the popup on failure; I should make it >> more flexible to allow to configure a popup on success too. > > By popup, do you mean an overlay where clicking on it will raise the > usual `org-babel-eval-error-notify' buffer? > If yes, what happens if a user executes a code block, it fails, and the > user executes it again without looking at the error? (Second execution > may be triggered from a different place, indirectly, via noweb ref or > similar). Current behavior is to remove the error, and, then re-execute: the previous errors are gone. My guess is that this could be improve later, without breaking the API. >> But if the execution fails, I guess I'll need to provide some feedbacks >> anyway, but I don't know yet how (asynchronous popups are not an option, >> modifying the file neither). > > We need something more than just an overlay. Maybe some kind of babel > execute history (in tabulated-list-mode buffer), keeping information > about execution stats and stderr data. > > We may provide a mode-line or fringe indicator that will warn user if > something went wrong with recent executions. Something akin compile-mode > mode line indicator listing errors and warnings during compilation. I was thinking about using a fringe indicator, for warnings on success (as I'm removing all overlays on success). Keeping a buffer of all asynchronous executions looks like a good idea. I'll think about it. > (Handling asynchronous messages is actually a complex topic. It has been > previously discussed on emacs-devel as well. See > <https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>) I'll look at this. Thanks! I'll publish a branch soon; it will be a major rewrite of my current proposal. It should be less confusing and, I hope, address some of your comments. Thanks again for your questions and feedbacks, Bruno > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-24 16:42 ` Bruno Barbier @ 2024-02-24 19:54 ` Matt 2024-02-28 10:21 ` Bruno Barbier 0 siblings, 1 reply; 55+ messages in thread From: Matt @ 2024-02-24 19:54 UTC (permalink / raw) To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode ---- On Sat, 24 Feb 2024 17:42:54 +0100 Bruno Barbier > I'll publish a branch soon; it will be a major rewrite of my current > proposal. It should be less confusing and, I hope, address some of your > comments. > > Thanks again for your questions and feedbacks, You're welcome and thanks for sharing your ideas. Any lack of comments from me recently is just limited time and trying to focus on maintenance. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) 2024-02-24 19:54 ` Matt @ 2024-02-28 10:21 ` Bruno Barbier 0 siblings, 0 replies; 55+ messages in thread From: Bruno Barbier @ 2024-02-28 10:21 UTC (permalink / raw) To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode Hi Matt, Matt <matt@excalamus.com> writes: > ---- On Sat, 24 Feb 2024 17:42:54 +0100 Bruno Barbier > > > I'll publish a branch soon; it will be a major rewrite of my current > > proposal. It should be less confusing and, I hope, address some of your > > comments. > > > > Thanks again for your questions and feedbacks, > > You're welcome and thanks for sharing your ideas. > > Any lack of comments from me recently is just limited time and trying to focus on maintenance. Thanks for letting me know. No worries. Take care. Bruno ^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] 2024-02-05 14:29 ` Ihor Radchenko 2024-02-06 19:24 ` Bruno Barbier @ 2024-02-08 3:26 ` Jack Kamm 1 sibling, 0 replies; 55+ messages in thread From: Jack Kamm @ 2024-02-08 3:26 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode Ihor Radchenko <yantar92@posteo.net> writes: >> I agree that it would be good to redesign it, but am not sure where to >> start. > > For example, > > 1. Change `org-babel-comint-async-register' to return UUID and to store > PARAMS as passed by the backend (current approach with PARAMS being > derived from src blocks prevents backends to transform src block > PARAMS dynamically). > 2. Change `org-babel-insert-result' to handle :async t specially, > inserting something reliable, like #+async: <UUID> in place of result > without performing extra transformations. > 3. Change `org-babel-insert-result' to accept an internal parameter > that will make it replace #+async: <UUID> keyword rather than perform > normal result insertion. > 4. Change `org-babel-comint-async-filter' to use the previously passed > PARAMS, remove :async t from them, and arrange the call to > `org-babel-insert-result' to replace the #+async: <UUID> keyword. That all sounds reasonable...if you work on this, let me know if you want any help with testing. ^ permalink raw reply [flat|nested] 55+ messages in thread
end of thread, other threads:[~2024-04-20 10:07 UTC | newest]
Thread overview: 55+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-01 11:58 [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Ihor Radchenko
2024-02-01 14:56 ` Bruno Barbier
2024-02-03 1:30 ` Jack Kamm
2024-02-04 15:07 ` Ihor Radchenko
2024-02-05 1:37 ` Jack Kamm
2024-02-05 14:29 ` Ihor Radchenko
2024-02-06 19:24 ` Bruno Barbier
2024-02-07 16:19 ` Ihor Radchenko
2024-02-07 17:40 ` Bruno Barbier
2024-02-08 3:21 ` Jack Kamm
2024-02-15 20:02 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt
2024-02-16 17:52 ` Bruno Barbier
2024-02-18 21:14 ` Matt
2024-02-19 0:31 ` Jack Kamm
2024-02-20 10:28 ` Ihor Radchenko
2024-02-20 10:46 ` tomas
2024-02-20 11:00 ` Ihor Radchenko
2024-02-20 11:03 ` tomas
2024-02-21 15:27 ` Bruno Barbier
[not found] ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
2024-02-28 10:18 ` Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) Bruno Barbier
2024-03-02 10:03 ` Ihor Radchenko
2024-03-02 10:57 ` Bruno Barbier
2024-03-02 11:13 ` Ihor Radchenko
2024-03-02 18:06 ` Bruno Barbier
[not found] ` <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b>
2024-03-07 17:08 ` Bruno Barbier
2024-03-07 18:29 ` Ihor Radchenko
2024-03-08 14:19 ` Bruno Barbier
2024-03-13 9:48 ` Ihor Radchenko
2024-03-19 9:33 ` Bruno Barbier
2024-03-20 10:23 ` Ihor Radchenko
2024-03-21 10:06 ` Bruno Barbier
2024-03-21 12:15 ` Ihor Radchenko
2024-03-25 17:46 ` Bruno Barbier
2024-03-27 11:29 ` Ihor Radchenko
2024-03-30 22:53 ` Rudolf Adamkovič
2024-04-04 16:35 ` Bruno Barbier
2024-04-04 16:33 ` Bruno Barbier
2024-04-11 11:44 ` Ihor Radchenko
2024-04-19 11:23 ` Bruno Barbier
2024-04-20 10:07 ` Ihor Radchenko
2024-02-19 0:15 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
2024-02-21 15:43 ` Bruno Barbier
2024-02-19 9:06 ` Ihor Radchenko
2024-02-19 19:47 ` Matt
2024-02-19 20:10 ` Ihor Radchenko
2024-02-20 8:32 ` Ihor Radchenko
2024-02-20 17:04 ` Jack Kamm
2024-02-21 16:03 ` Bruno Barbier
2024-02-23 12:11 ` Ihor Radchenko
2024-02-23 13:24 ` Bruno Barbier
2024-02-24 11:59 ` Ihor Radchenko
2024-02-24 16:42 ` Bruno Barbier
2024-02-24 19:54 ` Matt
2024-02-28 10:21 ` Bruno Barbier
2024-02-08 3:26 ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).