emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
@ 2024-02-01 11:58 Ihor Radchenko
  2024-02-01 14:56 ` Bruno Barbier
  0 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-01 11:58 UTC (permalink / raw)
  To: emacs-orgmode, Jack Kamm

This is a followup for another bug report
https://orgmode.org/list/87plxg95u4.fsf@gmail.com

Using the MWE from that report I am seeing a different bug and it is not
very clear for me what is causing it.

1. When I create /tmp/bug.org with the following contents:

    #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
    import matplotlib.pyplot as plt
    plt.figure(figsize=(1, 1))
    plt.plot([1, 2])
    figname = 'fig.svg'
    plt.savefig(figname)
    #+end_src

2. Go to Org git folder
3. make repro REPRO_ARGS="-l ob-python /tmp/bug.org"
4. Evaluate the code block
5. I get

    #+RESULTS:
    [[file:babel-GJy87v/python-xr8wq9]]

    rather than expected [[file:otherdir/fig.svg]]

I use matplotlib 3.8.2-r1.
The contents of Python REPL is suspicious:

Python 3.11.7 (main, Jan  8 2024, 18:11:18) [GCC 13.2.1 20231216] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
ob_comint_async_python_file_/tmp/babel-s8nwvB/python-NmMFRv
>>> 

This is most likely something about my current system setup - I can
reproduce with other Org mode and Emacs versions. But I have no clue
what is the cause.

Emacs  : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0)
 of 2024-01-30
Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-01 11:58 [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Ihor Radchenko
@ 2024-02-01 14:56 ` Bruno Barbier
  2024-02-03  1:30   ` Jack Kamm
  0 siblings, 1 reply; 35+ messages in thread
From: Bruno Barbier @ 2024-02-01 14:56 UTC (permalink / raw)
  To: Ihor Radchenko, emacs-orgmode, Jack Kamm


Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

>
> This is most likely something about my current system setup - I can
> reproduce with other Org mode and Emacs versions. But I have no clue
> what is the cause.

I'm getting the same as you with your MWE.
   
The tag, used by ob-comint async, is:

   "/tmp/babel-zqh04P/python-GL5N5d"
   
but, in "/tmp/bug.org" it becomes:

   "babel-zqh04P/python-GL5N5d"

(`org-babel-result-to-file' transformed it into a simpler relative
path).

The filter `org-babel-comint-async-filter' cannot spot it, because
it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw".

Here is how to reproduce:
   #+begin_src elisp :results table
     (let* ((tag "/tmp/babel-zqh04P/python-tXsdFw")
            (repro 
             (lambda (fn)
               (let ((lnk 
                      (with-temp-buffer
                        (org-mode)
                        (let* ((default-directory "/tmp")
                               (buffer-file-name  fn)
                               (cbuf (clone-indirect-buffer "tmp" nil)))
                          (with-current-buffer cbuf
                            (org-babel-result-to-file tag))))))
                 (list fn
                       (not (eq nil (string-match-p (regexp-quote tag) lnk)))
                       lnk)))))
       (cons (list "Filename" "string-match-p" "Org link")
             (cons 'hline
                   (mapcar repro (list "/tmp/bug.org" 
                                       "/somewhere/else/bug.org")))))
   #+end_src

   #+RESULTS:
   | Filename                | match-p | Org link                                      |
   |-------------------------+---------+-----------------------------------------------|
   | /tmp/bug.org            | nil     | [[file:babel-zqh04P/python-tXsdFw]]           |
   | /somewhere/else/bug.org | t       | [[file:../../tmp/babel-zqh04P/python-tXsdFw]] |


I don't know what a proper fix would be though.
   
Hoping this help,

Bruno


>
> Emacs  : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0)
>  of 2024-01-30
> Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-01 14:56 ` Bruno Barbier
@ 2024-02-03  1:30   ` Jack Kamm
  2024-02-04 15:07     ` Ihor Radchenko
  0 siblings, 1 reply; 35+ messages in thread
From: Jack Kamm @ 2024-02-03  1:30 UTC (permalink / raw)
  To: Bruno Barbier, Ihor Radchenko, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Ihor,
>
> Ihor Radchenko <yantar92@posteo.net> writes:
>
>>
>> This is most likely something about my current system setup - I can
>> reproduce with other Org mode and Emacs versions. But I have no clue
>> what is the cause.
>
> I'm getting the same as you with your MWE.
>    
> The tag, used by ob-comint async, is:
>
>    "/tmp/babel-zqh04P/python-GL5N5d"
>    
> but, in "/tmp/bug.org" it becomes:
>
>    "babel-zqh04P/python-GL5N5d"
>
> (`org-babel-result-to-file' transformed it into a simpler relative
> path).
>
> The filter `org-babel-comint-async-filter' cannot spot it, because
> it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw".

Thanks, I believe your diagnosis is correct, and can confirm the bug
occurs when the Org file is in /tmp. The problem goes away when
`org-link-file-path-type' is set to absolute.

I think the correct solution would be for `org-babel-insert-result' to
not insert file results (or any other special results) for async session
blocks.  Perhaps in this case, `org-babel-insert-result' could return a
new result type, named "async", "future", or similar.


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-03  1:30   ` Jack Kamm
@ 2024-02-04 15:07     ` Ihor Radchenko
  2024-02-05  1:37       ` Jack Kamm
  0 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-04 15:07 UTC (permalink / raw)
  To: Jack Kamm; +Cc: Bruno Barbier, emacs-orgmode

Jack Kamm <jackkamm@gmail.com> writes:

> I think the correct solution would be for `org-babel-insert-result' to
> not insert file results (or any other special results) for async session
> blocks.  Perhaps in this case, `org-babel-insert-result' could return a
> new result type, named "async", "future", or similar.

That will not work.
`org-babel-comint-async-filter' expects a unique result to be inserted
into Org buffer, so that it can be located, and replaced by the async
evaluation output.

So, we have to insert some kind of indicator for async result.

Of course, the existing scheme of coordination between
`org-babel-insert-result' and `org-babel-comint-async-filter' is
erroneous:

1. We have the problem with :results file value discussed here
2. We have a worse problem with :results file :file foo when the result
   may not be unique
3. We have :results append/prepend completely broken because
   `org-babel-comint-async-filter' simply calls
   `org-babel-insert-result' implicitly assuming that the existing
   indicator is replaced.

The whole thing should be re-designed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-04 15:07     ` Ihor Radchenko
@ 2024-02-05  1:37       ` Jack Kamm
  2024-02-05 14:29         ` Ihor Radchenko
  0 siblings, 1 reply; 35+ messages in thread
From: Jack Kamm @ 2024-02-05  1:37 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> I think the correct solution would be for `org-babel-insert-result' to
>> not insert file results (or any other special results) for async session
>> blocks.  Perhaps in this case, `org-babel-insert-result' could return a
>> new result type, named "async", "future", or similar.
>
> That will not work.
> `org-babel-comint-async-filter' expects a unique result to be inserted
> into Org buffer, so that it can be located, and replaced by the async
> evaluation output.
>
> So, we have to insert some kind of indicator for async result.

I meant that we could return something like "async:uuid-abcd-1234" or
"async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could
still find the result.

> Of course, the existing scheme of coordination between
> `org-babel-insert-result' and `org-babel-comint-async-filter' is
> erroneous:
>
> 1. We have the problem with :results file value discussed here
> 2. We have a worse problem with :results file :file foo when the result
>    may not be unique
> 3. We have :results append/prepend completely broken because
>    `org-babel-comint-async-filter' simply calls
>    `org-babel-insert-result' implicitly assuming that the existing
>    indicator is replaced.
>
> The whole thing should be re-designed.

I agree that it would be good to redesign it, but am not sure where to
start.

A bit of a tangent, but if you are thinking about re-designing this,
then it may be worth considering ob-jupyter's implementation of async
sessions [1]. In particular, I believe it leaves a marker [2] where it
needs to insert the future result. I don't remember the details,
e.g. how it keeps track of which marker is for which result. But it
seems neat, and might work better for some cases such as
appending/prepending results.

[1] https://github.com/emacs-jupyter/jupyter
[2] https://www.gnu.org/software/emacs/manual/html_node/elisp/Markers.html


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-05  1:37       ` Jack Kamm
@ 2024-02-05 14:29         ` Ihor Radchenko
  2024-02-06 19:24           ` Bruno Barbier
  2024-02-08  3:26           ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm
  0 siblings, 2 replies; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-05 14:29 UTC (permalink / raw)
  To: Jack Kamm; +Cc: Bruno Barbier, emacs-orgmode

Jack Kamm <jackkamm@gmail.com> writes:

>> So, we have to insert some kind of indicator for async result.
>
> I meant that we could return something like "async:uuid-abcd-1234" or
> "async:/path/to/tmpfile", so that `org-babel-comint-async-filter' could
> still find the result.

Of course, we could. But that would not solve all the possible problems.
In particular, when header arguments tell Org babel to write result to a
file, returning UUID will still not work.

>> Of course, the existing scheme of coordination between
>> `org-babel-insert-result' and `org-babel-comint-async-filter' is
>> erroneous:
>>
>> 1. We have the problem with :results file value discussed here
>> 2. We have a worse problem with :results file :file foo when the result
>>    may not be unique
>> 3. We have :results append/prepend completely broken because
>>    `org-babel-comint-async-filter' simply calls
>>    `org-babel-insert-result' implicitly assuming that the existing
>>    indicator is replaced.
>>
>> The whole thing should be re-designed.
> ...
> A bit of a tangent, but if you are thinking about re-designing this,
> then it may be worth considering ob-jupyter's implementation of async
> sessions [1]. In particular, I believe it leaves a marker [2] where it
> needs to insert the future result. I don't remember the details,
> e.g. how it keeps track of which marker is for which result. But it
> seems neat, and might work better for some cases such as
> appending/prepending results.

Markers are not as reliable as you think. If text around marker gets
deleted, the marker will still exist potentially causing the async
result to be inserted in the middle of unexpected place.

Having an actual text indicator is more reliable - if user removes it
before the async evaluation is completed, we will not write anything
unexpected.

Also,
https://github.com/emacs-jupyter/jupyter/blob/master/ob-jupyter.el#L540

        ;; KLUDGE: Remove the file result-parameter so that
        ;; `org-babel-insert-result' doesn't attempt to handle it while
        ;; async results are pending.  Do the same in the synchronous
        ;; case, but not if link or graphics are also result-parameters,
        ;; only in Org >= 9.2, since those in combination with file mean
        ;; to interpret the result as a file link, a useful meaning that
        ;; doesn't interfere with Jupyter style result insertion.

They also had to work around the same problem.

> I agree that it would be good to redesign it, but am not sure where to
> start.

For example,

1. Change `org-babel-comint-async-register' to return UUID and to store
   PARAMS as passed by the backend (current approach with PARAMS being
   derived from src blocks prevents backends to transform src block
   PARAMS dynamically).
2. Change `org-babel-insert-result' to handle :async t specially,
   inserting something reliable, like #+async: <UUID> in place of result
   without performing extra transformations.
3. Change `org-babel-insert-result' to accept an internal parameter
   that will make it replace #+async: <UUID> keyword rather than perform
   normal result insertion.
4. Change `org-babel-comint-async-filter' to use the previously passed
   PARAMS, remove :async t from them, and arrange the call to
   `org-babel-insert-result' to replace the #+async: <UUID> keyword.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-05 14:29         ` Ihor Radchenko
@ 2024-02-06 19:24           ` Bruno Barbier
  2024-02-07 16:19             ` Ihor Radchenko
                               ` (2 more replies)
  2024-02-08  3:26           ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm
  1 sibling, 3 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-06 19:24 UTC (permalink / raw)
  To: Ihor Radchenko, Jack Kamm; +Cc: emacs-orgmode


Hi Ihor, Jack,

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>

>> I agree that it would be good to redesign it, but am not sure where to
>> start.
>
> For example,
>
> 1. Change `org-babel-comint-async-register' to return UUID and to store
>    PARAMS as passed by the backend (current approach with PARAMS being
>    derived from src blocks prevents backends to transform src block
>    PARAMS dynamically).
> 2. Change `org-babel-insert-result' to handle :async t specially,
>    inserting something reliable, like #+async: <UUID> in place of result
>    without performing extra transformations.
> 3. Change `org-babel-insert-result' to accept an internal parameter
>    that will make it replace #+async: <UUID> keyword rather than perform
>    normal result insertion.
> 4. Change `org-babel-comint-async-filter' to use the previously passed
>    PARAMS, remove :async t from them, and arrange the call to
>    `org-babel-insert-result' to replace the #+async: <UUID> keyword.
>

FWIW, I've been trying to use asynchronous blocks for everything, not
only the source blocks that are based on the comint mode.  I think it
would be good if ob-core itself could provide an asynchronous API.  I've
modified my Org so that it does have such an API.  This is work in
progress; let me describe it.

I've modified ob-core itself to allow asynchronicity.  In the
asynchrosous case, instead of calling:

  (org-babel-execute:LANG body params)

I'm calling:

  (org-babel-schedule:LANG body params handle-result)

where `org-babel-schedule:LANG' is in charge of calling `handle-result'
with the result (or the error) when it is known; `handle-result' takes
care to call `org-babel-insert-result' at the correct place (and
`org-babel-insert-result' is only called with a real result).

While the execution is pending, I'm using the same technique that Org is
using when a source block is being edited: the result is left untouched,
but below an overlay.  The overlay is used to know where to insert the
result and to display the status/progress of the execution.  If the file
is closed and the execution fails, nothing is lost, the old result is
still available.

If that technique looks safe enough and interesting, I can prepare a set
of patches so that we can discuss it further and, maybe, add it in Org.

Let me know,
Thanks,

Bruno









^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-06 19:24           ` Bruno Barbier
@ 2024-02-07 16:19             ` Ihor Radchenko
  2024-02-07 17:40               ` Bruno Barbier
  2024-02-08  3:21             ` Jack Kamm
  2024-02-15 20:02             ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt
  2 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-07 16:19 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Jack Kamm, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> FWIW, I've been trying to use asynchronous blocks for everything, not
> only the source blocks that are based on the comint mode.  I think it
> would be good if ob-core itself could provide an asynchronous API.  I've
> modified my Org so that it does have such an API.  This is work in
> progress; let me describe it.
>
> I've modified ob-core itself to allow asynchronicity.  In the
> asynchrosous case, instead of calling:
>
>   (org-babel-execute:LANG body params)
>
> I'm calling:
>
>   (org-babel-schedule:LANG body params handle-result)
>
> where `org-babel-schedule:LANG' is in charge of calling `handle-result'
> with the result (or the error) when it is known; `handle-result' takes
> care to call `org-babel-insert-result' at the correct place (and
> `org-babel-insert-result' is only called with a real result).

LGTM.

> While the execution is pending, I'm using the same technique that Org is
> using when a source block is being edited: the result is left untouched,
> but below an overlay.  The overlay is used to know where to insert the
> result and to display the status/progress of the execution.  If the file
> is closed and the execution fails, nothing is lost, the old result is
> still available.
>
> If that technique looks safe enough and interesting, I can prepare a set
> of patches so that we can discuss it further and, maybe, add it in Org.

Using overlay also sounds good.

I am wondering what you do when there is no result yet or when user
edits whatever is under the overlay, but that's just a technical detail.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-07 16:19             ` Ihor Radchenko
@ 2024-02-07 17:40               ` Bruno Barbier
  0 siblings, 0 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-07 17:40 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Jack Kamm, emacs-orgmode



Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> While the execution is pending, I'm using the same technique that Org is
>> using when a source block is being edited: the result is left untouched,
>> but below an overlay.  The overlay is used to know where to insert the
>> result and to display the status/progress of the execution.  If the file
>> is closed and the execution fails, nothing is lost, the old result is
>> still available.
>>
>> If that technique looks safe enough and interesting, I can prepare a set
>> of patches so that we can discuss it further and, maybe, add it in Org.
>
> Using overlay also sounds good.
>
> I am wondering what you do when there is no result yet or when user
> edits whatever is under the overlay,
> but that's just a technical detail.

I don't think it's really robust yet. We'll see how to improve/change
it.

I'll prepare the set of patchs.


Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-06 19:24           ` Bruno Barbier
  2024-02-07 16:19             ` Ihor Radchenko
@ 2024-02-08  3:21             ` Jack Kamm
  2024-02-15 20:02             ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt
  2 siblings, 0 replies; 35+ messages in thread
From: Jack Kamm @ 2024-02-08  3:21 UTC (permalink / raw)
  To: Bruno Barbier, Ihor Radchenko; +Cc: emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> FWIW, I've been trying to use asynchronous blocks for everything, not
> only the source blocks that are based on the comint mode.  I think it
> would be good if ob-core itself could provide an asynchronous API.  I've
> modified my Org so that it does have such an API.  This is work in
> progress; let me describe it.
>
> I've modified ob-core itself to allow asynchronicity.  In the
> asynchrosous case, instead of calling:
>
>   (org-babel-execute:LANG body params)
>
> I'm calling:
>
>   (org-babel-schedule:LANG body params handle-result)
>
> where `org-babel-schedule:LANG' is in charge of calling `handle-result'
> with the result (or the error) when it is known; `handle-result' takes
> care to call `org-babel-insert-result' at the correct place (and
> `org-babel-insert-result' is only called with a real result).

Sounds interesting, a couple questions:

1. Which languages have you implemented/tested this on?

2. Does it apply for sessions, nonsessions, or both?

> While the execution is pending, I'm using the same technique that Org is
> using when a source block is being edited: the result is left untouched,
> but below an overlay.  The overlay is used to know where to insert the
> result and to display the status/progress of the execution.  If the file
> is closed and the execution fails, nothing is lost, the old result is
> still available.

Also interesting, I think it's worth exploring/testing this overlay idea
out. Does that mean that output is asynchronously printing into the Org
buffer? It sounds cool but I wonder if it might cause problems while
trying to edit another part of the buffer.


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]
  2024-02-05 14:29         ` Ihor Radchenko
  2024-02-06 19:24           ` Bruno Barbier
@ 2024-02-08  3:26           ` Jack Kamm
  1 sibling, 0 replies; 35+ messages in thread
From: Jack Kamm @ 2024-02-08  3:26 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode

Ihor Radchenko <yantar92@posteo.net> writes:

>> I agree that it would be good to redesign it, but am not sure where to
>> start.
>
> For example,
>
> 1. Change `org-babel-comint-async-register' to return UUID and to store
>    PARAMS as passed by the backend (current approach with PARAMS being
>    derived from src blocks prevents backends to transform src block
>    PARAMS dynamically).
> 2. Change `org-babel-insert-result' to handle :async t specially,
>    inserting something reliable, like #+async: <UUID> in place of result
>    without performing extra transformations.
> 3. Change `org-babel-insert-result' to accept an internal parameter
>    that will make it replace #+async: <UUID> keyword rather than perform
>    normal result insertion.
> 4. Change `org-babel-comint-async-filter' to use the previously passed
>    PARAMS, remove :async t from them, and arrange the call to
>    `org-babel-insert-result' to replace the #+async: <UUID> keyword.

That all sounds reasonable...if you work on this, let me know if you
want any help with testing.


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-06 19:24           ` Bruno Barbier
  2024-02-07 16:19             ` Ihor Radchenko
  2024-02-08  3:21             ` Jack Kamm
@ 2024-02-15 20:02             ` Matt
  2024-02-16 17:52               ` Bruno Barbier
  2 siblings, 1 reply; 35+ messages in thread
From: Matt @ 2024-02-15 20:02 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode

If I followed correctly, the topic switched to discussing async generally within Babel.  I've started a new thread accordingly.

 ---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier

 > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.

I've been trying to figure out how to make everything async by default.  I'm super interested to see what you've come up with.

 > I think it would be good if ob-core itself could provide an asynchronous API.

Fortunately or unfortunately, depending on how you look at it, Babel already does.

The challenge is that the Org Babel API has grown piecemeal over 14 years.  It's been written by several authors with limited time and knowledge.  The result, while powerful and useful, is a bit of a hodgepodge.  A prime example is that the concepts of "persistence" and "synchronicity" are conflated.  "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal.  Emacs provides primitives that could make non-session blocks asynchronous.  It's historical accident that blocks aren't async by default.

For me, the issue is that the Babel API needs some high level perspective in order to make it consistent.  I see the following terms as guides.  If we can separate these concepts within the API, then Babel to *feel* like an API:

- "Session" means a shell environment is "persistent."  Each call is executed in the same environment.  State exists between calls.

- "Non-session" means a shell environment is "temporary."  Each call is executed in an independent environment.  State does not exist between calls.

- "Synchronous" means that execution prevents the user from editing the document while results are obtained.

- "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

 > I've modified my Org so that it does have such an API.  This is work in progress; let me describe it.
 >
 > I've modified ob-core itself to allow asynchronicity.  In the asynchrosous case, instead of calling:
 >
 >   (org-babel-execute:LANG body params)
 >
 > I'm calling:
 >
 >   (org-babel-schedule:LANG body params handle-result)
 >
 > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result).
 >
 > While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay.  The overlay is used to know where to insert the result and to display the status/progress of the execution.  If the file is closed and the execution fails, nothing is lost, the old result is still available.

The use of the overlay is a really cool idea!

I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.

However, using an overlay to communicate "the process is still running" could be good.  We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.

 > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.

Please do!  I'm super interested.  I've put a lot of thought into how we might make Babel async by default.  I'm excited to see your interest in the topic and look forward to seeing what you've come up with.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode



^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-15 20:02             ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt
@ 2024-02-16 17:52               ` Bruno Barbier
  2024-02-18 21:14                 ` Matt
                                   ` (2 more replies)
  0 siblings, 3 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-16 17:52 UTC (permalink / raw)
  To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 8614 bytes --]


Hi Matt, Jack, Ihor,

Sorry for the late reply.  Cleaning the code took me longer than
expected.


Jack Kamm <jackkamm@gmail.com> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> FWIW, I've been trying to use asynchronous blocks for everything, not
>> only the source blocks that are based on the comint mode.
>> ...
>
> Sounds interesting, a couple questions:
>
> 1. Which languages have you implemented/tested this on?

I'm not using it with official org backends (yet).  I'm using it with
several custom backends that I'm working on.  One of the backend
delegate the block executions to emacs subprocesses: so I have a kind of
asynchronous executions for free for any language, including elisp
itself.


> 2. Does it apply for sessions, nonsessions, or both?
>

The new API itself is more about how to wait for and display one block
result.  So, it's not really aware of sessions.

But, I usually try to think as "no session" as a "one shot" session
(like Matt wrote in his email).  So, in that sense, it works for both
anyway ;-)


> Also interesting, I think it's worth exploring/testing this overlay idea
> out. Does that mean that output is asynchronously printing into the Org
> buffer? It sounds cool but I wonder if it might cause problems while
> trying to edit another part of the buffer.

Currently, I've limited the progress feedback to fit on one line to
avoid anoying vertical display jumps.  When the computation is
successful, the result is inserted as usual, and, that may be annoying;
when it updates a previous result of the same height, it's ok.  Else, we
could fold it to stay on one line too (using the overlay), until the
user explicitly request to see it.

Jack Kamm <jackkamm@gmail.com> writes:
>
> That all sounds reasonable...if you work on this, let me know if you
> want any help with testing.

Thanks.  I'll definitely appreciate your help, to test the current
code with the Python backend or any other backend if you prefer.


Matt <matt@excalamus.com> writes:

> If I followed correctly, the topic switched to discussing async
> generally within Babel.  I've started a new thread accordingly.

Indeed. Thank you!


>  > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.
>
> I've been trying to figure out how to make everything async by default.  I'm super interested to see what you've come up with.

Good to know.  Let's we make this happen! 


>  > I think it would be good if ob-core itself could provide an asynchronous API.
>
> Fortunately or unfortunately, depending on how you look at it, Babel already does.
>

> The challenge is that the Org Babel API has grown piecemeal over 14 years.  It's been written by several authors with limited time and knowledge.  The result, while powerful and useful, is a bit of a hodgepodge.  A prime example is that the concepts of "persistence" and "synchronicity" are conflated.  "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal.  Emacs provides primitives that could make non-session blocks asynchronous.  It's historical accident that blocks aren't async by default.

> For me, the issue is that the Babel API needs some high level perspective in order to make it consistent.

> I see the following terms as guides.  If we can separate these concepts within the API, then Babel to *feel* like an API:
>
> - "Session" means a shell environment is "persistent."  Each call is executed in the same environment.  State exists between calls.
>
> - "Non-session" means a shell environment is "temporary."  Each call is executed in an independent environment.  State does not exist between calls.
>
> - "Synchronous" means that execution prevents the user from editing the document while results are obtained.
>
> - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

I mostly think the same.  Sessions (including the "none" session)
definitely need some generic API and some generic tests that all
backends could just reuse.

To execute Python blocks, using the proposed async API:

   - I've (re)implemented the "asynchronous with session" case (copying/pasting
     the relevant part from ob-python).
   
   - The "synchronous case" is just artificially blocking the user until
     the asynchronous result is known (which looks incredibly tricky to
     implement if even possible...).
   
   - The "no session" case is just about creating a new unique session
     and throwing it away immediately.

But, some users may rely on some particular behavior, of the current
implementations, that may be hard to implement in such a generic way.


> The use of the overlay is a really cool idea!
>
> I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.

I'm not sure I fully understand what you mean. The API just assumes the
backend returns the outcome: either success or failure, where failure
means "no result" (the previous result, if it exists, is even preserved
in the document).  The backend is free to transform a failure into a
success to make that result available though.


> However, using an overlay to communicate "the process is still running" could be good.  We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.

You should be able to test my current implementation, see below.  It's
almost too visual in my opinion.  But, that's probably something that we
should make easy to configure.

>  > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.
>
> Please do!  I'm super interested.  I've put a lot of thought into how we might make Babel async by default.  I'm excited to see your interest in the topic and look forward to seeing what you've come up with.

Good to know!


So, here we go.  You'll find attach a set of patchs.  It works for me with
Emacsc 30.50 and 9.7-pre (from today).

I didn't check yet that the code and the commits follow all the
guidelines.  This is just a preliminary version for feedbacks.
Corrections/critiques are welcome, but don't check the details until I
check them myself.


The 5 first patchs provide an API to handle asynchronous execution in
ob-core, i.e. an API to report progress and to insert results in the
asynchronous case.  The 5th one isn't really about asynchronicity; but
it adds a new keyword `:execute-with' that allows to delegate block
executions to some other package; it's useful for testing or plugging
other execution engines.

That's a cleaned-up version of what I been using myself for a while,
with 4 different values for `:execute-with'.

The remaining patchs are new code that I've just written to show how to
use this new API.

I tried first to use it for ob-python, as an example.  I just needed to
figure out where to place the callbacks ... well ... "just" ... :)


So, I decided to rewrite the whole thing, taking code from the
synchronous case (following `org-babel-python-evaluate-session').  I
also created a package that contains all the functions that should be
reusable for any language.  The patch [1] adds some generic functions to
help dealing with asynchronicity (process, comint, etc.). The patch [2]
shows a new possible way to execute python code blocks, both
synchronously and asynchronously, with or without sessions.  You should
just need to open [3] and follow what's written there, and execute the
existing bash and python code blocks.

Note that this will create a folder `scratch/bba-ob-core-async' in your
repository to place the temporary files there.  If you know a better way
to do this, let me know, thanks.

I think the first 5 patchs could be included almost as-is in org.  About
the emaining ones, I'm not sure exactly how we should proceed.

Let me know if you need help to test it.

Feedbacks, corrections, critiques, etc are most welcome!

Thanks!

Bruno




[1] lisp/org-elib-async.el: New package about async helpers
[2] scratch/bba-ob-core-async: Some temporary test files
[3] scratch/bba-ob-core-async/my-async-tests.org



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 1367 bytes --]

From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:31:36 +0100
Subject: [PATCH 1/8] ob-core async: Add faces [1/5]

lisp/org-faces.el (org-async-scheduled, org-async-pending,
org-async-failure): new faces
---
 lisp/org-faces.el | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 0e20de51a..5a8a8fd51 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun
   "Face used for clock display for overrun tasks in mode line."
   :group 'org-faces)
 
+(defface org-async-scheduled '((t :inherit org-tag :background "gray"))
+  "Face for babel results for code blocks that are scheduled for execution."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-pending '((t :inherit org-checkbox :background "dark orange"))
+  "Face for babel results for code blocks that are running."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-failure '((t :inherit org-warning))
+  "Face for babel results for code blocks that have failed."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
 (provide 'org-faces)
 
 ;;; org-faces.el ends here
-- 
2.43.0


[-- Attachment #3: 0002-ob-core-async-Add-org-babel-async-tools-2-5.patch --]
[-- Type: text/x-patch, Size: 10450 bytes --]

From 8ecdc2159d85648949b477359793f007005bb7ca Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:00 +0100
Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5]

---
 lisp/ob-core.el | 219 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 219 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bfeac257b..bb44f91cb 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -792,6 +792,225 @@ (defun org-babel-session-buffer (&optional info)
     (when (org-babel-comint-buffer-livep buffer-name)
       buffer-name)))
 
+(defun org-babel--async-status-face (status)
+  (pcase status
+    (:scheduled 'org-async-scheduled)
+    (:pending   'org-async-pending)
+    (:failure   'org-async-failure)
+    (:success   nil)
+    (_ (error "Not a status"))
+    ))
+
+(defun org-babel--async-make-overlay (beg end)
+  "Create an overlay between positions BEG and END and return it."
+  (let ((overlay (make-overlay beg end))
+        (read-only
+	 (list
+	  (lambda (&rest _)
+	    (user-error
+	     "Cannot modify an area being updated"))))
+        )
+    (cl-flet ((make-read-only
+               (ovl)
+               (overlay-put ovl 'modification-hooks read-only)
+               (overlay-put ovl 'insert-in-front-hooks read-only)
+               (overlay-put ovl 'insert-behind-hooks read-only))
+              )
+      (overlay-put overlay 'org-babel--async-type 'org-babel--async-note)
+      (overlay-put overlay 'face 'secondary-selection)
+      (overlay-put overlay 'help-echo "Pending src block result...")
+      (make-read-only overlay)
+      overlay)))
+
+(defun org-babel--async-result-region (inline-elem &optional info)
+  "Return the region of the results, for the source block at point."
+  (unless info (setq info (org-babel-get-src-block-info)))
+  (save-excursion
+    (when-let ((res-begin  (org-babel-where-is-src-block-result nil info)))
+      (cons res-begin
+            (save-excursion
+              (goto-char res-begin)
+              (if inline-elem
+                  ;; Logic copy/pasted from org-babel-where-is-src-block-result.
+	          (let ((result (org-element-context)))
+		    (and (org-element-type-p result 'macro)
+		         (string= (org-element-property :key result)
+			          "results")
+		         (progn
+			   (goto-char (org-element-end result))
+			   (skip-chars-backward " \t")
+			   (point))))
+                ;; Logic copy/pasted from hide-result
+                (beginning-of-line)
+                (let ((case-fold-search t))
+                  (unless (re-search-forward org-babel-result-regexp nil t)
+	            (error "Not looking at a result line")))
+                (org-babel-result-end)
+                ))))))
+
+(defun org-babel--async-feedbacks (info handle-result
+                                       result-params exec-start-time)
+  "Flag the result as 'scheduled' and return how to handle feedbacks.
+
+Use overlays to report progress and status to the user.  Do not delete
+the existing result unless a new one is available.  When the result is
+available, remove the async overlays and insert the result as usual,
+like for a synchronous result.  In case of failure, use an overlay to
+report the error.
+
+The returned function handles 3 types of feedbacks:
+   - (:success R):   Evaluation is successful; result is R.
+   - (:failure ERR): Evaluation failed; error is ERR.
+   - (:pending P): Outcome still pending; current progress is P."
+  ;; FIXME: INFO CMD ... Nothing is used but handle-result here !!
+  (let (;; copy/pasted from org-babel-insert-result
+        (inline-elem (let ((context (org-element-context)))
+		       (and (memq (org-element-type context)
+			          '(inline-babel-call inline-src-block))
+			    context)))
+        result-indent
+        )
+    (cl-labels
+        ((eot-point (start)
+           "Move to End Of Title after START"
+           (if inline-elem
+               (org-element-end inline-elem)
+             (save-excursion (goto-char start)
+                             (forward-line 1) (point))))
+         (after-indent (pt)
+           "Move after indentation, starting at PT."
+           (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*")))
+         (mk-result-overlays ()
+           ;; Make 2 overlays to handle the pending result: one title
+           ;; (first line) and one for the body.
+           (pcase-let ((`(,start . ,end) (org-babel--async-result-region
+                                          inline-elem info)))
+             (let ((anchor-end (eot-point start)))
+               (cons (org-babel--async-make-overlay
+                      (after-indent start)
+                      (1- anchor-end))
+                     (org-babel--async-make-overlay
+                      anchor-end end)))))
+         (add-style (status txt)
+           ;; Add the style matching STATUS over the text TXT.
+           (propertize txt 'face (org-babel--async-status-face status)))
+
+         (short-version-of (msg)
+           ;; Compute the short version of MSG, to display in the header.
+           ;; Must return a string.
+           (if msg
+               (car (split-string (format "%s" msg) "\n" :omit-nulls))
+             ""))
+         (update (ovl-title status msg)
+           ;; Update the title overlay to match STATUS and MSG.
+           (let (header)
+             (overlay-put ovl-title
+                          'face
+                          (org-babel--async-status-face status)
+                          )
+             (overlay-put ovl-title
+                          'before-string (pcase status
+                                           (:scheduled "⏱")
+                                           (:pending "⏳")
+                                           (:failure "❌")
+                                           (:success "✔️")))
+
+             (overlay-put ovl-title
+                          'after-string
+                          (propertize (format " |%s|"
+                                              (if (eq :failure status)
+                                                  (if (consp msg) (car msg)
+                                                    (format "%s" msg))
+                                                  (short-version-of msg)))
+                                      'face (org-babel--async-status-face status)))
+             ))
+         (remove-previous-overlays ()
+           ;; Remove previous title and body overlays.
+           (mapc (lambda (ovl)
+                   (when (eq 'org-babel--async-note
+                             (overlay-get ovl 'org-babel--async-type))
+                     (delete-overlay ovl)))
+                 (when-let ((region (org-babel--async-result-region
+                                     inline-elem info)))
+                   ;; Not sure why, but we do need to start before
+                   ;; point min, else, in some cases, some overlays
+                   ;; are not found.
+                   (overlays-in (max (1- (car region)) (point-min))
+                                (cdr region))))))
+
+      (remove-previous-overlays)
+
+      ;; Ensure there is a non-empty region for the result.
+      (save-excursion
+         (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil)
+          (org-babel-insert-result
+           ;; Use " " for the empty result. That cannot be nil, else it's interpreted
+           ;; as a list. We need at least one char, to separate markers if any.
+           " \n"
+           result-params
+           info nil
+           (nth 0 info) ; lang
+           exec-start-time
+           )))
+
+      ;; Create the overlays that span the result title and its body.
+      (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays)))
+        ;; Flag the result as ":scheduled".
+        (update title-ovl :scheduled nil)
+
+        ;; The callback, that runs in the org buffer at point.
+        (let ((buf (current-buffer))
+              (pt  (point-marker)))
+          (lambda (feedback)
+            (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback)
+            (with-current-buffer buf
+              (save-excursion
+                (goto-char pt)
+                (pcase feedback
+                  (`(:success ,r)
+                   ;; Visual beep that the result is available.
+                   (update title-ovl :success r)
+                   (sit-for 0.2)
+                   ;; We remove all overlays and let org insert the result
+                   ;; as it would in the synchronous case.
+                   (delete-overlay title-ovl)
+                   (delete-overlay body-ovl)
+                   (funcall handle-result r))
+
+                  (`(:pending ,r)
+                   ;; Still waiting for the outcome. Update our
+                   ;; overlays with the progress info R.
+                   (message "Updating block at %s@%s" pt buf)
+                   (update title-ovl :pending r))
+
+                  (`(:failure ,err)
+                   ;; We didn't get a result. We update our overlays
+                   ;; to report that failure. And unlock the old
+                   ;; result.
+                   (overlay-put title-ovl 'face nil)
+                   (update title-ovl :failure err)
+                   (delete-overlay body-ovl))
+
+                  (_ (error "Invalid outcome"))
+                  )
+                ))
+            nil))))))
+
+
+(cl-defun org-babel--async-p (params &key default)
+  "Return a non-nil value when the execution is asynchronous.
+Get the value of the :nasync argument and convert it."
+  (if-let ((binding (assq :nasync params)))
+      (pcase (cdr binding)
+        ((pred (not stringp))
+         (error "Invalid value for :nasync argument"))
+        ((or "no" "n")  nil)
+        ((or "yes" "y") t)
+        (_ (error "Invalid value for :nasync argument")))
+    default))
+
+
+
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
   "Execute the current source code block and return the result.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --]
[-- Type: text/x-patch, Size: 5958 bytes --]

From e8fe2f922a992de48ca80c52f530b0aa5fb45ea3 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:22 +0100
Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5]

lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to
prepare for the next change: move the part handling the result in its
own function `handle-result'.
---
 lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------
 1 file changed, 53 insertions(+), 52 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bb44f91cb..f8071a534 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1092,7 +1092,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
-		 result exec-start-time)
+                 (exec-start-time (current-time))
+                 (handle-result
+                  (lambda (result)
+                    (setq result
+		          (if (and (eq (cdr (assq :result-type params)) 'value)
+			           (or (member "vector" result-params)
+				       (member "table" result-params))
+			           (not (listp result)))
+		              (list (list result))
+		            result))
+	            (let ((file (and (member "file" result-params)
+			             (cdr (assq :file params)))))
+		      ;; If non-empty result and :file then write to :file.
+		      (when file
+		        ;; If `:results' are special types like `link' or
+		        ;; `graphics', don't write result to `:file'.  Only
+		        ;; insert a link to `:file'.
+		        (when (and result
+			           (not (or (member "link" result-params)
+				            (member "graphics" result-params))))
+		          (with-temp-file file
+		            (insert (org-babel-format-result
+			             result
+			             (cdr (assq :sep params)))))
+		          ;; Set file permissions if header argument
+		          ;; `:file-mode' is provided.
+		          (when (assq :file-mode params)
+		            (set-file-modes file (cdr (assq :file-mode params)))))
+		        (setq result file))
+		      ;; Possibly perform post process provided its
+		      ;; appropriate.  Dynamically bind "*this*" to the
+		      ;; actual results of the block.
+		      (let ((post (cdr (assq :post params))))
+		        (when post
+		          (let ((*this* (if (not file) result
+				          (org-babel-result-to-file
+				           file
+				           (org-babel--file-desc params result)
+                                           'attachment))))
+		            (setq result (org-babel-ref-resolve post))
+		            (when file
+			        (setq result-params (remove "file" result-params))))))
+	              (unless (member "none" result-params)
+		        (org-babel-insert-result
+		         result result-params info
+                         ;; append/prepend cannot handle hash as we accumulate
+                         ;; multiple outputs together.
+                         (when (member "replace" result-params) new-hash)
+                         lang
+                         (time-subtract (current-time) exec-start-time)))
+	              (run-hooks 'org-babel-after-execute-hook)
+                      result))))
 	    (unless (fboundp cmd)
 	      (error "No org-babel-execute function for %s!" lang))
 	    (message "Executing %s %s %s..."
@@ -1107,57 +1158,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-	    (setq exec-start-time (current-time)
-                  result
-		  (let ((r (save-current-buffer (funcall cmd body params))))
-		    (if (and (eq (cdr (assq :result-type params)) 'value)
-			     (or (member "vector" result-params)
-				 (member "table" result-params))
-			     (not (listp r)))
-			(list (list r))
-		      r)))
-	    (let ((file (and (member "file" result-params)
-			     (cdr (assq :file params)))))
-	      ;; If non-empty result and :file then write to :file.
-	      (when file
-		;; If `:results' are special types like `link' or
-		;; `graphics', don't write result to `:file'.  Only
-		;; insert a link to `:file'.
-		(when (and result
-			   (not (or (member "link" result-params)
-				  (member "graphics" result-params))))
-		  (with-temp-file file
-		    (insert (org-babel-format-result
-			     result
-			     (cdr (assq :sep params)))))
-		  ;; Set file permissions if header argument
-		  ;; `:file-mode' is provided.
-		  (when (assq :file-mode params)
-		    (set-file-modes file (cdr (assq :file-mode params)))))
-		(setq result file))
-	      ;; Possibly perform post process provided its
-	      ;; appropriate.  Dynamically bind "*this*" to the
-	      ;; actual results of the block.
-	      (let ((post (cdr (assq :post params))))
-		(when post
-		  (let ((*this* (if (not file) result
-				  (org-babel-result-to-file
-				   file
-				   (org-babel--file-desc params result)
-                                   'attachment))))
-		    (setq result (org-babel-ref-resolve post))
-		    (when file
-		      (setq result-params (remove "file" result-params))))))
-	      (unless (member "none" result-params)
-	        (org-babel-insert-result
-	         result result-params info
-                 ;; append/prepend cannot handle hash as we accumulate
-                 ;; multiple outputs together.
-                 (when (member "replace" result-params) new-hash)
-                 lang
-                 (time-subtract (current-time) exec-start-time))))
-	    (run-hooks 'org-babel-after-execute-hook)
-	    result)))))))
+            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --]
[-- Type: text/x-patch, Size: 2620 bytes --]

From c9d64263af8d9d9b8efc83585323cc6d876018dc Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:45 +0100
Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5]

---
 lisp/ob-core.el | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index f8071a534..0f0a36b70 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1091,7 +1091,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		    (let ((d (file-name-as-directory (expand-file-name dir))))
 		      (make-directory d 'parents)
 		      d))))
-		 (cmd (intern (concat "org-babel-execute:" lang)))
+                 (async (org-babel--async-p params))
+                 (cmd (intern (concat "org-babel-"
+                                      (if async "schedule" "execute")
+                                      ":" lang)))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1145,8 +1148,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 	              (run-hooks 'org-babel-after-execute-hook)
                       result))))
 	    (unless (fboundp cmd)
-	      (error "No org-babel-execute function for %s!" lang))
-	    (message "Executing %s %s %s..."
+	      (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd)))
+	    (message "Executing %s %s %s %s..."
 		     (capitalize lang)
                      (pcase executor-type
                        ('src-block "code block")
@@ -1154,11 +1157,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                        ('babel-call "call")
                        ('inline-babel-call "inline call")
                        (e (symbol-name e)))
+                     (if async "async" "")
 		     (let ((name (nth 4 info)))
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
+            (if (not async)
+                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+              (let ((handle-feedback
+                     (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
+                (funcall cmd body params handle-feedback))))))))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --]
[-- Type: text/x-patch, Size: 2458 bytes --]

From 8b311516505f693488bfef661905f34e7375539c Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:09 +0100
Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5]

---
 lisp/ob-core.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 0f0a36b70..1de94330a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1092,9 +1092,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
                  (async (org-babel--async-p params))
-                 (cmd (intern (concat "org-babel-"
-                                      (if async "schedule" "execute")
-                                      ":" lang)))
+                 (execute-with (let ((be (cdr (assq :execute-with params))))
+                                 (when (equal be "none") (setq be nil))
+                                 be))
+                 (cmd (intern (or (and execute-with
+                                       (concat execute-with "-" (if async "schedule" "execute")))
+                                  (concat "org-babel-"
+                                          (if async "schedule" "execute")
+                                          ":" lang))))
+                 (cmd-args (let ((ps (list body params)))
+                             (when execute-with
+                               (setq ps (cons lang ps)))
+                             ps))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1163,10 +1172,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
-                (funcall cmd body params handle-feedback))))))))))
+                (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --]
[-- Type: text/x-patch, Size: 14462 bytes --]

From 50ecb37089b96cd0fe4107e4d74357c151540e34 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:23 +0100
Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers

---
 lisp/org-elib-async.el | 323 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 323 insertions(+)
 create mode 100644 lisp/org-elib-async.el

diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el
new file mode 100644
index 000000000..ba4e08869
--- /dev/null
+++ b/lisp/org-elib-async.el
@@ -0,0 +1,323 @@
+;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Version: 0.0.0
+;; Maintainer: Bruno BARBIER
+;; Keywords:
+;; Status: WORK IN PROGRESS.  DO NOT USE.
+;; URL:
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT (yet) part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+
+;;; Commentary:
+;; Names with "--" are for functions and variables that are meant to be for
+;; internal use only.
+
+;;;; Description
+;; Some functions to help dealing with asynchronous tasks.
+
+;; The prefix 'org-elib' means that this package should evenutally be
+;; moved into core Emacs.  The functions defined here do NOT depend
+;; nor rely on org itself.
+
+;;; TODOs
+;;
+;; - Keywords
+;;
+
+
+;;; Code:
+
+;;;; Process
+;;
+(cl-defun org-elib-async-process (command &key input callback)
+  "Execute COMMAND.
+
+A quick naive featureless boggus wrapper around `make-process' to
+receive the result when the process is done.
+
+When INPUT is non-nil, use it as the COMMAND standard input.  Let DATA
+be the COMMAND output, if COMMAND succeeds, call CALLBACK with
+'(:success DATA), else, call CALLBACK with '(:failure DATA)."
+  (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*"))
+         (get-outcome
+          (lambda (process)
+            (with-current-buffer stdout-buffer
+              (let* ((exit-code (process-exit-status process))
+                     (real-end ;; Getting rid of the user message.
+                      (progn (goto-char (point-max))
+                             (forward-line -1)
+                             (point)))
+                     (txt (string-trim (buffer-substring-no-properties
+                                        (point-min) real-end))))
+                (list (if (eq 0 exit-code) :success :failure)
+                      (if (not (string-empty-p txt)) txt
+                        (and (not (eq 0 exit-code)) exit-code)))))))
+         (process (make-process
+                   :name "*org-elib-async-process*"
+                   :buffer  stdout-buffer
+                   :command command
+                   :connection-type 'pipe))
+         (sentinel
+          (lambda (&rest _whatever)
+            (pcase (process-status process)
+              ('run )
+              ('stop)
+              ((or 'exit 'signal)
+               (funcall callback (funcall get-outcome process)))
+              (_ (error "Not a real process"))))))
+    (add-function :after (process-sentinel process) sentinel)
+    (when input
+      (process-send-string process input)
+      (process-send-eof process))
+    process))
+;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o)))
+
+
+;;;; Wait for a process until some condition becomes true.
+
+(define-error 'org-elib-async-timeout-error
+              "Timeout waiting for a process.")
+
+(cl-defun org-elib-async-wait-condition ( cond-p
+                                          &key
+                                          (tick .3) (message "Waiting")
+                                          (nb_secs_between_messages 5)
+                                          timeout)
+  "Wait until the condition COND-P returns non-nil.
+Repeatedly call COND-P with no arguments, about every TICK seconds,
+until it returns a non-nil value.  Return that non-nil value.  When
+TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if
+the COND-P is still nil after TIMEOUT seconds.  Assume COND-P calls cost
+0s.  Do NOT block display updates.  Do not block process outputs.  Do
+not block idle timers.  Do block the user, letting him/her know why, but
+do not display more messages than one every NB_SECS_BETWEEN_MESSAGES.
+Default MESSAGE is \"Waiting\".  Use 0.3s as the default for TICK."
+  ;; FIXME: Still not sure if it's possible to write such a function.
+  (let ((keep-waiting t)
+        (result nil)
+        (start (float-time))
+        elapsed
+        last-elapsed
+        )
+    (while keep-waiting
+      (setq result (funcall cond-p))
+      (if result
+          (setq keep-waiting nil)
+        (sleep-for 0.01)
+        (redisplay :force)
+        (setq elapsed (- (float-time) start))
+        (when (and timeout (> elapsed timeout))
+          (signal 'org-timeout-error message elapsed))
+        ;; Let the user know, without flooding the message area.
+        (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages))
+            (message (format "%s ...(%.1fs)" message elapsed)))
+        (unless (sit-for tick :redisplay)
+          ;; Emacs has something to do; let it process new
+          ;; sub-processes outputs in case there are some.
+          (accept-process-output nil 0.01))))
+    result))
+
+
+
+;;;; Comint: a FIFO queue of tasks with callbacks
+;; org-elib-async-comint-queue executes tasks in a FIFO order. For each
+;; task, it identifies the text output for that
+;; task. org-elib-async-comint-queue does NOT remove prompts, or other
+;; useless texts; this is the responsibility of the user.  Currently,
+;; org-elib-async-comint-queue assume it has the full control of the
+;; session: no user interaction, no other direct modifications.
+
+(defvar-local org-elib-async-comint-queue--todo :NOT-SET
+  "A FIFO queue of pending executions.")
+
+
+(defvar-local org-elib-async-comint-queue--unused-output ""
+  "Process output that has not been used yet.")
+
+(defvar-local org-elib-async-comint-queue--incoming-text ""
+  "Newly incoming text, added by the process filter, not yet handled.")
+
+(defvar-local org-elib-async-comint-queue--current-task nil
+  "The task that is currently running.")
+
+(defvar-local org-elib-async-comint-queue--process-filter-running nil
+  "non-nil when filter is running.")
+
+(defvar-local org-elib-async-comint-queue--incoming-timer nil
+  "A timer, when handling incoming text is scheduled or running.")
+
+
+(defvar-local org-elib-async-comint-queue--handle-incoming-running
+    nil
+  "True when the incoming text handler is running.")
+
+(defun org-elib-async-comint-queue--handle-incoming ()
+  (when org-elib-async-comint-queue--handle-incoming-running
+    (error "Bad filter call detected: Kill buffer %s!" (current-buffer)))
+  (setq org-elib-async-comint-queue--handle-incoming-running t)
+
+  ;; Take the incoming text.
+  (setq org-elib-async-comint-queue--unused-output
+        (concat org-elib-async-comint-queue--unused-output
+                org-elib-async-comint-queue--incoming-text))
+  (setq org-elib-async-comint-queue--incoming-text "")
+
+  ;; Process the unused text with the queued tasks
+  (unless org-elib-async-comint-queue--current-task
+    (when org-elib-async-comint-queue--todo
+      (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo))))
+  (when-let ((task org-elib-async-comint-queue--current-task))
+    (let ((unused org-elib-async-comint-queue--unused-output)
+          (session-buffer (current-buffer))
+          task-start)
+      (setq org-elib-async-comint-queue--unused-output
+            (with-temp-buffer
+              (insert unused)
+              (goto-char (point-min))
+              (while (and task
+                          (setq task-start (point))
+                          (search-forward (car task) nil t))
+                (when (cdr task)
+                  (let ((txt (buffer-substring-no-properties task-start
+                                                             (- (point) (length (car task))))))
+                    (save-excursion (funcall (cdr task) txt))))
+                (setq task (and (buffer-live-p session-buffer)
+                                (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo)))))
+              (buffer-substring (point) (point-max))))
+      (setq org-elib-async-comint-queue--current-task task)))
+
+  ;; Signal that we are done. If we already have some new incoming text,
+  ;; reschedule to run.
+  (setq org-elib-async-comint-queue--incoming-timer
+        (if (string-empty-p org-elib-async-comint-queue--incoming-text)
+            nil
+          (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+  ;; We reset it only on success. If it failed for some reason, the
+  ;; comint buffer is in an unknown state: you'll need to kill that
+  ;; buffer.
+  (setq org-elib-async-comint-queue--handle-incoming-running nil))
+
+
+(defun org-elib-async-comint-queue--wake-up-handle-incoming ()
+  "Wake up the handling of incoming chunks of text.
+Assume we are called from the comint buffer."
+  (setq org-elib-async-comint-queue--incoming-timer
+        (run-with-timer
+         0.01 nil
+         (let ((comint-buffer (current-buffer)))
+           (lambda ()
+             (with-local-quit
+               (with-current-buffer comint-buffer
+                 (org-elib-async-comint-queue--handle-incoming))))))))
+
+
+(defun org-elib-async-comint-queue--process-filter (chunk)
+  "Accept the arbitrary CHUNK of text."
+  (setq org-elib-async-comint-queue--incoming-text
+        (concat org-elib-async-comint-queue--incoming-text
+                chunk))
+  :; We delegate the real work outside the process filter, as it is
+   ; not reliable to do anything here.
+  (unless org-elib-async-comint-queue--incoming-timer
+    (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+
+
+(define-error 'org-elib-async-comint-queue-task-error
+              "Task failure.")
+
+(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback)
+  "Push the execution of EXEC into the FIFO queue.
+When the task completed, call HANDLE-FEEDBACK with its outcome.  Return
+a function that waits for and return the result on succes, raise on
+failure."
+  (let* ((tid (org-id-uuid))
+         (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid))
+         (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid))
+         (result-sb (make-symbol "result"))
+         (on-start
+          (lambda (_)
+            ;; TODO: Use (point) in session to link back to it.
+            (when handle-feedback
+              (funcall handle-feedback '(:pending "running")))))
+         (on-result
+          (lambda (result)
+            ;; Get the result, and report success using HANDLE-FEEDBACK.
+            ;; If something fails, report failure using HANDLE-FEEDBACK.
+            (unwind-protect
+                (let ((outcome
+                       (condition-case-unless-debug exc
+                           (list :success (funcall exec :post-process result))
+                         (error (list :failure exc)))))
+                  (when handle-feedback (save-excursion (funcall handle-feedback outcome)))
+                  (set result-sb outcome))
+              (funcall exec :finally)))))
+
+    (let ((comint-buffer (funcall exec :get-comint-buffer)))
+      (with-current-buffer comint-buffer
+        (setq org-elib-async-comint-queue--todo
+              (nconc org-elib-async-comint-queue--todo
+                     (list (cons start-tag on-start)
+                           (cons end-tag on-result))))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-enter))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag start-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :get-code))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag end-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-exit))
+
+        (lambda ()
+          (org-elib-async-wait-condition (lambda ()
+                                           (boundp result-sb)))
+          (pcase (symbol-value result-sb)
+            (`(:success ,r) r)
+            (`(:failure ,err) (signal (car err) (cdr err)))))
+        ))))
+
+
+
+(defun org-elib-async-comint-queue-init-if-needed (buffer)
+  "Initialize the FIFO queue in BUFFER if needed."
+  (with-current-buffer buffer
+    (unless (local-variable-p 'org-elib-async-comint-queue--todo)
+      (setq-local org-elib-async-comint-queue--todo nil)
+      (add-hook 'comint-output-filter-functions
+                #'org-elib-async-comint-queue--process-filter nil :local))))
+
+
+
+;;;; Provide
+(provide 'org-elib-async)
+;;; org-elib-async.el ends here
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 3038 bytes --]

From 8db393a06efc3e8a4a3a101be2e59125d1666817 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:40 +0100
Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails

lisp/ob-core.el (org-babel-popup-failure-details): New function.
(org-babel-execute-src-block): For synchronous execution, use
`org-babel-popup-failure-details' on failure.
(org-babel--async-feedbacks): Add call to
`org-babel-popup-failure-details' on user request.
---
 lisp/ob-core.el | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de94330a..c0f6a398a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -923,6 +923,16 @@ (defun org-babel--async-feedbacks (info handle-result
                                                     (format "%s" msg))
                                                   (short-version-of msg)))
                                       'face (org-babel--async-status-face status)))
+             (when (eq :failure status)
+               (overlay-put ovl-title
+                            'keymap
+                            (let ((km (make-sparse-keymap)))
+                              (define-key km (kbd "<mouse-1>")
+                                          (lambda ()
+                                            "Display failure details."
+                                            (interactive)
+                                            (org-babel-popup-failure-details msg)))
+                              km)))
              ))
          (remove-previous-overlays ()
            ;; Remove previous title and body overlays.
@@ -1172,11 +1182,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
+                (let ((res-sb (make-symbol "result")))
+                  (condition-case exc
+                      (set res-sb (save-current-buffer (apply cmd cmd-args)))
+                    (error (org-babel-popup-failure-details exc)))
+                  (when (boundp res-sb)
+                    (funcall handle-result (symbol-value res-sb))))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
                 (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
+(defun org-babel-popup-failure-details (exc)
+  "Notify/display"
+  (when-let ((buf (get-buffer org-babel-error-buffer-name)))
+    (with-current-buffer buf (erase-buffer)))
+  (org-babel-eval-error-notify
+   127 ; Don't have exit-code
+   (if (consp exc)
+       (format "%s\n%s\n" (car exc) (cdr exc))
+     (format "%s\n" exc))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-scratch-bba-ob-core-async-Some-temporary-test-files.patch --]
[-- Type: text/x-patch, Size: 20479 bytes --]

From 69db5eef49c69287270b1f171e53ddb5e76625dd Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:33 +0100
Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files

---
 scratch/bba-ob-core-async/my-async-tests.el  | 172 +++++++
 scratch/bba-ob-core-async/my-async-tests.org | 484 +++++++++++++++++++
 2 files changed, 656 insertions(+)
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.el
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.org

diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el
new file mode 100644
index 000000000..3550fc88f
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.el
@@ -0,0 +1,172 @@
+;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Status: Temporary tests.
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+(require 'org)
+(require 'org-elib-async)
+
+(defun my-shell-babel-schedule (lang body params handle-feedback)
+  "Execute the bash script BODY.
+Execute the shell script BODY using bash.  Use HANDLE-FEEDBACK to report
+the outcome (success or failure)."
+  (unless (equal "bash" lang)
+    (error "Only for bash"))
+  (funcall handle-feedback (list :pending "started"))
+  (org-elib-async-process (list "bash") :input body :callback handle-feedback))
+
+
+(defun my-org-babel-python-how-to-execute (body params)
+  "Return how to execute BODY using python.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-python, following
+  ;; `org-babel-python-evaluate-session'.
+  ;; Results are expected to differ from ob-python as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((org-babel-python-command
+                (or (cdr (assq :python params))
+                    org-babel-python-command))
+               (session-key (org-babel-python-initiate-session
+                                   (cdr (assq :session params))))
+               (graphics-file (and (member "graphics" (assq :result-params params))
+                                               (org-babel-graphical-output-file params)))
+         (result-params (cdr (assq :result-params params)))
+         (result-type (cdr (assq :result-type params)))
+         (results-file (when (eq 'value result-type)
+                         (or graphics-file
+                             (org-babel-temp-file "python-"))))
+               (return-val (when (eq result-type 'value)
+                                   (cdr (assq :return params))))
+               (preamble (cdr (assq :preamble params)))
+         (full-body
+                (concat
+                 (org-babel-expand-body:generic
+                  body params
+                  (org-babel-variable-assignments:python params))
+                 (when return-val
+                   (format "\n%s" return-val))))
+         (post-process
+          (lambda (r)
+            (setq r (string-trim r))
+            (when (string-prefix-p "Traceback (most recent call last):" r)
+              (signal 'user-error (list r)))
+            (when (eq 'value result-type)
+              (setq r (org-babel-eval-read-file results-file)))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (org-babel-python-table-or-string r))
+             (org-babel-pick-name (cdr (assq :colname-names params))
+                                                    (cdr (assq :colnames params)))
+             (org-babel-pick-name (cdr (assq :rowname-names params))
+                                                    (cdr (assq :rownames params))))))
+         (tmp-src-file (org-babel-temp-file "python-"))
+         (session-body
+          ;; The real code we evaluate in the session.
+          (pcase result-type
+                  (`output
+                   (format (string-join
+                      (list "with open('%s') as f:\n"
+                            "    exec(compile(f.read(), f.name, 'exec'))\n"))
+                                             (org-babel-process-file-name
+                                              tmp-src-file 'noquote)))
+            (`value
+             ;; FIXME: In this case, any output is an error.
+                                           (org-babel-python-format-session-value
+                                tmp-src-file results-file result-params))))
+         comint-buffer
+         finally)
+
+
+    (unless session-key
+      ;; No session. We create a temporary one and use 'finally' to
+      ;; destroy it once we are done.
+      ;;
+      ;; FIXME: This session code should be refactored and moved into
+      ;;        ob-core.
+      (setq session-key (org-babel-python-initiate-session
+                        ;; We can't use a simple `generate-new-buffer'
+                        ;; due to the earmuffs game.
+                         (org-babel-python-without-earmuffs
+                         (format "*ob-python-no-session-%s*" (org-id-uuid)))))
+      (setq finally (lambda ()
+                      (when-let ((s-buf
+                                  (get-buffer (org-babel-python-with-earmuffs session-key))))
+                        ;; We cannot delete it immediately as we are called from it.
+                        (run-with-idle-timer
+                         0.1 nil
+                         (lambda ()
+                           (when (buffer-live-p s-buf)
+                             (let ((kill-buffer-query-functions nil)
+                                   (kill-buffer-hook nil))
+                               (kill-buffer s-buf)))))))))
+
+    (org-elib-async-comint-queue-init-if-needed session-key)
+    (setq comint-buffer
+          (get-buffer (org-babel-python-with-earmuffs session-key)))
+    (with-temp-file tmp-src-file
+      (insert (if (and graphics-file (eq result-type 'output))
+                  (format org-babel-python--output-graphics-wrapper
+                          full-body graphics-file)
+                full-body)))
+
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "import sys; sys.ps1=''; sys.ps2=''")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         ;; See org-babel-python-send-string
+         (with-current-buffer comint-buffer
+           (let ((python-shell-buffer-name
+                        (org-babel-python-without-earmuffs session-key)))
+                   (python-shell-send-string (concat code "\n")))))
+        (`(:get-code) session-body)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+
+(defun my-org-babel-schedule (lang body params handle-feedback)
+  "Schedule the execution of BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'.  Return a
+function that waits and returns the result on success, raise on failure."
+  (cl-assert (equal "python" lang))
+  (let ((exec (my-org-babel-python-how-to-execute body params)))
+    (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback)))
+
+
+(defun my-org-babel-execute (lang body params)
+  "Execute Python BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'."
+  ;; We just start the asynchronous execution, wait for it, and return
+  ;; the result (or raise the exception). No custom code, and,
+  ;; synchronous and asynchronous should just mix nicely together.
+  (cl-assert (equal "python" lang))
+  (funcall (my-org-babel-schedule lang body params nil)))
diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org
new file mode 100644
index 000000000..263ca77a2
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.org
@@ -0,0 +1,484 @@
+#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both
+* Intro
+
+An org document with code blocks to help test the proposed patches.
+
+You need to load:
+   #+begin_src elisp :results silent
+   (load-file "my-async-tests.el")
+   #+end_src
+
+
+
+Emacs and org versions:
+   #+begin_src elisp
+   (mapcar (lambda (sb) (list sb (symbol-value sb)))
+           '(emacs-version org-version))
+   #+end_src
+
+   #+RESULTS:
+   | emacs-version | 30.0.50 |
+   | org-version   | 9.7-pre |
+
+Note that we've disabled eval on export: export doesn't know it needs
+to wait for asynchronous results.
+
+* A simple bash example
+   :PROPERTIES:
+    :header-args:bash: :execute-with my-shell-babel :nasync yes
+   :END:
+
+The package `my-async-tests.el' contains the function
+`my-shell-babel-schedule' to evaluate shell script asynchronously.
+
+The header-args properties above request asynchronous execution for
+bash (:nasync yes), and, tells ob-core to use the prefix
+`my-shell-babel' when looking for functions to evaluate a source
+block. Thus, org will delegate execution to `my-shell-babel-schedule'.
+We don't have `my-shell-babel-execute', so, in this case, :nasync must
+be yes.
+
+A simple execution:
+    #+begin_src bash
+      date
+    #+end_src
+
+    #+RESULTS:
+    : Fri Feb 16 18:11:08 CET 2024
+
+A tricky computation takes some time:
+    #+begin_src bash
+      sleep 5; date
+    #+end_src
+
+    #+RESULTS:
+    : Fri Feb 16 17:58:23 CET 2024
+
+An example of a failure:
+    #+begin_src bash
+      sleepdd 1; false
+    #+end_src
+
+    #+RESULTS:
+
+* Python
+   :PROPERTIES:
+    :header-args:python:  :execute-with my-org-babel :nasync yes
+    :header-args:python+: :session py-async
+   :END:
+
+Used =header-args= properties:
+   - =:execute-with my-org-babel=:  look for functions with the prefix `my-org-babel' to execute
+     blocks (for the asynchronous case use
+     `my-org-babel-schedule', and, for the synchronous case
+     `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]].
+
+   - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule').
+
+   - =:session py-async= by default, use a session named "py-async".
+
+** basic examples
+*** async with a session
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103472.9s | 1708103473.9s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** async with no session
+   :PROPERTIES:
+    :header-args:python+: :session none
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, it failed, as expected. "import time" was done in its own
+temporary session.  The old result is preserved; the error is display
+as an overlay. Click on it to get more info about the error.
+
+
+Let's fix it, adding the import line:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102948.9s | 1708102949.9s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+
+*** sync with a session
+   :PROPERTIES:
+    :header-args:python+: :session py-sync-session :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102997.5s | 1708102998.5s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** sync with no session
+   :PROPERTIES:
+    :header-args:python+: :session none :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, that fails (no session), displaying the details in a popup. Let's
+fix it:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103039.0s | 1708103040.0s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+** worg examples
+
+Let's import matplotlib in our session.
+
+    #+begin_src python
+    import matplotlib
+    import matplotlib.pyplot as plt
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+A figure in a PDF, asynchronous case.
+    #+begin_src python :results file link
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-async.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-async.pdf]]
+
+
+A figure in a PDF, synchronous case.
+    #+begin_src python :results file link :nasync no
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-sync.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-sync.pdf]]
+
+
+
+A PNG figure, asynchronous case.
+    #+begin_src python :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+      fig
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, but using the =:return= keyword.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, asynchronous but without a session this time.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none
+      import matplotlib
+      import matplotlib.pyplot as plt
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot-no-sess-a-y.png]]
+
+
+Lists are table,
+    #+begin_src python
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    | 1 | 2 | 3 |
+
+unless requested otherwise.
+    #+begin_src python :results verbatim
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    : [1, 2, 3]
+
+
+Dictionaries are tables too.
+    #+begin_src python :results table
+      {"a": 1, "b": 2}
+    #+end_src
+
+    #+RESULTS:
+    | a | 1 |
+    | b | 2 |
+
+
+Let's try the example with Panda.
+    #+begin_src python :results none
+      import pandas as pd
+      import numpy as np
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+    #+begin_src python :results table
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+And the synchronous case?
+
+    #+begin_src python :results table  :nasync no
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+
+Without session ?
+
+    #+begin_src python :results table :session none
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+Right, we need to import the libraries (no session).
+
+    #+begin_src python :results table :session none
+    import pandas as pd
+    import numpy as np
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+** inline examples
+
+    A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}.
+
+    An other one containing a mistake src_python{2/0} {{{results(=6=)}}}
+    (click on the error to see the details).
+
+
+    Some very slow inline asynchronous computations that all run in
+    the same session. You need to execute the 3 of them at once. Here
+    is the first one src_python[:return "\"OK1\""]{import time;
+    time.sleep(5)} {{{results(=OK1=)}}} and a second one
+    src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)}
+    {{{results(=OK1 bis=)}}} and the third one src_python[:return
+    "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Yes, the previous paragraph is unreadable; it's on purpose, to
+    check that ob-core can figure it out.
+
+    Let's repeat, in a more readable way, and making the last one
+    synchronous.
+
+    Some very slow inline computations that all run in the same
+    session. Here is the first asynchronous one
+          src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}}
+    and a second one, asynchronous too:
+          src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}}
+    and finally, a third one, synchronous this one:
+          src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Note that, once the user executes the last synchronous block, the
+    user is blocked until the synchronous execution can start
+    (i.e. all previous asynchronous executions are done) and until
+    it's done.  The display is updated though, to see the asynchronous
+    progress.
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-16 17:52               ` Bruno Barbier
@ 2024-02-18 21:14                 ` Matt
  2024-02-19  0:31                   ` Jack Kamm
                                     ` (3 more replies)
  2024-02-19  0:15                 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
  2024-02-19  9:06                 ` Ihor Radchenko
  2 siblings, 4 replies; 35+ messages in thread
From: Matt @ 2024-02-18 21:14 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 7215 bytes --]


 ---- On Fri, 16 Feb 2024 18:52:22 +0100  Bruno Barbier 

 > Sorry for the late reply.  Cleaning the code took me longer than expected.

No need to apologize, we're all volunteers here :)

 > Feedbacks, corrections, critiques, etc are most welcome!

Thank you for sharing!

If I understand correctly, there are several independent topics the code addresses:

| topic            | manner addressed                               |
|------------------+------------------------------------------------|
| execution status | using overlays to communicate execution status |
| locating results | using overlays to locate results               |
| blocking         | making all execution asynchronous              |
|------------------+------------------------------------------------|

I suggest these be discussed in separate threads.

 > > The use of the overlay is a really cool idea!
 > >
 > > I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.
 > 
 > I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document).  The backend is free to transform a failure into a success to make that result available though.

You can disregard my hesitation on this point.  I had not run your code yet and had misunderstood how it worked.

Since this thread is dedicated to blocking, let me share my thoughts on that subject.

 > To execute Python blocks, using the proposed async API:
 > 
 >    - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python).
 >    
 >    - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...).
 >    
 >    - The "no session" case is just about creating a new unique session and throwing it away immediately.

This is an interesting idea, feeding all processes through the same mechanism.

Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].

Processes are synchronous or asynchronous.

Three primitives exist in Emacs for making processes:

1. make-process (asynchronous)
2. call-process (synchronous)
3. call-process-region (synchronous)

There exist several convenience wrappers for these.  AFAIK, everything reduces to these three primitives.  For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal.  This background-foreground distinction is called "job control".

Output from a process typically goes to a buffer.  This may be changed and instead handle output with a filter function.  =call-process= has an option to directly send output to a file.

Subprocesses inherent the =default-directory= and the environment from Emacs.  The environment may be changed using =process-environment=.

There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe".  The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.).

In my previous message, I divided evaluation into 4 types:

  - non-persistent vs. persistent 
  - synchronous vs. asynchronous

I find the approach of feeding everything through, fundamentally, =make-process= interesting because if we make a chart of the 4 types, we see some ambiguities:

|              | non-persistent | persistent   |
|--------------+----------------+--------------|
| synchronous  | call-process   | ???          |
|--------------+----------------+--------------|
| asynchronous | ???            | make-process |
|--------------+----------------+--------------|

To make a non-persistent asynchronous process, the first thing that comes to mind is =async-shell-command=.  However, as the code shows, another option is to use =make-process= and throw away the state (the process buffer).

I'm not sure how we could make a persistent, synchronous process.  Persistence is achieved, currently, by a process buffer.  Is there another way persistence may be achieved?  Of course, this ignores whether a persistent, synchronous process is even desirable.  Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation.

All that is mainly academic.  The idea I think most interesting is using a single primitive to handle all evaluation.

It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached).

It works as follows.  I defined a new Babel "language" called "blub".   Blub could be a shell, python, ruby, whatever.  I wanted to test that the implementation could work with different interpreters or compilers.  Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out.

Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the same directory.  Run the first block in ob-blub-test.org.  This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again.  Hopefully ob-blub.el is documented sufficiently for you to experiment.

The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.

Curious to hear other people's thoughts!

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[-- Attachment #2: ob-blub-test.org --]
[-- Type: application/octet-stream, Size: 2532 bytes --]

#+begin_src emacs-lisp :results silent :var HERE=(buffer-file-name)
;; load blub
(add-to-list 'load-path (file-name-directory HERE))
(require 'ob-blub)
(org-babel-do-load-languages 'org-babel-load-languages '((blub . t)))

;; reset uuid on failed block
(setq my-org-babel-comint--async-uuid nil)

;; configure shell
(setq org-babel-blub-interpreter shell-file-name)
(setq org-babel-blub-remove-prompt-command "PROMPT_COMMAND=;PS1=;PS2=;")
(setq org-babel-blub-output-start-delimiter "echo \"start_%s\"")
(setq org-babel-blub-output-end-delimiter "echo \"end_%s\"")
(setq org-babel-blub-interpreter-args '())

;; configure python
;; (setq org-babel-blub-interpreter "python3")
;; (setq org-babel-blub-remove-prompt-command "import sys;sys.ps1='';sys.ps2='';")
;; (setq org-babel-blub-output-start-delimiter "print(\"start_%s\")")
;; (setq org-babel-blub-output-end-delimiter "print(\"end_%s\")")
;; (setq org-babel-blub-interpreter-args '())

;; configure ruby
;; (setq org-babel-blub-interpreter "ruby")  ; for non-sessions
;; (setq org-babel-blub-interpreter "irb")  ; for sessions
;; (setq org-babel-blub-remove-prompt-command nil)
;; (setq org-babel-blub-output-start-delimiter "puts \"start_%s\"")
;; (setq org-babel-blub-output-end-delimiter "puts \"end_%s\"")
;; (setq org-babel-blub-interpreter-args '("--noprompt" "--noreadline" "--nomultiline"))
#+end_src

* Non-persistent
** shell
#+begin_src blub
echo "hello"
sleep 3
echo "world!"
#+end_src

#+RESULTS:
: hello
: world!

#+begin_src blub
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub
import time
print("hello without session")
time.sleep(3)
print("world")
#+end_src

** ruby
#+begin_src blub
puts "cruel world"
sleep(3)
puts "good-bye"
#+end_src

* Persistent
** shell
#+begin_src blub :session *shell-blubber*
echo "hello"
sleep 3
echo "world!"
#+end_src

#+begin_src blub :session *shell-blubber*
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub :session *python-blubber*
import time
print("hello")
time.sleep(5)
print("world")
#+end_src

#+RESULTS:

#+begin_src blub :session *python-blubber*
import time
print("good-bye")
time.sleep(5)
print("cruel world")
#+end_src

** ruby
#+begin_src blub :session *ruby-blubber*
puts "good-bye"
sleep(3)
puts "cruel world"
#+end_src

* Failures
#+begin_src blub :session *bash-blubber*
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
#+end_src

#+begin_src blub :session *shell-blubber* :epilogue echo "bye"
ssh $USER@localhost echo "hi"
#+end_src

[-- Attachment #3: ob-blub.el --]
[-- Type: application/octet-stream, Size: 10821 bytes --]

;; -*- lexical-binding: t -*-

;;; ob-blub.el --- org-babel functions for blub evaluation

;; Copyright (C) Matt Trzcinski

;; Author: Matt Trzcinski
;; Keywords: literate programming, reproducible research
;; Homepage: https://orgmode.org
;; Version: 0.01

;;; License:

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;;; Commentary:
;;
;; A new language, blub, is defined using the Org Babel API
;; (`org-babel-execute:blub').  "Blub" is a dummy language.  If blub
;; is interpreted, set the interpreter using
;; `org-babel-blub-interpreter'.  Compiled languages are not
;; demonstrated here although the implementation may be extended to
;; support compiled languages.
;;
;; The code is split into three sections:
;;
;;   1. Eval
;;   2. Comint
;;   3. Blub
;;
;; Sections are based on their current implements in Org Babel.  It's
;; not clear if these are the "right" categorizations.  For example,
;; should comint evaluation be in ob-comint or ob-eval?
;;
;; Set the following:
;;
;;   `org-babel-blub-interpreter'
;;   `org-babel-blub-interpreter-args'
;;   `org-babel-blub-remove-prompt-command'
;;   `org-babel-blub-output-start-delimiter'
;;   `org-babel-blub-output-end-delimiter'
;;
;; Sessions run in a dedicated process buffer.  Output is captured
;; from between delimiters.  This is similar to the current Babel
;; implementation.  However, unlike the current Babel implementation,
;; both delimiters in this proof of concept are custom.  The current
;; Babel implementation uses the prompt for the start delimiter which
;; causes problems when it changes.
;;
;; For sessions, only a single process may run at a time.  The UUID is
;; stored in `my-org-babel-comint--async-uuid' and the associated Org
;; buffer is stored in `my-org-babel-comint--async-org-buffer'.  If an
;; error occurs that prevents the end delimiter from printing,
;; manually the clear `my-org-babel-comint--async-uuid' to run blocks.
;;
;;   (setq my-org-babel-comint--async-uuid nil)
;;
;; Results are extracted from standard output using a regexp.
;;
;; Noted problems:
;;   - running sudo

;;; Requirements:
(require 'ob)

\f
;;; Eval:
(defun my-org-babel-eval-async (command body)
  "Start process with COMMAND, send BODY to process, get
results.

Since results execute asynchronously, a UUID is returned.  The
UUID is later replaced when the async process finishes."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))
  (let* ((my-process
          (make-process
           :name "my-org-babel-eval-async"
           :buffer "*my-org-babel-eval-async*"
           :command `(,command)
           :connection-type 'pipe
           :sentinel '(lambda (process msg)
                        (cond ((string= msg "finished\n")
                               (my-org-babel-comint-replace-uuid-with-results
                                my-org-babel-comint--async-org-buffer
                                my-org-babel-comint--async-uuid
                                (with-current-buffer (process-buffer process)
                                  (buffer-substring-no-properties (point-min) (point-max))))
                               (setq my-org-babel-comint--async-uuid nil)
                               (let ((kill-buffer-query-functions nil))
                                 (if (kill-buffer "*my-org-babel-eval-async*")
                                     (setq my-org-babel-comint--async-org-buffer nil)))
                               ))))))
    (process-send-string my-process body)
    (process-send-eof my-process))
  my-org-babel-comint--async-uuid)

\f
;;; Comint:
(defvar my-org-babel-comint--async-uuid nil
  "Placeholder for results.")
(defvar my-org-babel-comint--async-org-buffer nil
  "Buffer containing UUID.")

(defun my-org-babel-comint-send-string (session string)
  "Send STRING to comint SESSION."
  (with-current-buffer session
    (goto-char (process-mark (get-buffer-process session)))
    (insert string)
    (comint-send-input)))

(defun my-org-babel-comint-replace-uuid-with-results (buffer uuid results)
  "Replace UUID string in BUFFER with RESULTS string."
  (with-current-buffer buffer
    (save-excursion
      (goto-char (point-min))
      (when (search-forward uuid nil t)
        (org-babel-previous-src-block)
        (let* ((info (org-babel-get-src-block-info))
               (params (nth 2 info))
               (result-params
                (cdr (assq :result-params params))))
          (org-babel-insert-result
           results result-params info))))))

(defun my-org-babel-comint-send-to-session-async (process-buffer &rest body)
  "Send BODY to PROCESS-BUFFER asynchronously."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))

  (defun my-org-babel-comint--async-filter (text)
    "Check TEXT for ending delimiter and replace results held by
`my-org-babel-comint--async-uuid' placeholder."
    (cond ((string-match-p (format "end_%s" my-org-babel-comint--async-uuid) text)
           (remove-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

           ;; replace my-org-babel-comint--async-uuid in Org buffer
           (let ((results
                  (with-current-buffer process-buffer ; e.g. "*blubber*"
                    (goto-char (point-min))
                    (re-search-forward
                     ;; Of course, all the problems with regexp happen here.  The goal is getting
                     ;; the text between the delimiters.
                     ;;
                     ;; Some programs (guix shell?) may reset PS1.  So, we can't always match on
                     ;; start_uuid being at the very start of the line.  Match on the one that
                     ;; doesn't have the quote (that is, the result of the echo).
                     (format "[^\"]start_%s[\r\n]*\\(\\(.*[\r\n]+\\)*.*\\)end_%s$"
                             my-org-babel-comint--async-uuid   ; start
                             my-org-babel-comint--async-uuid)  ; end
                     nil nil 1)
                    (let ((match (match-string 1)))
                      (substring-no-properties
                       match)))))

             (my-org-babel-comint-replace-uuid-with-results
              my-org-babel-comint--async-org-buffer
              my-org-babel-comint--async-uuid
              results)

             (setq my-org-babel-comint--async-uuid nil)
             (setq my-org-babel-comint--async-org-buffer nil)))))

  (let* ((proc (get-buffer-process process-buffer)))
    (with-current-buffer process-buffer
      (add-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-start-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      (insert (car body))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-end-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)))

  my-org-babel-comint--async-uuid)

\f
;;; Blub:
(defvar org-babel-blub-interpreter
  shell-file-name  ; bash/sh
  ;; "irb"         ; ruby
  ;; "python3"     ; python
  "Blub interpreter command or executable.")

(defvar org-babel-blub-interpreter-args
  '()                                                ; bash
  ;; '("--noprompt" "--noreadline" "--nomultiline")  ; ruby
  "Blub interpreter command or executable arguments.")

(defvar org-babel-blub-remove-prompt-command
  "PROMPT_COMMAND=;PS1=;PS2=;"  ; bash
  ;; "import sys;sys.ps1='';sys.ps2='';"  ; python
  "Command(s) to remove interpreter prompt.")

(defvar org-babel-blub-output-start-delimiter
  "echo \"start_%s\""       ; bash
  ;; "puts \"start_%s\""    ; ruby
  ;; "print(\"start_%s\")"  ; python
  "Format expression for writing the start delimiter to standard
output in blub.")

(defvar org-babel-blub-output-end-delimiter
  "echo \"end_%s\""       ; bash
  ;; "puts \"end_%s\""    ; ruby
  ;; "print(\"end_%s\")"  ; python
  "Format expression for writing the end delimiter to standard
output in blub.")

;; this removes the need for `org-babel-comint-buffer-livep'
(defun org-babel-blub-get-session (session)
  "Return SESSION buffer, making process buffer if none exists."
  (cond ((not (get-buffer-process session))
         (apply
          #'make-comint-in-buffer
          `(,session                     ; process name
            ,session                     ; buffer name
            ,org-babel-blub-interpreter  ; program
            nil                          ; start file
            ,@org-babel-blub-interpreter-args))
         (org-babel-comint-wait-for-output session)
         (if org-babel-blub-remove-prompt-command
             (my-org-babel-comint-send-string
              session
              org-babel-blub-remove-prompt-command))

         ;; Needed for Emacs 23 since the marker is initially
         ;; undefined and the filter functions try to use it without
         ;; checking.
         (with-current-buffer session
           (set-marker comint-last-output-start (point)))

         ;; return shell buffer
         (get-buffer session))
        (t session)))

(defun org-babel-execute:blub (body params)
  "Execute BODY of Blub code with org-babel."
  (let* ((session (cdr (assq :session params))))
    (cond ((and session
                (not (string= session "none"))
                (org-babel-blub-get-session session))
           (if (not my-org-babel-comint--async-uuid)
               (my-org-babel-comint-send-to-session-async session body)
             ;; TODO prompt user to just go ahead with it (clobbering
             ;; the "existing" process)
             (error "Block already running.  Call `(setq my-org-babel-comint--async-uuid nil)' to run a new process")))
          (t (my-org-babel-eval-async org-babel-blub-interpreter (org-trim body))))))

(provide 'ob-blub)
;;; ob-blub.el ends here

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-16 17:52               ` Bruno Barbier
  2024-02-18 21:14                 ` Matt
@ 2024-02-19  0:15                 ` Jack Kamm
  2024-02-21 15:43                   ` Bruno Barbier
  2024-02-19  9:06                 ` Ihor Radchenko
  2 siblings, 1 reply; 35+ messages in thread
From: Jack Kamm @ 2024-02-19  0:15 UTC (permalink / raw)
  To: Bruno Barbier, Matt; +Cc: Ihor Radchenko, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> I'm not using it with official org backends (yet).  I'm using it with
> several custom backends that I'm working on.  One of the backend
> delegate the block executions to emacs subprocesses: so I have a kind of
> asynchronous executions for free for any language, including elisp
> itself.

For sessions, wouldn't running in a subprocess prevent the user from
directly interacting with the REPL outside of Org? If so, that's a
problem. Org-babel sessions need to play nicely with inferior Python,
inferior ESS, and other interactive comint modes.

> So, here we go.  You'll find attach a set of patchs.  It works for me with
> Emacsc 30.50 and 9.7-pre (from today).

I suggest to keep these patches on a public branch somewhere, see:
https://orgmode.org/worg/org-contribute.html#patches

  "When discussing important changes, it is sometimes not so useful to
  send long and/or numerous patches.
  
  In this case, you can maintain your changes on a public branch of a
  public clone of Org and send a link to the diff between your changes
  and the latest Org commit that sits in your clone."

I tried running your example on emacs29 using

  emacs -q -L  /path/to/org-mode/lisp my-async-tests.org

but it fails with the error below. Also "make" gives a bunch of
compilation warnings (which I've put at the bottom).

> +You need to load:
> +   #+begin_src elisp :results silent
> +   (load-file "my-async-tests.el")
> +   #+end_src

This raises the following error in *Org-Babel Error Output*

  void-variable
  (org-elib-async-process)
  [ Babel evaluation exited with code 127 ]

All the subsequent blocks don't work because of that, for example:

> +A simple execution:
> +    #+begin_src bash
> +      date
> +    #+end_src

yields:

  org-babel-execute-src-block: No org-babel-execute function for bash: my-shell-babel-schedule!

Finally here are the warnings when running "make":

Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/ob-core.el...

In org-babel--async-feedbacks:
ob-core.el:851:2: Warning: docstring has wrong usage of unescaped single quotes (use \= or different quoting)
ob-core.el:871:9: Warning: Unused lexical variable `result-indent'
ob-core.el:906:18: Warning: Unused lexical variable `header'

Compiling single /home/jack/src/org-mode/2024-02-brubar-async/lisp/org-elib-async.el...

In toplevel form:
org-elib-async.el:52:11: Warning: reference to free variable ‘org-elib-async-process’
org-elib-async.el:52:43: Warning: reference to free variable ‘&key’
org-elib-async.el:52:48: Warning: reference to free variable ‘input’
org-elib-async.el:52:54: Warning: reference to free variable ‘callback’
org-elib-async.el:78:29: Warning: reference to free variable ‘command’
org-elib-async.el:127:9: Warning: Variable ‘last-elapsed’ left uninitialized

In org-elib-async-wait-condition:
org-elib-async.el:137:12: Warning: ‘signal’ called with 3 arguments, but accepts only 2

In end of data:
org-elib-async.el:262:16: Warning: the function ‘org-id-uuid’ is not known to be defined.
org-elib-async.el:52:35: Warning: the function ‘command’ is not known to be defined.
org-elib-async.el:52:2: Warning: the function ‘cl-defun’ might not be defined at runtime.



^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-18 21:14                 ` Matt
@ 2024-02-19  0:31                   ` Jack Kamm
  2024-02-20 10:28                   ` Ihor Radchenko
                                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 35+ messages in thread
From: Jack Kamm @ 2024-02-19  0:31 UTC (permalink / raw)
  To: Matt, Bruno Barbier; +Cc: Ihor Radchenko, emacs-orgmode

Matt <matt@excalamus.com> writes:

> The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.
>
> Curious to hear other people's thoughts!

I don't remember the details, but my past self [1] thought it would be
good to find a way to replace `process-file' with `make-process' in
`org-babel--shell-command-on-region' or `org-babel-eval', and it seems
you are thinking along those lines with `my-org-babel-eval-async'. Hope
you're able to make progress on this and get the improvements into
ob-eval.el eventually.

[1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-16 17:52               ` Bruno Barbier
  2024-02-18 21:14                 ` Matt
  2024-02-19  0:15                 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
@ 2024-02-19  9:06                 ` Ihor Radchenko
  2024-02-19 19:47                   ` Matt
  2024-02-21 16:03                   ` Bruno Barbier
  2 siblings, 2 replies; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-19  9:06 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Matt, Jack, Ihor,
>
> Sorry for the late reply.  Cleaning the code took me longer than
> expected.

Thanks for the code!
It is a lot more that I expected.
I have many questions ;)

> The new API itself is more about how to wait for and display one block
> result.  So, it's not really aware of sessions.

I think that it is important to think about sessions. For non-sessions,
we may execute the code in parallel, disregarding already running
execution. In contrast, for sessions, we need to maintain some queue and
wait until previous execution finishes before running next (if multiple
session blocks are executed in quick succession).

It may also be necessary to provide an UI listing the queued and running
execution. Users should be able to stop "stale" processes if they are
defunc (consider infinite loop).

>> Also interesting, I think it's worth exploring/testing this overlay idea
>> out. Does that mean that output is asynchronously printing into the Org
>> buffer? It sounds cool but I wonder if it might cause problems while
>> trying to edit another part of the buffer.
>
> Currently, I've limited the progress feedback to fit on one line to
> avoid anoying vertical display jumps.  When the computation is
> successful, the result is inserted as usual, and, that may be annoying;
> when it updates a previous result of the same height, it's ok.  Else, we
> could fold it to stay on one line too (using the overlay), until the
> user explicitly request to see it.

Let's not worry about "jumps" yet. It is a minor issue that can be
easily solved.

What is more important is when users, for example, remove the whole
subtree containing the place where the execution result is to be
inserted. Or when users perform edits at or around that place where the
result is to be inserted. Or consider subtree with pending result
refiled elsewhere (different file or different place in the same file);
or archived.

Or consider user running an src block, quickly editing it, and then
running again. What should we do then? Should we stop the first running
process?

> So, here we go.  You'll find attach a set of patchs.  It works for me with
> Emacsc 30.50 and 9.7-pre (from today).

I have several general questions:

- what if append/prepend to result asynchronously?
- what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
- on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
  Example: try running
  #+begin_src bash
  idontexist
  #+end_src

> I didn't check yet that the code and the commits follow all the
> guidelines.  This is just a preliminary version for feedbacks.
> Corrections/critiques are welcome, but don't check the details until I
> check them myself.
> So, I decided to rewrite the whole thing, taking code from the
> synchronous case (following `org-babel-python-evaluate-session').  I
> also created a package that contains all the functions that should be
> reusable for any language.  The patch [1] adds some generic functions to
> help dealing with asynchronicity (process, comint, etc.). The patch [2]
> shows a new possible way to execute python code blocks, both
> synchronously and asynchronously, with or without sessions.  You should
> just need to open [3] and follow what's written there, and execute the
> existing bash and python code blocks.

Note that we already have a WIP an asynchronous API in the works.
Check out `org-async-call' in
https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
If possible, we should reuse it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19  9:06                 ` Ihor Radchenko
@ 2024-02-19 19:47                   ` Matt
  2024-02-19 20:10                     ` Ihor Radchenko
                                       ` (2 more replies)
  2024-02-21 16:03                   ` Bruno Barbier
  1 sibling, 3 replies; 35+ messages in thread
From: Matt @ 2024-02-19 19:47 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode

 ---- On Mon, 19 Feb 2024 01:31:42 +0100  Jack Kamm  wrote --- 

 > I don't remember the details, but my past self [1] thought it would be
 > good to find a way to replace `process-file' with `make-process' in
 > `org-babel--shell-command-on-region' or `org-babel-eval', and it seems
 > you are thinking along those lines with `my-org-babel-eval-async'. Hope
 > you're able to make progress on this and get the improvements into
 > ob-eval.el eventually.
 > 
 > [1] https://list.orgmode.org/871rczg7bi.fsf@gmail.com/#t

AFAIK, aside from appending "&", =make-process= is the only way.  It would probably make sense to continue using =shell= though.  If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer?

 ---- On Mon, 19 Feb 2024 10:03:36 +0100  Ihor Radchenko  wrote --- 

 > Note that we already have a WIP an asynchronous API in the works.
 > Check out `org-async-call' in
 > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
 > If possible, we should reuse it.

:O

- What's the status? 
- Anything I could help with?  
- Are there any notes or supplements that go along with it?

> I have several general questions:
>
> - what if append/prepend to result asynchronously?
> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
> Example: try running
> #+begin_src bash
> idontexist
> #+end_src

Are these open questions for the `org-async-call' implementation?

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode



^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19 19:47                   ` Matt
@ 2024-02-19 20:10                     ` Ihor Radchenko
  2024-02-20  8:32                     ` Ihor Radchenko
  2024-02-20 17:04                     ` Jack Kamm
  2 siblings, 0 replies; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-19 20:10 UTC (permalink / raw)
  To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode

Matt <matt@excalamus.com> writes:

>  > Note that we already have a WIP an asynchronous API in the works.
>  > Check out `org-async-call' in
>  > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
>  > If possible, we should reuse it.
>
> :O
>
> - What's the status? 

Finalizing.

> - Anything I could help with?

Check https://github.com/tecosaur/org-latex-preview-todos/issues/

> - Are there any notes or supplements that go along with it?

Docstring should detail the API.

>> I have several general questions:
>>
>> - what if append/prepend to result asynchronously?
>> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?
>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>> Example: try running
>> #+begin_src bash
>> idontexist
>> #+end_src
>
> Are these open questions for the `org-async-call' implementation?

No. These are about the patch. And have nothing to do with
`org-async-call', which is just a way to call and queue async processes.
`org-async-call' was written for async latex preview, but its interface
should be generic enough to be reused for babel.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19 19:47                   ` Matt
  2024-02-19 20:10                     ` Ihor Radchenko
@ 2024-02-20  8:32                     ` Ihor Radchenko
  2024-02-20 17:04                     ` Jack Kamm
  2 siblings, 0 replies; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-20  8:32 UTC (permalink / raw)
  To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode

Matt <matt@excalamus.com> writes:

> - Anything I could help with?  

https://github.com/tecosaur/org-latex-preview-todos/issues/21
It looks like the main remaining blocker is the lack of LaTeX snippet
export for ox-odt.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-18 21:14                 ` Matt
  2024-02-19  0:31                   ` Jack Kamm
@ 2024-02-20 10:28                   ` Ihor Radchenko
  2024-02-20 10:46                     ` tomas
  2024-02-21 15:27                   ` Bruno Barbier
       [not found]                   ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
  3 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-20 10:28 UTC (permalink / raw)
  To: Matt; +Cc: Bruno Barbier, Jack Kamm, emacs-orgmode

Matt <matt@excalamus.com> writes:

> The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

Doing everything asynchronously is not always desired.
Consider, for example,

#+begin_src bash
echo "Contents" > /tmp/tmpfile
#+end_src bash

#+begin_src bash
cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
#+end_src

Also, using "print" statements to create output delimiters might
sometimes be tricky.
If I remember correctly Ruby repl is asynchronous and sometimes produces
unexpected artefacts when you send multiple commands in quick
succession. The commands may be executed in different order than you may expect.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-20 10:28                   ` Ihor Radchenko
@ 2024-02-20 10:46                     ` tomas
  2024-02-20 11:00                       ` Ihor Radchenko
  0 siblings, 1 reply; 35+ messages in thread
From: tomas @ 2024-02-20 10:46 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Tue, Feb 20, 2024 at 10:28:06AM +0000, Ihor Radchenko wrote:
> Matt <matt@excalamus.com> writes:
> 
> > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).
> 
> Doing everything asynchronously is not always desired.
> Consider, for example,
> 
> #+begin_src bash
> echo "Contents" > /tmp/tmpfile
> #+end_src bash
> 
> #+begin_src bash
> cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
> #+end_src

You "just" [1] need a way of stating dependencies :-)

Cheers

[1] In quotes, because this opens a vast space of interesting
   and strange worlds. The functional folks have their monads,
   the compiler backend builders have their dependency graphs.
   Definitely doable, but treading with care will be helpful
   to not step into a mess :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-20 10:46                     ` tomas
@ 2024-02-20 11:00                       ` Ihor Radchenko
  2024-02-20 11:03                         ` tomas
  0 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-20 11:00 UTC (permalink / raw)
  To: tomas; +Cc: emacs-orgmode

<tomas@tuxteam.de> writes:

>> Doing everything asynchronously is not always desired.
>> Consider, for example,
>> 
>> #+begin_src bash
>> echo "Contents" > /tmp/tmpfile
>> #+end_src bash
>> 
>> #+begin_src bash
>> cat /tmp/tmpfile # I must run after /tmp/tmpfile is created!
>> #+end_src
>
> You "just" [1] need a way of stating dependencies :-)
>
> Cheers
>
> [1] In quotes, because this opens a vast space of interesting
>    and strange worlds. The functional folks have their monads,
>    the compiler backend builders have their dependency graphs.
>    Definitely doable, but treading with care will be helpful
>    to not step into a mess :-)

Let's not jump into this rabbit hole yet before we have async code
working for less complicated scenarios.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-20 11:00                       ` Ihor Radchenko
@ 2024-02-20 11:03                         ` tomas
  0 siblings, 0 replies; 35+ messages in thread
From: tomas @ 2024-02-20 11:03 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Tue, Feb 20, 2024 at 11:00:28AM +0000, Ihor Radchenko wrote:
> <tomas@tuxteam.de> writes:

[...]

> > You "just" [1] need a way of stating dependencies :-)

[...]

> Let's not jump into this rabbit hole yet before we have async code
> working for less complicated scenarios.

:-)

My take is: it's never too early to *think* about it (and to have
a look what others are doing), but it might yet be too early to
design something.

Other than that... rabbit holes is what makes things interesting.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19 19:47                   ` Matt
  2024-02-19 20:10                     ` Ihor Radchenko
  2024-02-20  8:32                     ` Ihor Radchenko
@ 2024-02-20 17:04                     ` Jack Kamm
  2 siblings, 0 replies; 35+ messages in thread
From: Jack Kamm @ 2024-02-20 17:04 UTC (permalink / raw)
  To: Matt, Ihor Radchenko; +Cc: Bruno Barbier, emacs-orgmode

Matt <matt@excalamus.com> writes:

> AFAIK, aside from appending "&", =make-process= is the only way.  It would probably make sense to continue using =shell= though.  If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer?

Yes, I typically use ob-R and ob-python in this way. Aside from the
convenience of interacting directly with the REPL, the comint mode may
provide other useful facilities, for example in ob-R the inferior ESS
mode provides completion that is aware of the objects in the current
session.

However I think modifying `org-babel-eval' will mainly be useful to
provide async to the nonsession users. Comint-based sessions mainly use
functionality from ob-comint.el, not ob-eval.el, and I think that the
use cases are different enough that they shouldn't be merged into a
single implementation.


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-18 21:14                 ` Matt
  2024-02-19  0:31                   ` Jack Kamm
  2024-02-20 10:28                   ` Ihor Radchenko
@ 2024-02-21 15:27                   ` Bruno Barbier
       [not found]                   ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
  3 siblings, 0 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-21 15:27 UTC (permalink / raw)
  To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode


Hi Matt,

Thanks for your answer.

Matt <matt@excalamus.com> writes:

> ...
> If I understand correctly, there are several independent topics the code addresses:
>
> | topic            | manner addressed                               |
> |------------------+------------------------------------------------|
> | execution status | using overlays to communicate execution status |
> | locating results | using overlays to locate results               |
> | blocking         | making all execution asynchronous              |
> |------------------+------------------------------------------------|
>
> I suggest these be discussed in separate threads.

Actually my patch is only about reporting execution status and
inserting results, in a generic way, in ob-core.  I provided some demo
code on top of them, just to show how it could be used.

Sorry about the confusion.  

> ...
>
> Since this thread is dedicated to blocking, let me share my thoughts on that subject.

I guess I should start a new thread then :)


> ...
>
>
> Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].
>
> Processes are synchronous or asynchronous.
>
> Three primitives exist in Emacs for making processes:
>
> 1. make-process (asynchronous)
> 2. call-process (synchronous)
> 3. call-process-region (synchronous)
>
> There exist several convenience wrappers for these.  AFAIK, everything reduces to these three primitives.  For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal.  This background-foreground distinction is called "job control".
>

> Output from a process typically goes to a buffer.  This may be changed and instead handle output with a filter function.  =call-process= has an option to directly send output to a file.
>

I've reached the same conclusion. As I wanted one simple
implementation, I was thinking about using 'make-process', but, as we
have to deal with the output manually and use a process sentinel, and,
as offering a REPL to the user is nice, my personal pick is to use a
plain shell with comint.

> Subprocesses inherent the =default-directory= and the environment from Emacs.  The environment may be changed using =process-environment=.
>

One wonderful thing about using =default-directory= is that, if done
right, tramp will automatically handle executions on one or more remote
computers.

> There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe".  The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.).

I'm personally ignoring the difference between pty and pipe: as I
don't use the terminal features, it doesn't really matter to me.


> ...
> I'm not sure how we could make a persistent, synchronous process.  Persistence is achieved, currently, by a process buffer.  Is there another way persistence may be achieved?

If I understand correcly, "persistent" here means to preserve some
state between executions.  They are definitely other way to preserve
and provide a state.

If you focus on processes only, it's probably OK. But, if you use
threads or some other kinds of asynchronicity, that might be too
limiting to require a buffer and a process.


> Of course, this ignores whether a persistent, synchronous process is even desirable.  Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation.
>

I'm not so sure.  If the success of an execution is really important,
and, intrinsically unreliable (like a network connection), and, it
only takes a few seconds, I'll probably choose synchronous execution,
just to see that everything is OK, before moving to my next task.



> ...
> Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the same directory.  Run the first block in ob-blub-test.org.  This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again.  Hopefully ob-blub.el is documented sufficiently for you to experiment.

Thanks for sharing.  I didn't have time to try it yet.  But it looks
like you could use the 'execute-with' keyword that my patchs provide.
That way, you may redirect evaluation to use your "blub"
implementation, and still use the real language name for your code
blocks (that way, org knows how to fontify them, edit them, etc).  And, you
will not have to manually modify and reevaluate your elisp when
switching languages.

> The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

I'll try it. Thanks,


> The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.
>
> Curious to hear other people's thoughts!

One think that comes to mind is to check that our testsuite covers
everything we might break.

If 'execute-with' gets added to org (from my patchs), that may provide
an easy way to test/compare several execution engines.  I'm thinking
about modifying my patch to provide a 'feedbacks-with' keyword so that
we can also test/compare different ways to report the execution status
and outcome.

Bruno



^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19  0:15                 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
@ 2024-02-21 15:43                   ` Bruno Barbier
  0 siblings, 0 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-21 15:43 UTC (permalink / raw)
  To: Jack Kamm, Matt; +Cc: Ihor Radchenko, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

Hi Jack,

Jack Kamm <jackkamm@gmail.com> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> I'm not using it with official org backends (yet).  I'm using it with
>> several custom backends that I'm working on.  One of the backend
>> delegate the block executions to emacs subprocesses: so I have a kind of
>> asynchronous executions for free for any language, including elisp
>> itself.
>
> For sessions, wouldn't running in a subprocess prevent the user from
> directly interacting with the REPL outside of Org?

Good point. The REPL should be created in the same subprocess; the
REPL display and interaction must happen in the user main emacs.  If
the REPL is based on comint, it should be relatively easy to
implement.


> If so, that's a problem. Org-babel sessions need to play nicely with
> inferior Python, inferior ESS, and other interactive comint modes.

With this solution, the user and the REPL/execution will be in
separate processes; so there will be disavantages.  For basic
interactions, mostly based on text input/output, it should work well.


>> So, here we go.  You'll find attach a set of patchs.  It works for me with
>> Emacsc 30.50 and 9.7-pre (from today).
>
> I suggest to keep these patches on a public branch somewhere, see:
> https://orgmode.org/worg/org-contribute.html#patches
>
>   "When discussing important changes, it is sometimes not so useful to
>   send long and/or numerous patches.
>   
>   In this case, you can maintain your changes on a public branch of a
>   public clone of Org and send a link to the diff between your changes
>   and the latest Org commit that sits in your clone."

Good point.  I'll switch to such a solution as soon as possible.


> I tried running your example on emacs29 using
>
>   emacs -q -L  /path/to/org-mode/lisp my-async-tests.org
>
> but it fails with the error below. Also "make" gives a bunch of
> compilation warnings (which I've put at the bottom).
> ...

My bad: I should have compiled the demo code in a standalone emacs.
I forgot to require some libraries: cl-lib and org-id.

I've now tested with your command line (thanks).  It should now
work. Sorry about that.


> Finally here are the warnings when running "make":
I should have fixed everything; no more (new) warnings.
Thanks!


Please find attached the new set of patchs.  I'll switch to using a
clone and a branch soon, in case if you prefer to wait.

Thanks again!


Bruno




[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 1367 bytes --]

From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:31:36 +0100
Subject: [PATCH 1/8] ob-core async: Add faces [1/5]

lisp/org-faces.el (org-async-scheduled, org-async-pending,
org-async-failure): new faces
---
 lisp/org-faces.el | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 0e20de51a..5a8a8fd51 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun
   "Face used for clock display for overrun tasks in mode line."
   :group 'org-faces)
 
+(defface org-async-scheduled '((t :inherit org-tag :background "gray"))
+  "Face for babel results for code blocks that are scheduled for execution."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-pending '((t :inherit org-checkbox :background "dark orange"))
+  "Face for babel results for code blocks that are running."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-failure '((t :inherit org-warning))
+  "Face for babel results for code blocks that have failed."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
 (provide 'org-faces)
 
 ;;; org-faces.el ends here
-- 
2.43.0


[-- Attachment #3: 0001-ob-core-async-Add-faces-1-5.patch --]
[-- Type: text/x-patch, Size: 10343 bytes --]

From 9f135bd5e8e153323bed5a3274851fa78f246b83 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:00 +0100
Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5]

---
 lisp/ob-core.el | 213 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 213 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bfeac257b..d98626fe8 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -792,6 +792,219 @@ (defun org-babel-session-buffer (&optional info)
     (when (org-babel-comint-buffer-livep buffer-name)
       buffer-name)))
 
+(defun org-babel--async-status-face (status)
+  (pcase status
+    (:scheduled 'org-async-scheduled)
+    (:pending   'org-async-pending)
+    (:failure   'org-async-failure)
+    (:success   nil)
+    (_ (error "Not a status"))
+    ))
+
+(defun org-babel--async-make-overlay (beg end)
+  "Create an overlay between positions BEG and END and return it."
+  (let ((overlay (make-overlay beg end))
+        (read-only
+	 (list
+	  (lambda (&rest _)
+	    (user-error
+	     "Cannot modify an area being updated"))))
+        )
+    (cl-flet ((make-read-only
+               (ovl)
+               (overlay-put ovl 'modification-hooks read-only)
+               (overlay-put ovl 'insert-in-front-hooks read-only)
+               (overlay-put ovl 'insert-behind-hooks read-only))
+              )
+      (overlay-put overlay 'org-babel--async-type 'org-babel--async-note)
+      (overlay-put overlay 'face 'secondary-selection)
+      (overlay-put overlay 'help-echo "Pending src block result...")
+      (make-read-only overlay)
+      overlay)))
+
+(defun org-babel--async-result-region (inline-elem &optional info)
+  "Return the region of the results, for the source block at point."
+  (unless info (setq info (org-babel-get-src-block-info)))
+  (save-excursion
+    (when-let ((res-begin  (org-babel-where-is-src-block-result nil info)))
+      (cons res-begin
+            (save-excursion
+              (goto-char res-begin)
+              (if inline-elem
+                  ;; Logic copy/pasted from org-babel-where-is-src-block-result.
+	          (let ((result (org-element-context)))
+		    (and (org-element-type-p result 'macro)
+		         (string= (org-element-property :key result)
+			          "results")
+		         (progn
+			   (goto-char (org-element-end result))
+			   (skip-chars-backward " \t")
+			   (point))))
+                ;; Logic copy/pasted from hide-result
+                (beginning-of-line)
+                (let ((case-fold-search t))
+                  (unless (re-search-forward org-babel-result-regexp nil t)
+	            (error "Not looking at a result line")))
+                (org-babel-result-end)
+                ))))))
+
+(defun org-babel--async-feedbacks (info handle-result
+                                       result-params exec-start-time)
+  "Flag the result as \='scheduled\=' and return how to handle feedbacks.
+
+Use overlays to report progress and status to the user.  Do not delete
+the existing result unless a new one is available.  When the result is
+available, remove the async overlays and insert the result as usual,
+like for a synchronous result.  In case of failure, use an overlay to
+report the error.
+
+The returned function handles 3 types of feedbacks:
+   - (:success R):   Evaluation is successful; result is R.
+   - (:failure ERR): Evaluation failed; error is ERR.
+   - (:pending P): Outcome still pending; current progress is P."
+  ;; FIXME: INFO CMD ... Nothing is used but handle-result here !!
+  (let (;; copy/pasted from org-babel-insert-result
+        (inline-elem (let ((context (org-element-context)))
+		       (and (memq (org-element-type context)
+			          '(inline-babel-call inline-src-block))
+			    context))))
+    (cl-labels
+        ((eot-point (start)
+           "Move to End Of Title after START"
+           (if inline-elem
+               (org-element-end inline-elem)
+             (save-excursion (goto-char start)
+                             (forward-line 1) (point))))
+         (after-indent (pt)
+           "Move after indentation, starting at PT."
+           (save-excursion (goto-char pt) (re-search-forward "[[:blank:]]*")))
+         (mk-result-overlays ()
+           ;; Make 2 overlays to handle the pending result: one title
+           ;; (first line) and one for the body.
+           (pcase-let ((`(,start . ,end) (org-babel--async-result-region
+                                          inline-elem info)))
+             (let ((anchor-end (eot-point start)))
+               (cons (org-babel--async-make-overlay
+                      (after-indent start)
+                      (1- anchor-end))
+                     (org-babel--async-make-overlay
+                      anchor-end end)))))
+         (add-style (status txt)
+           ;; Add the style matching STATUS over the text TXT.
+           (propertize txt 'face (org-babel--async-status-face status)))
+
+         (short-version-of (msg)
+           ;; Compute the short version of MSG, to display in the header.
+           ;; Must return a string.
+           (if msg
+               (car (split-string (format "%s" msg) "\n" :omit-nulls))
+             ""))
+         (update (ovl-title status msg)
+           ;; Update the title overlay to match STATUS and MSG.
+             (overlay-put ovl-title
+                          'face
+                          (org-babel--async-status-face status))
+             (overlay-put ovl-title
+                          'before-string (pcase status
+                                           (:scheduled "⏱")
+                                           (:pending "⏳")
+                                           (:failure "❌")
+                                           (:success "✔️")))
+             (overlay-put ovl-title
+                          'after-string
+                          (propertize (format " |%s|"
+                                              (if (eq :failure status)
+                                                  (if (consp msg) (car msg)
+                                                    (format "%s" msg))
+                                                  (short-version-of msg)))
+                                      'face (org-babel--async-status-face status))))
+         (remove-previous-overlays ()
+           ;; Remove previous title and body overlays.
+           (mapc (lambda (ovl)
+                   (when (eq 'org-babel--async-note
+                             (overlay-get ovl 'org-babel--async-type))
+                     (delete-overlay ovl)))
+                 (when-let ((region (org-babel--async-result-region
+                                     inline-elem info)))
+                   ;; Not sure why, but we do need to start before
+                   ;; point min, else, in some cases, some overlays
+                   ;; are not found.
+                   (overlays-in (max (1- (car region)) (point-min))
+                                (cdr region))))))
+
+      (remove-previous-overlays)
+
+      ;; Ensure there is a non-empty region for the result.
+      (save-excursion
+         (unless (org-babel-where-is-src-block-result (not inline-elem) nil nil)
+          (org-babel-insert-result
+           ;; Use " " for the empty result. That cannot be nil, else it's interpreted
+           ;; as a list. We need at least one char, to separate markers if any.
+           " \n"
+           result-params
+           info nil
+           (nth 0 info) ; lang
+           exec-start-time
+           )))
+
+      ;; Create the overlays that span the result title and its body.
+      (pcase-let ((`(,title-ovl . ,body-ovl) (mk-result-overlays)))
+        ;; Flag the result as ":scheduled".
+        (update title-ovl :scheduled nil)
+
+        ;; The callback, that runs in the org buffer at point.
+        (let ((buf (current-buffer))
+              (pt  (point-marker)))
+          (lambda (feedback)
+            (message "ob-core: Handling outcome at %s@%s: %s" pt buf feedback)
+            (with-current-buffer buf
+              (save-excursion
+                (goto-char pt)
+                (pcase feedback
+                  (`(:success ,r)
+                   ;; Visual beep that the result is available.
+                   (update title-ovl :success r)
+                   (sit-for 0.2)
+                   ;; We remove all overlays and let org insert the result
+                   ;; as it would in the synchronous case.
+                   (delete-overlay title-ovl)
+                   (delete-overlay body-ovl)
+                   (funcall handle-result r))
+
+                  (`(:pending ,r)
+                   ;; Still waiting for the outcome. Update our
+                   ;; overlays with the progress info R.
+                   (message "Updating block at %s@%s" pt buf)
+                   (update title-ovl :pending r))
+
+                  (`(:failure ,err)
+                   ;; We didn't get a result. We update our overlays
+                   ;; to report that failure. And unlock the old
+                   ;; result.
+                   (overlay-put title-ovl 'face nil)
+                   (update title-ovl :failure err)
+                   (delete-overlay body-ovl))
+
+                  (_ (error "Invalid outcome"))
+                  )
+                ))
+            nil))))))
+
+
+(cl-defun org-babel--async-p (params &key default)
+  "Return a non-nil value when the execution is asynchronous.
+Get the value of the :nasync argument and convert it."
+  (if-let ((binding (assq :nasync params)))
+      (pcase (cdr binding)
+        ((pred (not stringp))
+         (error "Invalid value for :nasync argument"))
+        ((or "no" "n")  nil)
+        ((or "yes" "y") t)
+        (_ (error "Invalid value for :nasync argument")))
+    default))
+
+
+
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
   "Execute the current source code block and return the result.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-ob-core-async-Refactor-handle-result-3-5.patch --]
[-- Type: text/x-patch, Size: 5958 bytes --]

From b0cdd3f5a9bd6e4e72adeac91f968741da95f98f Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:22 +0100
Subject: [PATCH 3/8] ob-core async: Refactor handle-result [3/5]

lisp/ob-core.el (org-babel-execute-src-block): Refactor the code to
prepare for the next change: move the part handling the result in its
own function `handle-result'.
---
 lisp/ob-core.el | 105 ++++++++++++++++++++++++------------------------
 1 file changed, 53 insertions(+), 52 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index d98626fe8..d1adba61c 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1086,7 +1086,58 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
-		 result exec-start-time)
+                 (exec-start-time (current-time))
+                 (handle-result
+                  (lambda (result)
+                    (setq result
+		          (if (and (eq (cdr (assq :result-type params)) 'value)
+			           (or (member "vector" result-params)
+				       (member "table" result-params))
+			           (not (listp result)))
+		              (list (list result))
+		            result))
+	            (let ((file (and (member "file" result-params)
+			             (cdr (assq :file params)))))
+		      ;; If non-empty result and :file then write to :file.
+		      (when file
+		        ;; If `:results' are special types like `link' or
+		        ;; `graphics', don't write result to `:file'.  Only
+		        ;; insert a link to `:file'.
+		        (when (and result
+			           (not (or (member "link" result-params)
+				            (member "graphics" result-params))))
+		          (with-temp-file file
+		            (insert (org-babel-format-result
+			             result
+			             (cdr (assq :sep params)))))
+		          ;; Set file permissions if header argument
+		          ;; `:file-mode' is provided.
+		          (when (assq :file-mode params)
+		            (set-file-modes file (cdr (assq :file-mode params)))))
+		        (setq result file))
+		      ;; Possibly perform post process provided its
+		      ;; appropriate.  Dynamically bind "*this*" to the
+		      ;; actual results of the block.
+		      (let ((post (cdr (assq :post params))))
+		        (when post
+		          (let ((*this* (if (not file) result
+				          (org-babel-result-to-file
+				           file
+				           (org-babel--file-desc params result)
+                                           'attachment))))
+		            (setq result (org-babel-ref-resolve post))
+		            (when file
+			        (setq result-params (remove "file" result-params))))))
+	              (unless (member "none" result-params)
+		        (org-babel-insert-result
+		         result result-params info
+                         ;; append/prepend cannot handle hash as we accumulate
+                         ;; multiple outputs together.
+                         (when (member "replace" result-params) new-hash)
+                         lang
+                         (time-subtract (current-time) exec-start-time)))
+	              (run-hooks 'org-babel-after-execute-hook)
+                      result))))
 	    (unless (fboundp cmd)
 	      (error "No org-babel-execute function for %s!" lang))
 	    (message "Executing %s %s %s..."
@@ -1101,57 +1152,7 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-	    (setq exec-start-time (current-time)
-                  result
-		  (let ((r (save-current-buffer (funcall cmd body params))))
-		    (if (and (eq (cdr (assq :result-type params)) 'value)
-			     (or (member "vector" result-params)
-				 (member "table" result-params))
-			     (not (listp r)))
-			(list (list r))
-		      r)))
-	    (let ((file (and (member "file" result-params)
-			     (cdr (assq :file params)))))
-	      ;; If non-empty result and :file then write to :file.
-	      (when file
-		;; If `:results' are special types like `link' or
-		;; `graphics', don't write result to `:file'.  Only
-		;; insert a link to `:file'.
-		(when (and result
-			   (not (or (member "link" result-params)
-				  (member "graphics" result-params))))
-		  (with-temp-file file
-		    (insert (org-babel-format-result
-			     result
-			     (cdr (assq :sep params)))))
-		  ;; Set file permissions if header argument
-		  ;; `:file-mode' is provided.
-		  (when (assq :file-mode params)
-		    (set-file-modes file (cdr (assq :file-mode params)))))
-		(setq result file))
-	      ;; Possibly perform post process provided its
-	      ;; appropriate.  Dynamically bind "*this*" to the
-	      ;; actual results of the block.
-	      (let ((post (cdr (assq :post params))))
-		(when post
-		  (let ((*this* (if (not file) result
-				  (org-babel-result-to-file
-				   file
-				   (org-babel--file-desc params result)
-                                   'attachment))))
-		    (setq result (org-babel-ref-resolve post))
-		    (when file
-		      (setq result-params (remove "file" result-params))))))
-	      (unless (member "none" result-params)
-	        (org-babel-insert-result
-	         result result-params info
-                 ;; append/prepend cannot handle hash as we accumulate
-                 ;; multiple outputs together.
-                 (when (member "replace" result-params) new-hash)
-                 lang
-                 (time-subtract (current-time) exec-start-time))))
-	    (run-hooks 'org-babel-after-execute-hook)
-	    result)))))))
+            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-ob-core-async-Handle-nasync-param-4-5.patch --]
[-- Type: text/x-patch, Size: 2620 bytes --]

From 019a0d2d8ba042606632e13976f8dfaeb37a8e74 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:32:45 +0100
Subject: [PATCH 4/8] ob-core async: Handle :nasync param [4/5]

---
 lisp/ob-core.el | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index d1adba61c..262218923 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1085,7 +1085,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		    (let ((d (file-name-as-directory (expand-file-name dir))))
 		      (make-directory d 'parents)
 		      d))))
-		 (cmd (intern (concat "org-babel-execute:" lang)))
+                 (async (org-babel--async-p params))
+                 (cmd (intern (concat "org-babel-"
+                                      (if async "schedule" "execute")
+                                      ":" lang)))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1139,8 +1142,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 	              (run-hooks 'org-babel-after-execute-hook)
                       result))))
 	    (unless (fboundp cmd)
-	      (error "No org-babel-execute function for %s!" lang))
-	    (message "Executing %s %s %s..."
+	      (error "No org-babel-execute function for %s: %s!" lang (symbol-name cmd)))
+	    (message "Executing %s %s %s %s..."
 		     (capitalize lang)
                      (pcase executor-type
                        ('src-block "code block")
@@ -1148,11 +1151,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                        ('babel-call "call")
                        ('inline-babel-call "inline call")
                        (e (symbol-name e)))
+                     (if async "async" "")
 		     (let ((name (nth 4 info)))
 		       (if name
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
-            (funcall handle-result (save-current-buffer (funcall cmd body params))))))))))
+            (if (not async)
+                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+              (let ((handle-feedback
+                     (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
+                (funcall cmd body params handle-feedback))))))))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-ob-core-async-Add-execute-with-5-5.patch --]
[-- Type: text/x-patch, Size: 2458 bytes --]

From c17aaea885e3aa6087563d55045e74ce71557dbf Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:09 +0100
Subject: [PATCH 5/8] ob-core async: Add :execute-with [5/5]

---
 lisp/ob-core.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 262218923..64f434f71 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1086,9 +1086,18 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		      (make-directory d 'parents)
 		      d))))
                  (async (org-babel--async-p params))
-                 (cmd (intern (concat "org-babel-"
-                                      (if async "schedule" "execute")
-                                      ":" lang)))
+                 (execute-with (let ((be (cdr (assq :execute-with params))))
+                                 (when (equal be "none") (setq be nil))
+                                 be))
+                 (cmd (intern (or (and execute-with
+                                       (concat execute-with "-" (if async "schedule" "execute")))
+                                  (concat "org-babel-"
+                                          (if async "schedule" "execute")
+                                          ":" lang))))
+                 (cmd-args (let ((ps (list body params)))
+                             (when execute-with
+                               (setq ps (cons lang ps)))
+                             ps))
                  (exec-start-time (current-time))
                  (handle-result
                   (lambda (result)
@@ -1157,10 +1166,10 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (funcall cmd body params)))
+                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
-                (funcall cmd body params handle-feedback))))))))))
+                (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch --]
[-- Type: text/x-patch, Size: 14628 bytes --]

From d5766eada2c31d0886f514f5ac6b38ad342e158f Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:23 +0100
Subject: [PATCH 6/8] lisp/org-elib-async.el: New package about async helpers

---
 lisp/org-elib-async.el | 327 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 327 insertions(+)
 create mode 100644 lisp/org-elib-async.el

diff --git a/lisp/org-elib-async.el b/lisp/org-elib-async.el
new file mode 100644
index 000000000..f0a1e4432
--- /dev/null
+++ b/lisp/org-elib-async.el
@@ -0,0 +1,327 @@
+;;; org-elib-async.el --- Helper to write asynchronous functions -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Version: 0.0.0
+;; Maintainer: Bruno BARBIER
+;; Keywords:
+;; Status: WORK IN PROGRESS.  DO NOT USE.
+;; URL:
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT (yet) part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+
+;;; Commentary:
+;; Names with "--" are for functions and variables that are meant to be for
+;; internal use only.
+
+;;;; Description
+;; Some functions to help dealing with asynchronous tasks.
+
+;; The prefix 'org-elib' means that this package should evenutally be
+;; moved into core Emacs.  The functions defined here do NOT depend
+;; nor rely on org itself.
+
+;;; TODOs
+;;
+;; - Keywords
+;;
+
+
+;;; Code:
+;;
+(require 'cl-lib)
+(require 'org-id)
+
+;;;; Process
+;;
+(cl-defun org-elib-async-process (command &key input callback)
+  "Execute COMMAND.
+
+A quick naive featureless boggus wrapper around `make-process' to
+receive the result when the process is done.
+
+When INPUT is non-nil, use it as the COMMAND standard input.  Let DATA
+be the COMMAND output, if COMMAND succeeds, call CALLBACK with
+(:success DATA), else, call CALLBACK with (:failure DATA)."
+  (let* ((stdout-buffer (generate-new-buffer "*org-elib-async-process*"))
+         (get-outcome
+          (lambda (process)
+            (with-current-buffer stdout-buffer
+              (let* ((exit-code (process-exit-status process))
+                     (real-end ;; Getting rid of the user message.
+                      (progn (goto-char (point-max))
+                             (forward-line -1)
+                             (point)))
+                     (txt (string-trim (buffer-substring-no-properties
+                                        (point-min) real-end))))
+                (list (if (eq 0 exit-code) :success :failure)
+                      (if (not (string-empty-p txt)) txt
+                        (and (not (eq 0 exit-code)) exit-code)))))))
+         (process (make-process
+                   :name "*org-elib-async-process*"
+                   :buffer  stdout-buffer
+                   :command command
+                   :connection-type 'pipe))
+         (sentinel
+          (lambda (&rest _whatever)
+            (pcase (process-status process)
+              ('run )
+              ('stop)
+              ((or 'exit 'signal)
+               (funcall callback (funcall get-outcome process)))
+              (_ (error "Not a real process"))))))
+    (add-function :after (process-sentinel process) sentinel)
+    (when input
+      (process-send-string process input)
+      (process-send-eof process))
+    process))
+;; (org-elib-async-process (list "date") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "false") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "true") :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash" "-c" "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "date" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "false" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "true" :callback (lambda (o) (message "outcome: %S" o)))
+;; (org-elib-async-process (list "bash") :input "sleep 2; date" :callback (lambda (o) (message "outcome: %S" o)))
+
+
+;;;; Wait for a process until some condition becomes true.
+
+(define-error 'org-elib-async-timeout-error
+              "Timeout waiting for a process.")
+
+(cl-defun org-elib-async-wait-condition ( cond-p
+                                          &key
+                                          (tick .3) (message "Waiting")
+                                          (nb_secs_between_messages 5)
+                                          timeout)
+  "Wait until the condition COND-P returns non-nil.
+Repeatedly call COND-P with no arguments, about every TICK seconds,
+until it returns a non-nil value.  Return that non-nil value.  When
+TIMEOUT (seconds) is non-nil, raise an `org-elib-async-timeout-error' if
+the COND-P is still nil after TIMEOUT seconds.  Assume COND-P calls cost
+0s.  Do NOT block display updates.  Do not block process outputs.  Do
+not block idle timers.  Do block the user, letting him/her know why, but
+do not display more messages than one every NB_SECS_BETWEEN_MESSAGES.
+Default MESSAGE is \"Waiting\".  Use 0.3s as the default for TICK."
+  ;; FIXME: Still not sure if it's possible to write such a function.
+  (let ((keep-waiting t)
+        (result nil)
+        (start (float-time))
+        elapsed
+        last-elapsed)
+    (while keep-waiting
+      (setq result (funcall cond-p))
+      (if result
+          (setq keep-waiting nil)
+        (sleep-for 0.01)
+        (redisplay :force)
+        (setq elapsed (- (float-time) start))
+        (when (and timeout (> elapsed timeout))
+          (signal 'org-timeout-error (list message elapsed)))
+        ;; Let the user know, without flooding the message area.
+        (if (and last-elapsed (> (- elapsed last-elapsed) nb_secs_between_messages))
+            (message (format "%s ...(%.1fs)" message elapsed)))
+        (unless (sit-for tick :redisplay)
+          ;; Emacs has something to do; let it process new
+          ;; sub-processes outputs in case there are some.
+          (accept-process-output nil 0.01))
+        (setq last-elapsed elapsed)))
+    result))
+
+
+
+;;;; Comint: a FIFO queue of tasks with callbacks
+;; org-elib-async-comint-queue executes tasks in a FIFO order. For each
+;; task, it identifies the text output for that
+;; task. org-elib-async-comint-queue does NOT remove prompts, or other
+;; useless texts; this is the responsibility of the user.  Currently,
+;; org-elib-async-comint-queue assume it has the full control of the
+;; session: no user interaction, no other direct modifications.
+
+(defvar-local org-elib-async-comint-queue--todo :NOT-SET
+  "A FIFO queue of pending executions.")
+
+
+(defvar-local org-elib-async-comint-queue--unused-output ""
+  "Process output that has not been used yet.")
+
+(defvar-local org-elib-async-comint-queue--incoming-text ""
+  "Newly incoming text, added by the process filter, not yet handled.")
+
+(defvar-local org-elib-async-comint-queue--current-task nil
+  "The task that is currently running.")
+
+(defvar-local org-elib-async-comint-queue--process-filter-running nil
+  "non-nil when filter is running.")
+
+(defvar-local org-elib-async-comint-queue--incoming-timer nil
+  "A timer, when handling incoming text is scheduled or running.")
+
+
+(defvar-local org-elib-async-comint-queue--handle-incoming-running
+    nil
+  "True when the incoming text handler is running.")
+
+(defun org-elib-async-comint-queue--handle-incoming ()
+  (when org-elib-async-comint-queue--handle-incoming-running
+    (error "Bad call to handle-incoming: kill buffer %s!" (current-buffer)))
+  (setq org-elib-async-comint-queue--handle-incoming-running t)
+
+  ;; Take the incoming text.
+  (setq org-elib-async-comint-queue--unused-output
+        (concat org-elib-async-comint-queue--unused-output
+                org-elib-async-comint-queue--incoming-text))
+  (setq org-elib-async-comint-queue--incoming-text "")
+
+  ;; Process the unused text with the queued tasks
+  (unless org-elib-async-comint-queue--current-task
+    (when org-elib-async-comint-queue--todo
+      (setq org-elib-async-comint-queue--current-task (pop org-elib-async-comint-queue--todo))))
+  (when-let ((task org-elib-async-comint-queue--current-task))
+    (let ((unused org-elib-async-comint-queue--unused-output)
+          (session-buffer (current-buffer))
+          task-start)
+      (setq org-elib-async-comint-queue--unused-output
+            (with-temp-buffer
+              (insert unused)
+              (goto-char (point-min))
+              (while (and task
+                          (setq task-start (point))
+                          (search-forward (car task) nil t))
+                (when (cdr task)
+                  (let ((txt (buffer-substring-no-properties task-start
+                                                             (- (point) (length (car task))))))
+                    (save-excursion (funcall (cdr task) txt))))
+                (setq task (and (buffer-live-p session-buffer)
+                                (with-current-buffer session-buffer (pop org-elib-async-comint-queue--todo)))))
+              (buffer-substring (point) (point-max))))
+      (setq org-elib-async-comint-queue--current-task task)))
+
+  ;; Signal that we are done. If we already have some new incoming text,
+  ;; reschedule to run.
+  (setq org-elib-async-comint-queue--incoming-timer
+        (if (string-empty-p org-elib-async-comint-queue--incoming-text)
+            nil
+          (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+  ;; We reset it only on success. If it failed for some reason, the
+  ;; comint buffer is in an unknown state: you'll need to kill that
+  ;; buffer.
+  (setq org-elib-async-comint-queue--handle-incoming-running nil))
+
+
+(defun org-elib-async-comint-queue--wake-up-handle-incoming ()
+  "Wake up the handling of incoming chunks of text.
+Assume we are called from the comint buffer."
+  (setq org-elib-async-comint-queue--incoming-timer
+        (run-with-timer
+         0.01 nil
+         (let ((comint-buffer (current-buffer)))
+           (lambda ()
+             (with-local-quit
+               (with-current-buffer comint-buffer
+                 (org-elib-async-comint-queue--handle-incoming))))))))
+
+
+(defun org-elib-async-comint-queue--process-filter (chunk)
+  "Accept the arbitrary CHUNK of text."
+  (setq org-elib-async-comint-queue--incoming-text
+        (concat org-elib-async-comint-queue--incoming-text
+                chunk))
+  :; We delegate the real work outside the process filter, as it is
+   ; not reliable to do anything here.
+  (unless org-elib-async-comint-queue--incoming-timer
+    (org-elib-async-comint-queue--wake-up-handle-incoming)))
+
+
+
+(define-error 'org-elib-async-comint-queue-task-error
+              "Task failure.")
+
+(cl-defun org-elib-async-comint-queue--push (exec &key handle-feedback)
+  "Push the execution of EXEC into the FIFO queue.
+When the task completed, call HANDLE-FEEDBACK with its outcome.  Return
+a function that waits for and return the result on succes, raise on
+failure."
+  (let* ((tid (org-id-uuid))
+         (start-tag (format "ORG-ELIB-ASYNC_START_%s" tid))
+         (end-tag (format "ORG-ELIB-ASYNC_END___%s" tid))
+         (result-sb (make-symbol "result"))
+         (on-start
+          (lambda (_)
+            ;; TODO: Use (point) in session to link back to it.
+            (when handle-feedback
+              (funcall handle-feedback '(:pending "running")))))
+         (on-result
+          (lambda (result)
+            ;; Get the result, and report success using HANDLE-FEEDBACK.
+            ;; If something fails, report failure using HANDLE-FEEDBACK.
+            (unwind-protect
+                (let ((outcome
+                       (condition-case-unless-debug exc
+                           (list :success (funcall exec :post-process result))
+                         (error (list :failure exc)))))
+                  (when handle-feedback (save-excursion (funcall handle-feedback outcome)))
+                  (set result-sb outcome))
+              (funcall exec :finally)))))
+
+    ;; TODO: Add detect-properties => alist of properties that can be used: PS1 and PS2
+    (let ((comint-buffer (funcall exec :get-comint-buffer)))
+      (with-current-buffer comint-buffer
+        (setq org-elib-async-comint-queue--todo
+              (nconc org-elib-async-comint-queue--todo
+                     (list (cons start-tag on-start)
+                           (cons end-tag on-result))))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-enter))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag start-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :get-code))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instr-to-emit-tag end-tag))
+        (funcall exec :send-instrs-to-session
+                 (funcall exec :instrs-to-exit))
+
+        (lambda ()
+          (org-elib-async-wait-condition (lambda ()
+                                           (boundp result-sb)))
+          (pcase (symbol-value result-sb)
+            (`(:success ,r) r)
+            (`(:failure ,err) (signal (car err) (cdr err)))))
+        ))))
+
+
+
+(defun org-elib-async-comint-queue-init-if-needed (buffer)
+  "Initialize the FIFO queue in BUFFER if needed."
+  (with-current-buffer buffer
+    (unless (local-variable-p 'org-elib-async-comint-queue--todo)
+      (setq-local org-elib-async-comint-queue--todo nil)
+      (add-hook 'comint-output-filter-functions
+                #'org-elib-async-comint-queue--process-filter nil :local))))
+
+
+
+;;;; Provide
+(provide 'org-elib-async)
+;;; org-elib-async.el ends here
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 3220 bytes --]

From 1129770cc62b0580c71ba2a3f6a94f6fd18574b3 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 16 Feb 2024 14:33:40 +0100
Subject: [PATCH 7/8] lisp/ob-core.el: Notify when execution fails

lisp/ob-core.el (org-babel-popup-failure-details): New function.
(org-babel-execute-src-block): For synchronous execution, use
`org-babel-popup-failure-details' on failure.
(org-babel--async-feedbacks): Add call to
`org-babel-popup-failure-details' on user request.
---
 lisp/ob-core.el | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 64f434f71..e8f9e9ad9 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -917,7 +917,17 @@ (defun org-babel--async-feedbacks (info handle-result
                                                   (if (consp msg) (car msg)
                                                     (format "%s" msg))
                                                   (short-version-of msg)))
-                                      'face (org-babel--async-status-face status))))
+                                      'face (org-babel--async-status-face status)))
+             (when (eq :failure status)
+               (overlay-put ovl-title
+                            'keymap
+                            (let ((km (make-sparse-keymap)))
+                              (define-key km (kbd "<mouse-1>")
+                                          (lambda ()
+                                            "Display failure details."
+                                            (interactive)
+                                            (org-babel-popup-failure-details msg)))
+                              km))))
          (remove-previous-overlays ()
            ;; Remove previous title and body overlays.
            (mapc (lambda (ovl)
@@ -1166,11 +1176,26 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
                            (format "(%s)" name)
                          (format "at position %S" (nth 5 info)))))
             (if (not async)
-                (funcall handle-result (save-current-buffer (apply cmd cmd-args)))
+                (let ((res-sb (make-symbol "result")))
+                  (condition-case exc
+                      (set res-sb (save-current-buffer (apply cmd cmd-args)))
+                    (error (org-babel-popup-failure-details exc)))
+                  (when (boundp res-sb)
+                    (funcall handle-result (symbol-value res-sb))))
               (let ((handle-feedback
                      (org-babel--async-feedbacks info handle-result result-params exec-start-time)))
                 (apply cmd (nconc cmd-args (list handle-feedback))))))))))))
 
+(defun org-babel-popup-failure-details (exc)
+  "Notify/display"
+  (when-let ((buf (get-buffer org-babel-error-buffer-name)))
+    (with-current-buffer buf (erase-buffer)))
+  (org-babel-eval-error-notify
+   127 ; Don't have exit-code
+   (if (consp exc)
+       (format "%s\n%s\n" (car exc) (cdr exc))
+     (format "%s\n" exc))))
+
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch --]
[-- Type: text/x-patch, Size: 33567 bytes --]

From 2078be0ebd741127c383a386f672dbc3d741206d Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Wed, 21 Feb 2024 15:54:05 +0100
Subject: [PATCH 8/8] scratch/bba-ob-core-async: Some temporary test files

---
 scratch/bba-ob-core-async/my-async-tests.el  | 339 ++++++++
 scratch/bba-ob-core-async/my-async-tests.org | 820 +++++++++++++++++++
 2 files changed, 1159 insertions(+)
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.el
 create mode 100644 scratch/bba-ob-core-async/my-async-tests.org

diff --git a/scratch/bba-ob-core-async/my-async-tests.el b/scratch/bba-ob-core-async/my-async-tests.el
new file mode 100644
index 000000000..918a4b142
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.el
@@ -0,0 +1,339 @@
+;;; my-async-tests.el --- Scratch/temporary file: some tests about async -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Bruno BARBIER
+
+;; Author: Bruno BARBIER
+;; Status: Temporary tests.
+;; Compatibility: GNU Emacs 30.0.50
+;;
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 2 of
+;; the License, or (at your option) any later version.
+
+;; This program is distributed in the hope that it will be
+;; useful, but WITHOUT ANY WARRANTY; without even the implied
+;; warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+;; PURPOSE.  See the GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public
+;; License along with this program; if not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+;; MA 02111-1307 USA
+
+(require 'cl-lib)
+(require 'org)
+(require 'org-elib-async)
+
+(require 'ob-shell)
+(require 'ob-python)
+
+;;; Shells
+;;
+
+;;;; One shell script
+
+;; Standalone direct asynchronous execution.
+
+(defun my-shell-babel-schedule (lang body _params handle-feedback)
+  "Execute the bash script BODY.
+Execute the shell script BODY using bash.  Use HANDLE-FEEDBACK to report
+the outcome (success or failure)."
+  (unless (equal "bash" lang)
+    (error "Only for bash"))
+  (funcall handle-feedback (list :pending "started"))
+  (org-elib-async-process (list "bash") :input body :callback handle-feedback))
+
+
+;;;; Asynchronous using ob-shell
+
+(defun my-org-babel-shell-how-to-execute (body params)
+  "Return how to execute BODY using a POSIX shell.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-shell, following
+  ;; `org-babel-execute:shell' and `org-babel-sh-evaluate'.
+  ;; Results are expected to differ from ob-shell as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((session (org-babel-sh-initiate-session
+		   (cdr (assq :session params))))
+	 (stdin (let ((stdin (cdr (assq :stdin params))))
+                  (when stdin (org-babel-sh-var-to-string
+                               (org-babel-ref-resolve stdin)))))
+	 (result-params (cdr (assq :result-params params)))
+	 (value-is-exit-status
+	  (or (and
+	       (equal '("replace") result-params)
+	       (not org-babel-shell-results-defaults-to-output))
+	      (member "value" result-params)))
+	 (cmdline (cdr (assq :cmdline params)))
+         (shebang (cdr (assq :shebang params)))
+         (full-body (concat
+		     (org-babel-expand-body:generic
+		      body params (org-babel-variable-assignments:shell params))
+		     (when value-is-exit-status "\necho $?")))
+         (post-process
+          (lambda (r)
+            (setq r (org-trim r))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (let ((tmp-file (org-babel-temp-file "sh-")))
+                 (with-temp-file tmp-file (insert r))
+                 (org-babel-import-elisp-from-file tmp-file)))
+             (org-babel-pick-name
+              (cdr (assq :colname-names params)) (cdr (assq :colnames params)))
+             (org-babel-pick-name
+              (cdr (assq :rowname-names params)) (cdr (assq :rownames params))))))
+         comint-buffer
+         finally
+         to-run)
+
+    (setq comint-buffer
+          (if session session
+            ;; No session. We create a temporary one and use 'finally' to
+            ;; destroy it once we are done.
+            ;;
+            ;; FIXME: This session code should be refactored and moved into
+            ;;        ob-core.
+            (let ((s-buf (org-babel-sh-initiate-session
+                          (generate-new-buffer-name (format "*ob-shell-no-session*")))))
+              (setq finally (lambda ()
+                                ;; We cannot delete it immediately as we are called from it.
+                                (run-with-idle-timer
+                                 0.1 nil
+                                 (lambda ()
+                                   (when (buffer-live-p s-buf)
+                                     (let ((kill-buffer-query-functions nil)
+                                           (kill-buffer-hook nil))
+                                       (kill-buffer s-buf)))))))
+              s-buf)))
+
+    (org-elib-async-comint-queue-init-if-needed comint-buffer)
+
+    (setq to-run
+          (cond
+           ((or stdin cmdline)	       ; external shell script w/STDIN
+            (let ((script-file (org-babel-temp-file "sh-script-"))
+	          (stdin-file (org-babel-temp-file "sh-stdin-"))
+	          (padline (not (string= "no" (cdr (assq :padline params))))))
+	      (with-temp-file script-file
+	        (when shebang (insert shebang "\n"))
+	        (when padline (insert "\n"))
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+	      (with-temp-file stdin-file (insert (or stdin "")))
+	      (with-temp-buffer
+                (with-connection-local-variables
+                 (concat
+                  (mapconcat #'shell-quote-argument
+                             (cons (if shebang (file-local-name script-file)
+                                     shell-file-name)
+                                   (if shebang (when cmdline (list cmdline))
+                                     (list shell-command-switch
+                                           (concat (file-local-name script-file)  " " cmdline))))
+                             " ")
+                  "<" (shell-quote-argument stdin-file))))))
+           (session ; session evaluation
+            full-body)
+           ;; External shell script, with or without a predefined
+           ;; shebang.
+           ((org-string-nw-p shebang)
+            (let ((script-file (org-babel-temp-file "sh-script-"))
+	          (padline (not (equal "no" (cdr (assq :padline params))))))
+	      (with-temp-file script-file
+	        (insert shebang "\n")
+	        (when padline (insert "\n"))
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+              (if (file-remote-p script-file)
+                  ;; Run remote script using its local path as COMMAND.
+                  ;; The remote execution is ensured by setting
+                  ;; correct `default-directory'.
+                  (let ((default-directory (file-name-directory script-file)))
+                    (file-local-name script-file)
+	            script-file ""))))
+           (t
+            (let ((script-file (org-babel-temp-file "sh-script-")))
+	      (with-temp-file script-file
+	        (insert full-body))
+	      (set-file-modes script-file #o755)
+              (mapconcat #'shell-quote-argument
+                         (list shell-file-name
+                               shell-command-switch
+                               (if (file-remote-p script-file)
+                                   (file-local-name script-file)
+                                 script-file))
+                         " ")))))
+    ;; TODO: How to handle `value-is-exit-status'?
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "export PS1=''; export PS2='';")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "printf '%s\\n'" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         (with-current-buffer comint-buffer
+           (when code
+             (goto-char (point-max))
+             (insert code) (insert "\n")
+             (comint-send-input nil t))))
+        (`(:get-code) to-run)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+
+
+
+;;; Python
+;;
+
+;;;; Asynchronous using ob-python
+
+(defun my-org-babel-python-how-to-execute (body params)
+  "Return how to execute BODY using python.
+Return how to execute, as expected by
+`org-elib-async-comint-queue--execution'."
+  ;; Code mostly extracted from ob-python, following
+  ;; `org-babel-python-evaluate-session'.
+  ;; Results are expected to differ from ob-python as we follow the
+  ;; same process for all execution paths: asynchronous or not, with
+  ;; session or without.
+  (let* ((org-babel-python-command
+          (or (cdr (assq :python params))
+              org-babel-python-command))
+         (session-key (org-babel-python-initiate-session
+                       (cdr (assq :session params))))
+         (graphics-file (and (member "graphics" (assq :result-params params))
+                             (org-babel-graphical-output-file params)))
+         (result-params (cdr (assq :result-params params)))
+         (result-type (cdr (assq :result-type params)))
+         (results-file (when (eq 'value result-type)
+                         (or graphics-file
+                             (org-babel-temp-file "python-"))))
+         (return-val (when (eq result-type 'value)
+                       (cdr (assq :return params))))
+         (full-body
+          (concat
+           (org-babel-expand-body:generic
+            body params
+            (org-babel-variable-assignments:python params))
+           (when return-val
+             (format "\n%s" return-val))))
+         (post-process
+          (lambda (r)
+            (setq r (string-trim r))
+            (when (string-prefix-p "Traceback (most recent call last):" r)
+              (signal 'user-error (list r)))
+            (when (eq 'value result-type)
+              (setq r (org-babel-eval-read-file results-file)))
+            (org-babel-reassemble-table
+             (org-babel-result-cond result-params
+               r
+               (org-babel-python-table-or-string r))
+             (org-babel-pick-name (cdr (assq :colname-names params))
+                                  (cdr (assq :colnames params)))
+             (org-babel-pick-name (cdr (assq :rowname-names params))
+                                  (cdr (assq :rownames params))))))
+         (tmp-src-file (org-babel-temp-file "python-"))
+         (session-body
+          ;; The real code we evaluate in the session.
+          (pcase result-type
+            (`output
+             (format (string-join
+                      (list "with open('%s') as f:\n"
+                            "    exec(compile(f.read(), f.name, 'exec'))\n"))
+                     (org-babel-process-file-name
+                      tmp-src-file 'noquote)))
+            (`value
+             ;; FIXME: In this case, any output is an error.
+             (org-babel-python-format-session-value
+              tmp-src-file results-file result-params))))
+         comint-buffer
+         finally)
+
+
+    (unless session-key
+      ;; No session. We create a temporary one and use 'finally' to
+      ;; destroy it once we are done.
+      ;;
+      ;; FIXME: This session code should be refactored and moved into
+      ;;        ob-core.
+      (setq session-key (org-babel-python-initiate-session
+                         ;; We can't use a simple `generate-new-buffer'
+                         ;; due to the earmuffs game.
+                         (org-babel-python-without-earmuffs
+                          (format "*ob-python-no-session-%s*" (org-id-uuid)))))
+      (setq finally (lambda ()
+                      (when-let ((s-buf
+                                  (get-buffer (org-babel-python-with-earmuffs session-key))))
+                        ;; We cannot delete it immediately as we are called from it.
+                        (run-with-idle-timer
+                         0.1 nil
+                         (lambda ()
+                           (when (buffer-live-p s-buf)
+                             (let ((kill-buffer-query-functions nil)
+                                   (kill-buffer-hook nil))
+                               (kill-buffer s-buf)))))))))
+
+    (setq comint-buffer
+          (get-buffer (org-babel-python-with-earmuffs session-key)))
+    (org-elib-async-comint-queue-init-if-needed comint-buffer)
+    (with-temp-file tmp-src-file
+      (insert (if (and graphics-file (eq result-type 'output))
+                  (format org-babel-python--output-graphics-wrapper
+                          full-body graphics-file)
+                full-body)))
+
+    (lambda (&rest q)
+      (pcase q
+        (`(:instrs-to-enter)
+         ;; FIXME: This is wrong.
+         "import sys; sys.ps1=''; sys.ps2=''")
+        (`(:instrs-to-exit))
+        (`(:finally) (when finally (funcall finally)))
+        (`(:instr-to-emit-tag ,tag) (format "print ('%s')" tag))
+        (`(:post-process ,r) (when post-process (funcall post-process r)))
+        (`(:send-instrs-to-session ,code)
+         ;; See org-babel-python-send-string
+         (with-current-buffer comint-buffer
+           (let ((python-shell-buffer-name
+                  (org-babel-python-without-earmuffs session-key)))
+             (python-shell-send-string (concat code "\n")))))
+        (`(:get-code) session-body)
+        (`(:get-comint-buffer) comint-buffer)
+        (_ (error "Unknown query"))))))
+
+
+;;; Org babel 'execute-with'.
+
+
+;;;; Asynchronous
+;;
+(defun my-org-babel-schedule (lang body params handle-feedback)
+  "Schedule the execution of BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'.  Return a
+function that waits and returns the result on success, raise on failure."
+  (let ((exec (pcase lang
+                ("python" (my-org-babel-python-how-to-execute body params))
+                ("bash"   (my-org-babel-shell-how-to-execute body params))
+                (_ (error "Not handled (yet): %s" lang)))))
+    (org-elib-async-comint-queue--push exec :handle-feedback handle-feedback)))
+
+
+;;;; Synchronous
+;;
+(defun my-org-babel-execute (lang body params)
+  "Execute Python BODY according to PARAMS.
+This function is called by `org-babel-execute-src-block'."
+  ;; We just start the asynchronous execution, wait for it, and return
+  ;; the result (or raise the exception). No custom code, and,
+  ;; synchronous and asynchronous should just mix nicely together.
+  (funcall (my-org-babel-schedule lang body params nil)))
diff --git a/scratch/bba-ob-core-async/my-async-tests.org b/scratch/bba-ob-core-async/my-async-tests.org
new file mode 100644
index 000000000..08284c470
--- /dev/null
+++ b/scratch/bba-ob-core-async/my-async-tests.org
@@ -0,0 +1,820 @@
+#+PROPERTY: HEADER-ARGS+ :eval no-export :exports both
+* Intro
+
+An org document with code blocks to help test the proposed patchs.
+
+See [[*On top of ob-shell][On top of ob-shell]] for asynchronous shell scripts using ob-shell.
+
+See [[*On top of ob-python][On top of ob-python]] for asynchronous python scripts using ob-python.
+
+You need to load:
+   #+begin_src elisp :results silent
+   (setq-local org-confirm-babel-evaluate nil)
+   (load-file "my-async-tests.el")
+   #+end_src
+
+
+Emacs and org versions:
+   #+begin_src elisp
+   (mapcar (lambda (sb) (list sb (symbol-value sb)))
+           '(emacs-version org-version))
+   #+end_src
+
+   #+RESULTS:
+   | emacs-version | 30.0.50 |
+   | org-version   | 9.7-pre |
+
+Note that we've disabled eval on export: export doesn't know it needs
+to wait for asynchronous results.
+
+* POSIX shells
+** A simple bash example
+   :PROPERTIES:
+    :header-args:bash: :execute-with my-shell-babel :nasync yes
+   :END:
+
+The package `my-async-tests.el' contains the function
+`my-shell-babel-schedule' to evaluate shell script asynchronously.
+
+The header-args properties above request asynchronous execution for
+bash (:nasync yes), and, tells ob-core to use the prefix
+`my-shell-babel' when looking for functions to evaluate a source
+block. Thus, org will delegate execution to `my-shell-babel-schedule'.
+We don't have `my-shell-babel-execute', so, in this case, :nasync must
+be yes.
+
+Examples taken from the org mailing list and from worg.
+
+A simple execution:
+    #+begin_src bash
+      date
+    #+end_src
+
+    #+RESULTS:
+    : Wed Feb 21 15:56:23 CET 2024
+
+A tricky computation takes some time:
+    #+begin_src bash
+      sleep 1; date
+    #+end_src
+
+    #+RESULTS:
+    : Wed Feb 21 15:56:25 CET 2024
+
+An example of a failure:
+    #+begin_src bash
+      sleepdd 1; false
+    #+end_src
+
+    #+RESULTS:
+
+** On top of ob-shell
+   :PROPERTIES:
+    :header-args:bash:  :execute-with my-org-babel :nasync yes
+    :header-args:bash+: :session sh-async
+   :END:
+
+   #+begin_src bash
+   sleep 1; date
+   #+end_src
+
+   #+RESULTS:
+   : Wed Feb 21 15:56:42 CET 2024
+
+
+   #+begin_src bash :results output :session *test* :nasync yes
+   cd /tmp
+   echo "hello world"
+   #+end_src
+
+   #+RESULTS:
+   : hello world
+
+   #+begin_src bash :results output
+   # comment
+   # comment
+   #+end_src
+
+   #+RESULTS:
+
+    #+begin_src bash :results output
+    # print message
+    echo \"hello world\"
+    #+end_src
+
+    #+RESULTS:
+    : "hello world"
+
+    #+begin_src bash :results output
+    echo "hello"
+    echo "world"
+    #+end_src
+
+    #+RESULTS:
+    : hello
+    : world
+
+
+    #+begin_src bash :results output
+      echo PID: "$$"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22212
+
+    #+begin_src bash :results output
+      echo PID: "$$"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22212
+
+
+    #+begin_src bash :results output :session shared
+      echo PID: "$$"
+      X=5
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22218
+
+    #+begin_src bash :results output :session shared
+      echo PID: "$$"
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : PID: 22218
+    : X was set to 5
+
+    #+begin_src bash :nasync yes :results value scalar
+    echo "Execute session blocks in the background"
+    sleep 3
+    echo "Using the :async header"
+    #+end_src
+
+    #+RESULTS:
+    : Execute session blocks in the background
+    : Using the :async header
+    : 0
+
+
+
+    #+name: their-os
+    Linux
+
+
+    #+begin_src bash :results output :shebang #!/usr/bin/env bash :stdin their-os :cmdline RMS :tangle ask_for_os.sh
+
+      # call as ./ask_for_os.sh NAME, where NAME is who to ask
+
+      if [ -z "$1" ]; then
+          asked="$USER"
+      else
+          asked="$1"
+      fi
+
+      echo Hi, "$asked"! What operating system are you using?
+      read my_os
+
+      if [ "$asked" = "RMS" ]; then
+          echo You\'re using GNU/"$my_os"!
+      elif [ "$asked" = "Linus" ]; then
+          echo You\'re using "$my_os"!
+      else
+          echo You\'re using `uname -o`!
+      fi
+    #+end_src
+
+    #+RESULTS:
+    : Hi, RMS! What operating system are you using?
+    : You're using GNU/Linux!
+
+
+
+    #+begin_src bash
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+    | 7592 | 13920 | 4911 |
+
+
+    #+begin_src bash :results list
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+    - 17593
+      384
+      1439
+
+    #+begin_src bash :results file :file my_output.txt
+    declare -a array
+
+    m=4
+    n=3
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            a[${i},${j}]=$RANDOM
+        done
+    done
+    for ((i=0; i<m; i++))
+    do
+        for ((j=0; j<n; j++))
+        do
+            echo -ne "${a[${i},${j}]}\t"
+        done
+        echo
+    done
+    #+end_src
+
+    #+RESULTS:
+    [[file:my_output.txt]]
+
+
+    #+begin_src bash :results output
+      cat my_output.txt
+    #+end_src
+
+    #+RESULTS:
+    : 30132	16194	19934	
+    : 30132	16194	19934	
+    : 30132	16194	19934	
+    : 30132	16194	19934
+
+
+    #+begin_src bash :results output :dir /ssh:phone: :session none
+    if [ ! -e  "foo_file" ];
+    then
+        echo "foo" > foo_file
+        echo "Created foo_file"
+    else
+        echo "foo_file already exists!"
+    fi
+    #+end_src
+
+    #+RESULTS:
+    : foo_file already exists!
+
+
+    #+begin_src bash :results output :dir /ssh:phone: :session *remote*
+    if [ ! -e  "foo_file" ];
+    then
+        echo "foo" > foo_file
+        echo "Created foo_file"
+    else
+        echo "foo_file already exists!"
+    fi
+    #+end_src
+
+    #+RESULTS:
+    : Created foo_file
+
+
+    #+RESULTS:
+
+
+    #+begin_src bash :results none :session *my-session*
+      X=1
+    #+end_src
+
+    #+RESULTS:
+
+    #+begin_src bash :results output :session *my-session*
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : X was set to 1
+
+    #+begin_src bash :results output :session *another-session*
+      echo X was set to "$X"
+    #+end_src
+
+    #+RESULTS:
+    : X was set to
+
+    #+RESULTS:
+
+
+    #+begin_src bash :results output
+    echo "Hello, world!"
+    sleep 3
+    echo "Good-bye, cruel World..."
+    #+end_src
+
+    #+RESULTS:
+    : Hello, world!
+    : Good-bye, cruel World...
+
+
+
+    #+begin_src bash :var by_two=0  x=3 :session none
+      if [ "$by_two" = "0" ]; then
+        echo $(($x * 2))
+      else
+        echo $(($x * 3))
+      fi
+    #+end_src
+
+    #+RESULTS:
+    : 6
+
+
+    #+begin_src bash :results output :var arr='("apple" "banana" "cherry")
+      echo The first element is...
+      echo \"${arr[1]}\"
+    #+end_src
+
+    #+RESULTS:
+    : The first element is...
+    : "banana"
+
+*** TODO Doesn't work yet: asynchronous that depends from other blocks
+    #+name: multiply_by_2
+    #+begin_src bash :var data="" :results output
+      echo $(($data * 2))
+    #+end_src
+
+    #+RESULTS: multiply_by_2
+    : bash: * 2: syntax error: operand expected (error token is "* 2")
+
+    #+begin_src bash :post multiply_by_2(data=*this*)
+      echo 3
+    #+end_src
+
+    #+results:
+
+
+* On top of ob-python
+   :PROPERTIES:
+    :header-args:python:  :execute-with my-org-babel :nasync yes
+    :header-args:python+: :session py-async
+   :END:
+
+Used =header-args= properties:
+   - =:execute-with my-org-babel=:  look for functions with the prefix `my-org-babel' to execute
+     blocks (for the asynchronous case use
+     `my-org-babel-schedule', and, for the synchronous case
+     `my-org-babel-execute'). These functions are defined in [[file:my-async-tests.el]].
+
+   - =:nasync yes=: by default, execute asynchronously (use `my-org-babel-schedule').
+
+   - =:session py-async= by default, use a session named "py-async".
+
+** basic examples
+*** async with a session
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708515666.8s | 1708515667.8s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** async with no session
+   :PROPERTIES:
+    :header-args:python+: :session none
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+    #+RESULTS:
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, it failed, as expected. "import time" was done in its own
+temporary session.  The old result is preserved; the error is display
+as an overlay. Click on it to get more info about the error.
+
+
+Let's fix it, adding the import line:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708515915.0s | 1708515916.0s | 1.0s |
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+
+*** sync with a session
+   :PROPERTIES:
+    :header-args:python+: :session py-sync-session :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708102997.5s | 1708102998.5s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+*** sync with no session
+   :PROPERTIES:
+    :header-args:python+: :session none :nasync no
+   :END:
+
+A very simple test:
+    #+begin_src python
+      2+3
+    #+end_src
+
+    #+RESULTS:
+    : 5
+
+Let's import the module time in our session.
+    #+begin_src python :results silent
+      import time
+    #+end_src
+
+(Yes, =:results silent= needs some work.)
+
+
+
+A table that requires some time to compute:
+    #+begin_src python
+      start = time.time()
+      time.sleep(1)
+      end = time.time()
+      ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708083470.9s | 1708083471.9s | 1.0s |
+
+Yes, that fails (no session), displaying the details in a popup. Let's
+fix it:
+    #+begin_src python
+    import time
+    start = time.time()
+    time.sleep(1)
+    end = time.time()
+    ["%.1fs" % t for t in [start, end, end-start]]
+    #+end_src
+
+    #+RESULTS:
+    | 1708103039.0s | 1708103040.0s | 1.0s |
+
+
+
+An error (click on the error , <mouse-1>, to see the details):
+    #+begin_src python
+      2/0
+    #+end_src
+
+    #+RESULTS:
+
+
+** worg examples
+
+Let's import matplotlib in our session.
+
+    #+begin_src python
+    import matplotlib
+    import matplotlib.pyplot as plt
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+A figure in a PDF, asynchronous case.
+    #+begin_src python :results file link
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-async.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-async.pdf]]
+
+
+A figure in a PDF, synchronous case.
+    #+begin_src python :results file link :nasync no
+    fig=plt.figure(figsize=(3,2))
+    plt.plot([1,3,2])
+    fig.tight_layout()
+
+    fname = 'myfig-sync.pdf'
+    plt.savefig(fname)
+    fname # return this to org-mode
+    #+end_src
+
+    #+RESULTS:
+    [[file:myfig-sync.pdf]]
+
+
+
+A PNG figure, asynchronous case.
+    #+begin_src python :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+      fig
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, but using the =:return= keyword.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot.png
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot.png]]
+
+Same, asynchronous but without a session this time.
+    #+begin_src python :return "plt.gcf()" :results graphics file output :file boxplot-no-sess-a-y.png :session none
+      import matplotlib
+      import matplotlib.pyplot as plt
+      fig=plt.figure(figsize=(3,2))
+      plt.plot([1,3,2])
+      fig.tight_layout()
+    #+end_src
+
+    #+RESULTS:
+    [[file:boxplot-no-sess-a-y.png]]
+
+
+Lists are table,
+    #+begin_src python
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    | 1 | 2 | 3 |
+
+unless requested otherwise.
+    #+begin_src python :results verbatim
+      [1,2,3]
+    #+end_src
+
+    #+RESULTS:
+    : [1, 2, 3]
+
+
+Dictionaries are tables too.
+    #+begin_src python :results table
+      {"a": 1, "b": 2}
+    #+end_src
+
+    #+RESULTS:
+    | a | 1 |
+    | b | 2 |
+
+
+Let's try the example with Panda.
+    #+begin_src python :results none
+      import pandas as pd
+      import numpy as np
+    #+end_src
+
+    #+RESULTS:
+    : None
+
+    #+begin_src python :results table
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+And the synchronous case?
+
+    #+begin_src python :results table  :nasync no
+      pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                   columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+
+Without session ?
+
+    #+begin_src python :results table :session none
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+Right, we need to import the libraries (no session).
+
+    #+begin_src python :results table :session none
+    import pandas as pd
+    import numpy as np
+    pd.DataFrame(np.array([[1,2,3],[4,5,6]]),
+                 columns=['a','b','c'])
+    #+end_src
+
+    #+RESULTS:
+    |   | a | b | c |
+    |---+---+---+---|
+    | 0 | 1 | 2 | 3 |
+    | 1 | 4 | 5 | 6 |
+
+
+** inline examples
+
+    A simple asynchronous inline src_python{3*2} {{{results(=6=)}}}.
+
+    An other one containing a mistake src_python{2/0} {{{results(=6=)}}}
+    (click on the error to see the details).
+
+
+    Some very slow inline asynchronous computations that all run in
+    the same session. You need to execute the 3 of them at once. Here
+    is the first one src_python[:return "\"OK1\""]{import time;
+    time.sleep(5)} {{{results(=OK1=)}}} and a second one
+    src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)}
+    {{{results(=OK1 bis=)}}} and the third one src_python[:return
+    "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Yes, the previous paragraph is unreadable; it's on purpose, to
+    check that ob-core can figure it out.
+
+    Let's repeat, in a more readable way, and making the last one
+    synchronous.
+
+    Some very slow inline computations that all run in the same
+    session. Here is the first asynchronous one
+          src_python[:return"\"OK1\""]{import time; time.sleep(5)} {{{results(=None=)}}}
+    and a second one, asynchronous too:
+          src_python[:return "\"OK1 bis\""]{import time; time.sleep(5)} {{{results(=OK1 bis=)}}}
+    and finally, a third one, synchronous this one:
+          src_python[:nasync no :return "\"OK2\""]{import time; time.sleep(5)} {{{results(=OK2=)}}}.
+
+    Note that, once the user executes the last synchronous block, the
+    user is blocked until the synchronous execution can start
+    (i.e. all previous asynchronous executions are done) and until
+    it's done.  The display is updated though, to see the asynchronous
+    progress.
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-19  9:06                 ` Ihor Radchenko
  2024-02-19 19:47                   ` Matt
@ 2024-02-21 16:03                   ` Bruno Barbier
  2024-02-23 12:11                     ` Ihor Radchenko
  1 sibling, 1 reply; 35+ messages in thread
From: Bruno Barbier @ 2024-02-21 16:03 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode


Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

>
> Thanks for the code!
> It is a lot more that I expected.

Note that only the first 5 patchs are real patchs.  The remaining things
are just a demo how it could be used.  The current async (ob-comint)
depends on writing UUIDs in org files, and, that's why I couldn't use it
as a demo.  I'm thinking about dropping the patch:

   'ob-core async: Add org-babel--async tools [2/5]'

and just provide an other new keyword (feedbacks-with) so that anybody
may plug its own feedback process.

> I have many questions ;)

Thanks! They are welcome :-)


>> The new API itself is more about how to wait for and display one block
>> result.  So, it's not really aware of sessions.
>
> I think that it is important to think about sessions. For non-sessions,
> we may execute the code in parallel, disregarding already running
> execution. In contrast, for sessions, we need to maintain some queue and
> wait until previous execution finishes before running next (if multiple
> session blocks are executed in quick succession).

I agree.  My proposed patch leaves this open to further work.

My proposed patch allows to have one way, in ob-core, to display
progress of asynchronous executions: like not started, 10%, success,
failed; and a way to plug in asynchronous executions in ob-core.

I've included some demo code just to show how that API could be used.


> It may also be necessary to provide an UI listing the queued and running
> execution. Users should be able to stop "stale" processes if they are
> defunc (consider infinite loop).

That looks nice but that may be too limiting.  Like it will probably forbid
to execute something asynchronously using elisp threads.  Anyway, that
outside the intended scope of my set of patchs.


> What is more important is when users, for example, remove the whole
> subtree containing the place where the execution result is to be
> inserted. Or when users perform edits at or around that place where the
> result is to be inserted. Or consider subtree with pending result
> refiled elsewhere (different file or different place in the same file);
> or archived.

My patch definitely needs to address those, even if only by raising an
error.

> Or consider user running an src block, quickly editing it, and then
> running again. What should we do then? Should we stop the first running
> process?

To keep things simple and generic, I would just forbid the user to
reschedule the task until the previous outcome is available.


> I have several general questions:
>
> - what if append/prepend to result asynchronously?

You mean if org is executing several times the same code concurrently?
I think we should forbid it.

> - what if we replace the result, call async evaluation two times, and they arrive in opposite order (first called is calculated after the second)?

"One execution at a given time" will solve this too :-)

> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?

In the asynchronous case, I've decided to leave the overlay in place
in case of errors, with the error description.  Clicking on that error
pops up the same kind of popups as in the synchronous case.  You can
give it a try using examples of errors my demo file:

   scratch/bba-ob-core-async/my-async-tests.org

(Please use the updated set of patchs that I've sent today, else, you
will have to fix the required libraries).


> Note that we already have a WIP an asynchronous API in the works.
> Check out `org-async-call' in
> https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
> If possible, we should reuse it.

Thanks. I will have a look at it.


Bruno


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-21 16:03                   ` Bruno Barbier
@ 2024-02-23 12:11                     ` Ihor Radchenko
  2024-02-23 13:24                       ` Bruno Barbier
  0 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-23 12:11 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

> Note that only the first 5 patchs are real patchs.  The remaining things
> are just a demo how it could be used.  The current async (ob-comint)
> depends on writing UUIDs in org files, and, that's why I couldn't use it
> as a demo.  I'm thinking about dropping the patch:
>
>    'ob-core async: Add org-babel--async tools [2/5]'
>
> and just provide an other new keyword (feedbacks-with) so that anybody
> may plug its own feedback process.

May you please clarify if adding the new code block parameter that
defines custom execute function is something you want to add to Org mode
or just a helper code to demo you main patch?

>> Or consider user running an src block, quickly editing it, and then
>> running again. What should we do then? Should we stop the first running
>> process?
>
> To keep things simple and generic, I would just forbid the user to
> reschedule the task until the previous outcome is available.

This may not be as trivial as one might think.
Consider

#+name: foo
#+begin_src bash :async yes
...
#+end_src

#+results: foo
<pending results>

Another code block with the same name will write results under
#+results: foo
#+name: foo
#+begin_src bash :async yes
...
#+end_src

We can currently have multiple code blocks writing to the same place.
And async execution should not only check is the current src block is
scheduled, but also whether we are attempting to write to a place where
no other running block is scheduled to write.

>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>
> In the asynchronous case, I've decided to leave the overlay in place
> in case of errors, with the error description.  Clicking on that error
> pops up the same kind of popups as in the synchronous case.

Error buffer does not necessarily appear on failure. When the code leads
to process writing to stderr, we also display error buffer. Even if the
process exits with 0 code.


Also, your code currently creates overlays unconditionally.
However, when we have :results silent or :results none, Org babel must
not modify buffer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-23 12:11                     ` Ihor Radchenko
@ 2024-02-23 13:24                       ` Bruno Barbier
  2024-02-24 11:59                         ` Ihor Radchenko
  0 siblings, 1 reply; 35+ messages in thread
From: Bruno Barbier @ 2024-02-23 13:24 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> Note that only the first 5 patchs are real patchs.  The remaining things
>> are just a demo how it could be used.  The current async (ob-comint)
>> depends on writing UUIDs in org files, and, that's why I couldn't use it
>> as a demo.  I'm thinking about dropping the patch:
>>
>>    'ob-core async: Add org-babel--async tools [2/5]'
>>
>> and just provide an other new keyword (feedbacks-with) so that anybody
>> may plug its own feedback process.
>
> May you please clarify if adding the new code block parameter that
> defines custom execute function is something you want to add to Org mode
> or just a helper code to demo you main patch?

Yes. (I thought about adding it in a separate request;  but that would be
simpler to include it.)


>>> Or consider user running an src block, quickly editing it, and then
>>> running again. What should we do then? Should we stop the first running
>>> process?
>>
>> To keep things simple and generic, I would just forbid the user to
>> reschedule the task until the previous outcome is available.
>
> This may not be as trivial as one might think.
> Consider
>
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src
>
> #+results: foo
> <pending results>
>
> Another code block with the same name will write results under
> #+results: foo
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src

> We can currently have multiple code blocks writing to the same place.
> And async execution should not only check is the current src block is
> scheduled, but also whether we are attempting to write to a place where
> no other running block is scheduled to write.

Today, the asynchronous execution is attached to the result; the source
code itself is not locked and stays editable.

With the current implementation, trying your example, the second execution fails with:
    Error running timer: (user-error "Cannot modify an area being
    updated")

So, it's already OK I guess.

I should improve the behavior (to fail before launching the second
execution).  But, the current design should work in that case.


>
>>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How will it interact with asynchronous evaluation? What if we have multiple simultaneously arriving error notifications?
>>
>> In the asynchronous case, I've decided to leave the overlay in place
>> in case of errors, with the error description.  Clicking on that error
>> pops up the same kind of popups as in the synchronous case.
>
> Error buffer does not necessarily appear on failure. When the code leads
> to process writing to stderr, we also display error buffer. Even if the
> process exits with 0 code.

Got it.  The current design adds the popup on failure; I should make it
more flexible to allow to configure a popup on success too.


> Also, your code currently creates overlays unconditionally.
> However, when we have :results silent or :results none, Org babel must
> not modify buffer.

Yes. I'll fix it (and that one was in my TODO list :-))

But if the execution fails, I guess I'll need to provide some feedbacks
anyway, but I don't know yet how (asynchronous popups are not an option,
modifying the file neither).

Thanks for your review and questions,

Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-23 13:24                       ` Bruno Barbier
@ 2024-02-24 11:59                         ` Ihor Radchenko
  2024-02-24 16:42                           ` Bruno Barbier
  0 siblings, 1 reply; 35+ messages in thread
From: Ihor Radchenko @ 2024-02-24 11:59 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Matt, Jack Kamm, emacs-orgmode

Bruno Barbier <brubar.cs@gmail.com> writes:

>> May you please clarify if adding the new code block parameter that
>> defines custom execute function is something you want to add to Org mode
>> or just a helper code to demo you main patch?
>
> Yes. (I thought about adding it in a separate request;  but that would be
> simpler to include it.)

I have doubts about how useful such parameter would be for _users_. It
is certainly useful for developers of new babel backends, but I do not
see how users will use it.

I'd prefer to discuss this in a separate thread.

>> Error buffer does not necessarily appear on failure. When the code leads
>> to process writing to stderr, we also display error buffer. Even if the
>> process exits with 0 code.
>
> Got it.  The current design adds the popup on failure; I should make it
> more flexible to allow to configure a popup on success too.

By popup, do you mean an overlay where clicking on it will raise the
usual `org-babel-eval-error-notify' buffer?
If yes, what happens if a user executes a code block, it fails, and the
user executes it again without looking at the error? (Second execution
may be triggered from a different place, indirectly, via noweb ref or
similar).

> But if the execution fails, I guess I'll need to provide some feedbacks
> anyway, but I don't know yet how (asynchronous popups are not an option,
> modifying the file neither).

We need something more than just an overlay. Maybe some kind of babel
execute history (in tabulated-list-mode buffer), keeping information
about execution stats and stderr data.

We may provide a mode-line or fringe indicator that will warn user if
something went wrong with recent executions. Something akin compile-mode
mode line indicator listing errors and warnings during compilation.

(Handling asynchronous messages is actually a complex topic. It has been
previously discussed on emacs-devel as well. See
<https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-24 11:59                         ` Ihor Radchenko
@ 2024-02-24 16:42                           ` Bruno Barbier
  2024-02-24 19:54                             ` Matt
  0 siblings, 1 reply; 35+ messages in thread
From: Bruno Barbier @ 2024-02-24 16:42 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Matt, Jack Kamm, emacs-orgmode



Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>>> May you please clarify if adding the new code block parameter that
>>> defines custom execute function is something you want to add to Org mode
>>> or just a helper code to demo you main patch?
>>
>> Yes. (I thought about adding it in a separate request;  but that would be
>> simpler to include it.)
>
> I have doubts about how useful such parameter would be for _users_. It
> is certainly useful for developers of new babel backends, but I do not
> see how users will use it.
>
> I'd prefer to discuss this in a separate thread.

Sure. I'll keep it for an other thread.


>>> Error buffer does not necessarily appear on failure. When the code leads
>>> to process writing to stderr, we also display error buffer. Even if the
>>> process exits with 0 code.
>>
>> Got it.  The current design adds the popup on failure; I should make it
>> more flexible to allow to configure a popup on success too.
>
> By popup, do you mean an overlay where clicking on it will raise the
> usual `org-babel-eval-error-notify' buffer?
> If yes, what happens if a user executes a code block, it fails, and the
> user executes it again without looking at the error? (Second execution
> may be triggered from a different place, indirectly, via noweb ref or
> similar).

Current behavior is to remove the error, and, then re-execute: the
previous errors are gone.  My guess is that this could be improve later,
without breaking the API.


>> But if the execution fails, I guess I'll need to provide some feedbacks
>> anyway, but I don't know yet how (asynchronous popups are not an option,
>> modifying the file neither).
>
> We need something more than just an overlay. Maybe some kind of babel
> execute history (in tabulated-list-mode buffer), keeping information
> about execution stats and stderr data.
>
> We may provide a mode-line or fringe indicator that will warn user if
> something went wrong with recent executions. Something akin compile-mode
> mode line indicator listing errors and warnings during compilation.

I was thinking about using a fringe indicator, for warnings on success (as I'm
removing all overlays on success).

Keeping a buffer of all asynchronous executions looks like a good idea.
I'll think about it.


> (Handling asynchronous messages is actually a complex topic. It has been
> previously discussed on emacs-devel as well. See
> <https://yhetil.org/emacs-devel/838t59j821.fsf@gnu.org/>)

I'll look at this.  Thanks!

I'll publish a branch soon; it will be a major rewrite of my current
proposal.  It should be less confusing and, I hope, address some of your
comments.

Thanks again for your questions and feedbacks,

Bruno

>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-24 16:42                           ` Bruno Barbier
@ 2024-02-24 19:54                             ` Matt
  2024-02-28 10:21                               ` Bruno Barbier
  0 siblings, 1 reply; 35+ messages in thread
From: Matt @ 2024-02-24 19:54 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode


 ---- On Sat, 24 Feb 2024 17:42:54 +0100  Bruno Barbier 

 > I'll publish a branch soon; it will be a major rewrite of my current
 > proposal.  It should be less confusing and, I hope, address some of your
 > comments.
 > 
 > Thanks again for your questions and feedbacks,

You're welcome and thanks for sharing your ideas.

Any lack of comments from me recently is just limited time and trying to focus on maintenance.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode




^ permalink raw reply	[flat|nested] 35+ messages in thread

* Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))
       [not found]                   ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
@ 2024-02-28 10:18                     ` Bruno Barbier
  0 siblings, 0 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-28 10:18 UTC (permalink / raw)
  To: emacs-orgmode; +Cc: Ihor Radchenko, Jack Kamm, Matt


Hi,

Bruno Barbier <brubar.cs@gmail.com> writes:

> Hi Matt,
[...]
>> Since this thread is dedicated to blocking, let me share my thoughts on that subject.
>
> I guess I should start a new thread then :)


I finally got a new version.  I've renamed the proposed feature "pending
contents", that is, some contents that something will update *later*.
With that feature, source code blocks and dynamic blocks can be made
asynchronous … and more!

I've publish the proposed changes as a branch.  You can fecth that
branch there:
┌────
│ repo:   https://framagit.org/brubar/org-mode-mirror
│ branch: bba-pending-contents
└────

The file [my-async-tests.org] describes the proposed changes, its
implementation and contains examples to play with it (shell, python,
ruby, results{append,prepend,silent}, inline blocks, multithreading,
dynamic asynchronous blocks, source codes that depend on other blocks
{post, stdin}, etc.).

I've tried to fully describe the feature in the new section "Pending
contents", in the file `lisp/org-macs.el'.

Testing it in a bare Emacs should now work (I'm using 30.0.50).

*DO NOT TEST* in your production Emacs: this is *alpha* software and it
hooks itself deeply into Emacs (kill hooks), and (only when trying the
multithread examples) uses thread mutexes&locks.

Here is a simple recipe to test the proposed "pending contents" feature:
┌────
│ git clone -b bba-pending-contents https://framagit.org/brubar/org-mode-mirror
│ cd org-mode-mirror
│ make compile
│ cd scratch/bba-pending-contents/
│ emacs -q -L ../../lisp my-async-tests.org
└────


Thanks you all for your previous comments.  I hope I've addressed most
of them.

Comments, critiques, ideas, corrections are most welcome.

Thanks,

Bruno


[my-async-tests.org]
<https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org>


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
  2024-02-24 19:54                             ` Matt
@ 2024-02-28 10:21                               ` Bruno Barbier
  0 siblings, 0 replies; 35+ messages in thread
From: Bruno Barbier @ 2024-02-28 10:21 UTC (permalink / raw)
  To: Matt; +Cc: Ihor Radchenko, Jack Kamm, emacs-orgmode


Hi Matt,

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 24 Feb 2024 17:42:54 +0100  Bruno Barbier 
>
>  > I'll publish a branch soon; it will be a major rewrite of my current
>  > proposal.  It should be less confusing and, I hope, address some of your
>  > comments.
>  > 
>  > Thanks again for your questions and feedbacks,
>
> You're welcome and thanks for sharing your ideas.
>
> Any lack of comments from me recently is just limited time and trying to focus on maintenance.

Thanks for letting me know.

No worries. Take care.

Bruno



^ permalink raw reply	[flat|nested] 35+ messages in thread

end of thread, other threads:[~2024-02-28 10:21 UTC | newest]

Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-01 11:58 [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Ihor Radchenko
2024-02-01 14:56 ` Bruno Barbier
2024-02-03  1:30   ` Jack Kamm
2024-02-04 15:07     ` Ihor Radchenko
2024-02-05  1:37       ` Jack Kamm
2024-02-05 14:29         ` Ihor Radchenko
2024-02-06 19:24           ` Bruno Barbier
2024-02-07 16:19             ` Ihor Radchenko
2024-02-07 17:40               ` Bruno Barbier
2024-02-08  3:21             ` Jack Kamm
2024-02-15 20:02             ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Matt
2024-02-16 17:52               ` Bruno Barbier
2024-02-18 21:14                 ` Matt
2024-02-19  0:31                   ` Jack Kamm
2024-02-20 10:28                   ` Ihor Radchenko
2024-02-20 10:46                     ` tomas
2024-02-20 11:00                       ` Ihor Radchenko
2024-02-20 11:03                         ` tomas
2024-02-21 15:27                   ` Bruno Barbier
     [not found]                   ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
2024-02-28 10:18                     ` Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) Bruno Barbier
2024-02-19  0:15                 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
2024-02-21 15:43                   ` Bruno Barbier
2024-02-19  9:06                 ` Ihor Radchenko
2024-02-19 19:47                   ` Matt
2024-02-19 20:10                     ` Ihor Radchenko
2024-02-20  8:32                     ` Ihor Radchenko
2024-02-20 17:04                     ` Jack Kamm
2024-02-21 16:03                   ` Bruno Barbier
2024-02-23 12:11                     ` Ihor Radchenko
2024-02-23 13:24                       ` Bruno Barbier
2024-02-24 11:59                         ` Ihor Radchenko
2024-02-24 16:42                           ` Bruno Barbier
2024-02-24 19:54                             ` Matt
2024-02-28 10:21                               ` Bruno Barbier
2024-02-08  3:26           ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).