Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt <matt@excalamus.com>]

If I followed correctly, the topic switched to discussing async generally within Babel.  I've started a new thread accordingly.

 ---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier

 > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.

I've been trying to figure out how to make everything async by default.  I'm super interested to see what you've come up with.

 > I think it would be good if ob-core itself could provide an asynchronous API.

Fortunately or unfortunately, depending on how you look at it, Babel already does.

The challenge is that the Org Babel API has grown piecemeal over 14 years.  It's been written by several authors with limited time and knowledge.  The result, while powerful and useful, is a bit of a hodgepodge.  A prime example is that the concepts of "persistence" and "synchronicity" are conflated.  "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal.  Emacs provides primitives that could make non-session blocks asynchronous.  It's historical accident that blocks aren't async by default.

For me, the issue is that the Babel API needs some high level perspective in order to make it consistent.  I see the following terms as guides.  If we can separate these concepts within the API, then Babel to *feel* like an API:

- "Session" means a shell environment is "persistent."  Each call is executed in the same environment.  State exists between calls.

- "Non-session" means a shell environment is "temporary."  Each call is executed in an independent environment.  State does not exist between calls.

- "Synchronous" means that execution prevents the user from editing the document while results are obtained.

- "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

 > I've modified my Org so that it does have such an API.  This is work in progress; let me describe it.
 >
 > I've modified ob-core itself to allow asynchronicity.  In the asynchrosous case, instead of calling:
 >
 >   (org-babel-execute:LANG body params)
 >
 > I'm calling:
 >
 >   (org-babel-schedule:LANG body params handle-result)
 >
 > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result).
 >
 > While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay.  The overlay is used to know where to insert the result and to display the status/progress of the execution.  If the file is closed and the execution fails, nothing is lost, the old result is still available.

The use of the overlay is a really cool idea!

I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.

However, using an overlay to communicate "the process is still running" could be good.  We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.

 > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.

Please do!  I'm super interested.  I've put a lot of thought into how we might make Babel async by default.  I'm excited to see your interest in the topic and look forward to seeing what you've come up with.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode