From: Matt <matt@excalamus.com>
To: "Bruno Barbier" <brubar.cs@gmail.com>
Cc: "Ihor Radchenko" <yantar92@posteo.net>,
"Jack Kamm" <jackkamm@gmail.com>,
"emacs-orgmode" <emacs-orgmode@gnu.org>
Subject: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])
Date: Thu, 15 Feb 2024 21:02:05 +0100 [thread overview]
Message-ID: <18dae5cab1d.bf1c7563863897.4896289306902277373@excalamus.com> (raw)
In-Reply-To: <65c2875f.050a0220.caf6d.8291@mx.google.com>
If I followed correctly, the topic switched to discussing async generally within Babel. I've started a new thread accordingly.
---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier
> FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode.
I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with.
> I think it would be good if ob-core itself could provide an asynchronous API.
Fortunately or unfortunately, depending on how you look at it, Babel already does.
The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default.
For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API:
- "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls.
- "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls.
- "Synchronous" means that execution prevents the user from editing the document while results are obtained.
- "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained.
> I've modified my Org so that it does have such an API. This is work in progress; let me describe it.
>
> I've modified ob-core itself to allow asynchronicity. In the asynchrosous case, instead of calling:
>
> (org-babel-execute:LANG body params)
>
> I'm calling:
>
> (org-babel-schedule:LANG body params handle-result)
>
> where `org-babel-schedule:LANG' is in charge of calling `handle-result' with the result (or the error) when it is known; `handle-result' takes care to call `org-babel-insert-result' at the correct place (and `org-babel-insert-result' is only called with a real result).
>
> While the execution is pending, I'm using the same technique that Org is using when a source block is being edited: the result is left untouched, but below an overlay. The overlay is used to know where to insert the result and to display the status/progress of the execution. If the file is closed and the execution fails, nothing is lost, the old result is still available.
The use of the overlay is a really cool idea!
I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else.
However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise.
> If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org.
Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with.
--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
next prev parent reply other threads:[~2024-02-15 20:03 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-01 11:58 [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Ihor Radchenko
2024-02-01 14:56 ` Bruno Barbier
2024-02-03 1:30 ` Jack Kamm
2024-02-04 15:07 ` Ihor Radchenko
2024-02-05 1:37 ` Jack Kamm
2024-02-05 14:29 ` Ihor Radchenko
2024-02-06 19:24 ` Bruno Barbier
2024-02-07 16:19 ` Ihor Radchenko
2024-02-07 17:40 ` Bruno Barbier
2024-02-08 3:21 ` Jack Kamm
2024-02-15 20:02 ` Matt [this message]
2024-02-16 17:52 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Bruno Barbier
2024-02-18 21:14 ` Matt
2024-02-19 0:31 ` Jack Kamm
2024-02-20 10:28 ` Ihor Radchenko
2024-02-20 10:46 ` tomas
2024-02-20 11:00 ` Ihor Radchenko
2024-02-20 11:03 ` tomas
2024-02-21 15:27 ` Bruno Barbier
[not found] ` <notmuch-sha1-61e086e33bd1faf1a123c1b0353cf2102c71bdac>
2024-02-28 10:18 ` Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...)) Bruno Barbier
2024-03-02 10:03 ` Ihor Radchenko
2024-03-02 10:57 ` Bruno Barbier
2024-03-02 11:13 ` Ihor Radchenko
2024-03-02 18:06 ` Bruno Barbier
[not found] ` <notmuch-sha1-d2799a191385bf51811d7788856a83b4f5a1fe3b>
2024-03-07 17:08 ` Bruno Barbier
2024-03-07 18:29 ` Ihor Radchenko
2024-03-08 14:19 ` Bruno Barbier
2024-03-13 9:48 ` Ihor Radchenko
2024-03-19 9:33 ` Bruno Barbier
2024-03-20 10:23 ` Ihor Radchenko
2024-03-21 10:06 ` Bruno Barbier
2024-03-21 12:15 ` Ihor Radchenko
2024-03-25 17:46 ` Bruno Barbier
2024-03-27 11:29 ` Ihor Radchenko
2024-03-30 22:53 ` Rudolf Adamkovič
2024-04-04 16:35 ` Bruno Barbier
2024-04-04 16:33 ` Bruno Barbier
2024-04-11 11:44 ` Ihor Radchenko
2024-04-19 11:23 ` Bruno Barbier
2024-04-20 10:07 ` Ihor Radchenko
2024-02-19 0:15 ` Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) Jack Kamm
2024-02-21 15:43 ` Bruno Barbier
2024-02-19 9:06 ` Ihor Radchenko
2024-02-19 19:47 ` Matt
2024-02-19 20:10 ` Ihor Radchenko
2024-02-20 8:32 ` Ihor Radchenko
2024-02-20 17:04 ` Jack Kamm
2024-02-21 16:03 ` Bruno Barbier
2024-02-23 12:11 ` Ihor Radchenko
2024-02-23 13:24 ` Bruno Barbier
2024-02-24 11:59 ` Ihor Radchenko
2024-02-24 16:42 ` Bruno Barbier
2024-02-24 19:54 ` Matt
2024-02-28 10:21 ` Bruno Barbier
2024-02-08 3:26 ` [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)] Jack Kamm
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=18dae5cab1d.bf1c7563863897.4896289306902277373@excalamus.com \
--to=matt@excalamus.com \
--cc=brubar.cs@gmail.com \
--cc=emacs-orgmode@gnu.org \
--cc=jackkamm@gmail.com \
--cc=yantar92@posteo.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).