Hi Matt, Jack, Ihor, Sorry for the late reply. Cleaning the code took me longer than expected. Jack Kamm writes: > Bruno Barbier writes: > >> FWIW, I've been trying to use asynchronous blocks for everything, not >> only the source blocks that are based on the comint mode. >> ... > > Sounds interesting, a couple questions: > > 1. Which languages have you implemented/tested this on? I'm not using it with official org backends (yet). I'm using it with several custom backends that I'm working on. One of the backend delegate the block executions to emacs subprocesses: so I have a kind of asynchronous executions for free for any language, including elisp itself. > 2. Does it apply for sessions, nonsessions, or both? > The new API itself is more about how to wait for and display one block result. So, it's not really aware of sessions. But, I usually try to think as "no session" as a "one shot" session (like Matt wrote in his email). So, in that sense, it works for both anyway ;-) > Also interesting, I think it's worth exploring/testing this overlay idea > out. Does that mean that output is asynchronously printing into the Org > buffer? It sounds cool but I wonder if it might cause problems while > trying to edit another part of the buffer. Currently, I've limited the progress feedback to fit on one line to avoid anoying vertical display jumps. When the computation is successful, the result is inserted as usual, and, that may be annoying; when it updates a previous result of the same height, it's ok. Else, we could fold it to stay on one line too (using the overlay), until the user explicitly request to see it. Jack Kamm writes: > > That all sounds reasonable...if you work on this, let me know if you > want any help with testing. Thanks. I'll definitely appreciate your help, to test the current code with the Python backend or any other backend if you prefer. Matt writes: > If I followed correctly, the topic switched to discussing async > generally within Babel. I've started a new thread accordingly. Indeed. Thank you! > > FWIW, I've been trying to use asynchronous blocks for everything, not only the source blocks that are based on the comint mode. > > I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with. Good to know. Let's we make this happen! > > I think it would be good if ob-core itself could provide an asynchronous API. > > Fortunately or unfortunately, depending on how you look at it, Babel already does. > > The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default. > For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. > I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API: > > - "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. > > - "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. > > - "Synchronous" means that execution prevents the user from editing the document while results are obtained. > > - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained. I mostly think the same. Sessions (including the "none" session) definitely need some generic API and some generic tests that all backends could just reuse. To execute Python blocks, using the proposed async API: - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python). - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...). - The "no session" case is just about creating a new unique session and throwing it away immediately. But, some users may rely on some particular behavior, of the current implementations, that may be hard to implement in such a generic way. > The use of the overlay is a really cool idea! > > I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document). The backend is free to transform a failure into a success to make that result available though. > However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise. You should be able to test my current implementation, see below. It's almost too visual in my opinion. But, that's probably something that we should make easy to configure. > > If that technique looks safe enough and interesting, I can prepare a set of patches so that we can discuss it further and, maybe, add it in Org. > > Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with. Good to know! So, here we go. You'll find attach a set of patchs. It works for me with Emacsc 30.50 and 9.7-pre (from today). I didn't check yet that the code and the commits follow all the guidelines. This is just a preliminary version for feedbacks. Corrections/critiques are welcome, but don't check the details until I check them myself. The 5 first patchs provide an API to handle asynchronous execution in ob-core, i.e. an API to report progress and to insert results in the asynchronous case. The 5th one isn't really about asynchronicity; but it adds a new keyword `:execute-with' that allows to delegate block executions to some other package; it's useful for testing or plugging other execution engines. That's a cleaned-up version of what I been using myself for a while, with 4 different values for `:execute-with'. The remaining patchs are new code that I've just written to show how to use this new API. I tried first to use it for ob-python, as an example. I just needed to figure out where to place the callbacks ... well ... "just" ... :) So, I decided to rewrite the whole thing, taking code from the synchronous case (following `org-babel-python-evaluate-session'). I also created a package that contains all the functions that should be reusable for any language. The patch [1] adds some generic functions to help dealing with asynchronicity (process, comint, etc.). The patch [2] shows a new possible way to execute python code blocks, both synchronously and asynchronously, with or without sessions. You should just need to open [3] and follow what's written there, and execute the existing bash and python code blocks. Note that this will create a folder `scratch/bba-ob-core-async' in your repository to place the temporary files there. If you know a better way to do this, let me know, thanks. I think the first 5 patchs could be included almost as-is in org. About the emaining ones, I'm not sure exactly how we should proceed. Let me know if you need help to test it. Feedbacks, corrections, critiques, etc are most welcome! Thanks! Bruno [1] lisp/org-elib-async.el: New package about async helpers [2] scratch/bba-ob-core-async: Some temporary test files [3] scratch/bba-ob-core-async/my-async-tests.org