---- On Fri, 16 Feb 2024 18:52:22 +0100 Bruno Barbier > Sorry for the late reply. Cleaning the code took me longer than expected. No need to apologize, we're all volunteers here :) > Feedbacks, corrections, critiques, etc are most welcome! Thank you for sharing! If I understand correctly, there are several independent topics the code addresses: | topic | manner addressed | |------------------+------------------------------------------------| | execution status | using overlays to communicate execution status | | locating results | using overlays to locate results | | blocking | making all execution asynchronous | |------------------+------------------------------------------------| I suggest these be discussed in separate threads. > > The use of the overlay is a really cool idea! > > > > I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. > > I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document). The backend is free to transform a failure into a success to make that result available though. You can disregard my hesitation on this point. I had not run your code yet and had misunderstood how it worked. Since this thread is dedicated to blocking, let me share my thoughts on that subject. > To execute Python blocks, using the proposed async API: > > - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python). > > - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...). > > - The "no session" case is just about creating a new unique session and throwing it away immediately. This is an interesting idea, feeding all processes through the same mechanism. Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]]. Processes are synchronous or asynchronous. Three primitives exist in Emacs for making processes: 1. make-process (asynchronous) 2. call-process (synchronous) 3. call-process-region (synchronous) There exist several convenience wrappers for these. AFAIK, everything reduces to these three primitives. For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal. This background-foreground distinction is called "job control". Output from a process typically goes to a buffer. This may be changed and instead handle output with a filter function. =call-process= has an option to directly send output to a file. Subprocesses inherent the =default-directory= and the environment from Emacs. The environment may be changed using =process-environment=. There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe". The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.). In my previous message, I divided evaluation into 4 types: - non-persistent vs. persistent - synchronous vs. asynchronous I find the approach of feeding everything through, fundamentally, =make-process= interesting because if we make a chart of the 4 types, we see some ambiguities: | | non-persistent | persistent | |--------------+----------------+--------------| | synchronous | call-process | ??? | |--------------+----------------+--------------| | asynchronous | ??? | make-process | |--------------+----------------+--------------| To make a non-persistent asynchronous process, the first thing that comes to mind is =async-shell-command=. However, as the code shows, another option is to use =make-process= and throw away the state (the process buffer). I'm not sure how we could make a persistent, synchronous process. Persistence is achieved, currently, by a process buffer. Is there another way persistence may be achieved? Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. All that is mainly academic. The idea I think most interesting is using a single primitive to handle all evaluation. It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached). It works as follows. I defined a new Babel "language" called "blub". Blub could be a shell, python, ruby, whatever. I wanted to test that the implementation could work with different interpreters or compilers. Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out. Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. Curious to hear other people's thoughts! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode