Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt <matt@excalamus.com>]

[-- Attachment #1: Type: text/plain, Size: 7215 bytes --]


 ---- On Fri, 16 Feb 2024 18:52:22 +0100  Bruno Barbier 

 > Sorry for the late reply.  Cleaning the code took me longer than expected.

No need to apologize, we're all volunteers here :)

 > Feedbacks, corrections, critiques, etc are most welcome!

Thank you for sharing!

If I understand correctly, there are several independent topics the code addresses:

| topic            | manner addressed                               |
|------------------+------------------------------------------------|
| execution status | using overlays to communicate execution status |
| locating results | using overlays to locate results               |
| blocking         | making all execution asynchronous              |
|------------------+------------------------------------------------|

I suggest these be discussed in separate threads.

 > > The use of the overlay is a really cool idea!
 > >
 > > I hesitate to say that's a good way to convey success or failure.  If a process failed, I want to see the output which tells me why so that I can correct it.  Or, I might actually want the failure output.  Maybe I want to literally demonstrate what a code failure looks like.  Maybe I want to use that output in another block.  For example, shell blocks have multiple output types.  A shell process may return standard output/error or a failure code.  The result of the failure may trigger something else.
 > 
 > I'm not sure I fully understand what you mean. The API just assumes the backend returns the outcome: either success or failure, where failure means "no result" (the previous result, if it exists, is even preserved in the document).  The backend is free to transform a failure into a success to make that result available though.

You can disregard my hesitation on this point.  I had not run your code yet and had misunderstood how it worked.

Since this thread is dedicated to blocking, let me share my thoughts on that subject.

 > To execute Python blocks, using the proposed async API:
 > 
 >    - I've (re)implemented the "asynchronous with session" case (copying/pasting the relevant part from ob-python).
 >    
 >    - The "synchronous case" is just artificially blocking the user until the asynchronous result is known (which looks incredibly tricky to implement if even possible...).
 >    
 >    - The "no session" case is just about creating a new unique session and throwing it away immediately.

This is an interesting idea, feeding all processes through the same mechanism.

Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].

Processes are synchronous or asynchronous.

Three primitives exist in Emacs for making processes:

1. make-process (asynchronous)
2. call-process (synchronous)
3. call-process-region (synchronous)

There exist several convenience wrappers for these.  AFAIK, everything reduces to these three primitives.  For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal.  This background-foreground distinction is called "job control".

Output from a process typically goes to a buffer.  This may be changed and instead handle output with a filter function.  =call-process= has an option to directly send output to a file.

Subprocesses inherent the =default-directory= and the environment from Emacs.  The environment may be changed using =process-environment=.

There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe".  The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.).

In my previous message, I divided evaluation into 4 types:

  - non-persistent vs. persistent 
  - synchronous vs. asynchronous

I find the approach of feeding everything through, fundamentally, =make-process= interesting because if we make a chart of the 4 types, we see some ambiguities:

|              | non-persistent | persistent   |
|--------------+----------------+--------------|
| synchronous  | call-process   | ???          |
|--------------+----------------+--------------|
| asynchronous | ???            | make-process |
|--------------+----------------+--------------|

To make a non-persistent asynchronous process, the first thing that comes to mind is =async-shell-command=.  However, as the code shows, another option is to use =make-process= and throw away the state (the process buffer).

I'm not sure how we could make a persistent, synchronous process.  Persistence is achieved, currently, by a process buffer.  Is there another way persistence may be achieved?  Of course, this ignores whether a persistent, synchronous process is even desirable.  Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation.

All that is mainly academic.  The idea I think most interesting is using a single primitive to handle all evaluation.

It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached).

It works as follows.  I defined a new Babel "language" called "blub".   Blub could be a shell, python, ruby, whatever.  I wanted to test that the implementation could work with different interpreters or compilers.  Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out.

Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the same directory.  Run the first block in ob-blub-test.org.  This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again.  Hopefully ob-blub.el is documented sufficiently for you to experiment.

The blub implementation has the same shortcomings, at least for shells, as the current shell implementation.  It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally.  The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings.  If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil).

The challenge I've found with Babel is figuring out how to make the changes.  My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation.  I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make.

Curious to hear other people's thoughts!

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[-- Attachment #2: ob-blub-test.org --]
[-- Type: application/octet-stream, Size: 2532 bytes --]

#+begin_src emacs-lisp :results silent :var HERE=(buffer-file-name)
;; load blub
(add-to-list 'load-path (file-name-directory HERE))
(require 'ob-blub)
(org-babel-do-load-languages 'org-babel-load-languages '((blub . t)))

;; reset uuid on failed block
(setq my-org-babel-comint--async-uuid nil)

;; configure shell
(setq org-babel-blub-interpreter shell-file-name)
(setq org-babel-blub-remove-prompt-command "PROMPT_COMMAND=;PS1=;PS2=;")
(setq org-babel-blub-output-start-delimiter "echo \"start_%s\"")
(setq org-babel-blub-output-end-delimiter "echo \"end_%s\"")
(setq org-babel-blub-interpreter-args '())

;; configure python
;; (setq org-babel-blub-interpreter "python3")
;; (setq org-babel-blub-remove-prompt-command "import sys;sys.ps1='';sys.ps2='';")
;; (setq org-babel-blub-output-start-delimiter "print(\"start_%s\")")
;; (setq org-babel-blub-output-end-delimiter "print(\"end_%s\")")
;; (setq org-babel-blub-interpreter-args '())

;; configure ruby
;; (setq org-babel-blub-interpreter "ruby")  ; for non-sessions
;; (setq org-babel-blub-interpreter "irb")  ; for sessions
;; (setq org-babel-blub-remove-prompt-command nil)
;; (setq org-babel-blub-output-start-delimiter "puts \"start_%s\"")
;; (setq org-babel-blub-output-end-delimiter "puts \"end_%s\"")
;; (setq org-babel-blub-interpreter-args '("--noprompt" "--noreadline" "--nomultiline"))
#+end_src

* Non-persistent
** shell
#+begin_src blub
echo "hello"
sleep 3
echo "world!"
#+end_src

#+RESULTS:
: hello
: world!

#+begin_src blub
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub
import time
print("hello without session")
time.sleep(3)
print("world")
#+end_src

** ruby
#+begin_src blub
puts "cruel world"
sleep(3)
puts "good-bye"
#+end_src

* Persistent
** shell
#+begin_src blub :session *shell-blubber*
echo "hello"
sleep 3
echo "world!"
#+end_src

#+begin_src blub :session *shell-blubber*
==echo "hello"
sleep 3
echo "world!"
#+end_src

** python
#+begin_src blub :session *python-blubber*
import time
print("hello")
time.sleep(5)
print("world")
#+end_src

#+RESULTS:

#+begin_src blub :session *python-blubber*
import time
print("good-bye")
time.sleep(5)
print("cruel world")
#+end_src

** ruby
#+begin_src blub :session *ruby-blubber*
puts "good-bye"
sleep(3)
puts "cruel world"
#+end_src

* Failures
#+begin_src blub :session *bash-blubber*
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
#+end_src

#+begin_src blub :session *shell-blubber* :epilogue echo "bye"
ssh $USER@localhost echo "hi"
#+end_src

[-- Attachment #3: ob-blub.el --]
[-- Type: application/octet-stream, Size: 10821 bytes --]

;; -*- lexical-binding: t -*-

;;; ob-blub.el --- org-babel functions for blub evaluation

;; Copyright (C) Matt Trzcinski

;; Author: Matt Trzcinski
;; Keywords: literate programming, reproducible research
;; Homepage: https://orgmode.org
;; Version: 0.01

;;; License:

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;;; Commentary:
;;
;; A new language, blub, is defined using the Org Babel API
;; (`org-babel-execute:blub').  "Blub" is a dummy language.  If blub
;; is interpreted, set the interpreter using
;; `org-babel-blub-interpreter'.  Compiled languages are not
;; demonstrated here although the implementation may be extended to
;; support compiled languages.
;;
;; The code is split into three sections:
;;
;;   1. Eval
;;   2. Comint
;;   3. Blub
;;
;; Sections are based on their current implements in Org Babel.  It's
;; not clear if these are the "right" categorizations.  For example,
;; should comint evaluation be in ob-comint or ob-eval?
;;
;; Set the following:
;;
;;   `org-babel-blub-interpreter'
;;   `org-babel-blub-interpreter-args'
;;   `org-babel-blub-remove-prompt-command'
;;   `org-babel-blub-output-start-delimiter'
;;   `org-babel-blub-output-end-delimiter'
;;
;; Sessions run in a dedicated process buffer.  Output is captured
;; from between delimiters.  This is similar to the current Babel
;; implementation.  However, unlike the current Babel implementation,
;; both delimiters in this proof of concept are custom.  The current
;; Babel implementation uses the prompt for the start delimiter which
;; causes problems when it changes.
;;
;; For sessions, only a single process may run at a time.  The UUID is
;; stored in `my-org-babel-comint--async-uuid' and the associated Org
;; buffer is stored in `my-org-babel-comint--async-org-buffer'.  If an
;; error occurs that prevents the end delimiter from printing,
;; manually the clear `my-org-babel-comint--async-uuid' to run blocks.
;;
;;   (setq my-org-babel-comint--async-uuid nil)
;;
;; Results are extracted from standard output using a regexp.
;;
;; Noted problems:
;;   - running sudo

;;; Requirements:
(require 'ob)

\f
;;; Eval:
(defun my-org-babel-eval-async (command body)
  "Start process with COMMAND, send BODY to process, get
results.

Since results execute asynchronously, a UUID is returned.  The
UUID is later replaced when the async process finishes."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))
  (let* ((my-process
          (make-process
           :name "my-org-babel-eval-async"
           :buffer "*my-org-babel-eval-async*"
           :command `(,command)
           :connection-type 'pipe
           :sentinel '(lambda (process msg)
                        (cond ((string= msg "finished\n")
                               (my-org-babel-comint-replace-uuid-with-results
                                my-org-babel-comint--async-org-buffer
                                my-org-babel-comint--async-uuid
                                (with-current-buffer (process-buffer process)
                                  (buffer-substring-no-properties (point-min) (point-max))))
                               (setq my-org-babel-comint--async-uuid nil)
                               (let ((kill-buffer-query-functions nil))
                                 (if (kill-buffer "*my-org-babel-eval-async*")
                                     (setq my-org-babel-comint--async-org-buffer nil)))
                               ))))))
    (process-send-string my-process body)
    (process-send-eof my-process))
  my-org-babel-comint--async-uuid)

\f
;;; Comint:
(defvar my-org-babel-comint--async-uuid nil
  "Placeholder for results.")
(defvar my-org-babel-comint--async-org-buffer nil
  "Buffer containing UUID.")

(defun my-org-babel-comint-send-string (session string)
  "Send STRING to comint SESSION."
  (with-current-buffer session
    (goto-char (process-mark (get-buffer-process session)))
    (insert string)
    (comint-send-input)))

(defun my-org-babel-comint-replace-uuid-with-results (buffer uuid results)
  "Replace UUID string in BUFFER with RESULTS string."
  (with-current-buffer buffer
    (save-excursion
      (goto-char (point-min))
      (when (search-forward uuid nil t)
        (org-babel-previous-src-block)
        (let* ((info (org-babel-get-src-block-info))
               (params (nth 2 info))
               (result-params
                (cdr (assq :result-params params))))
          (org-babel-insert-result
           results result-params info))))))

(defun my-org-babel-comint-send-to-session-async (process-buffer &rest body)
  "Send BODY to PROCESS-BUFFER asynchronously."
  (setq my-org-babel-comint--async-uuid (org-id-uuid))
  (setq my-org-babel-comint--async-org-buffer (current-buffer))

  (defun my-org-babel-comint--async-filter (text)
    "Check TEXT for ending delimiter and replace results held by
`my-org-babel-comint--async-uuid' placeholder."
    (cond ((string-match-p (format "end_%s" my-org-babel-comint--async-uuid) text)
           (remove-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

           ;; replace my-org-babel-comint--async-uuid in Org buffer
           (let ((results
                  (with-current-buffer process-buffer ; e.g. "*blubber*"
                    (goto-char (point-min))
                    (re-search-forward
                     ;; Of course, all the problems with regexp happen here.  The goal is getting
                     ;; the text between the delimiters.
                     ;;
                     ;; Some programs (guix shell?) may reset PS1.  So, we can't always match on
                     ;; start_uuid being at the very start of the line.  Match on the one that
                     ;; doesn't have the quote (that is, the result of the echo).
                     (format "[^\"]start_%s[\r\n]*\\(\\(.*[\r\n]+\\)*.*\\)end_%s$"
                             my-org-babel-comint--async-uuid   ; start
                             my-org-babel-comint--async-uuid)  ; end
                     nil nil 1)
                    (let ((match (match-string 1)))
                      (substring-no-properties
                       match)))))

             (my-org-babel-comint-replace-uuid-with-results
              my-org-babel-comint--async-org-buffer
              my-org-babel-comint--async-uuid
              results)

             (setq my-org-babel-comint--async-uuid nil)
             (setq my-org-babel-comint--async-org-buffer nil)))))

  (let* ((proc (get-buffer-process process-buffer)))
    (with-current-buffer process-buffer
      (add-hook 'comint-output-filter-functions 'my-org-babel-comint--async-filter)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-start-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      (insert (car body))
      (comint-send-input nil t)

      (goto-char (process-mark proc))
      ;; TODO need better abstraction
      (insert (format org-babel-blub-output-end-delimiter my-org-babel-comint--async-uuid))
      (comint-send-input nil t)))

  my-org-babel-comint--async-uuid)

\f
;;; Blub:
(defvar org-babel-blub-interpreter
  shell-file-name  ; bash/sh
  ;; "irb"         ; ruby
  ;; "python3"     ; python
  "Blub interpreter command or executable.")

(defvar org-babel-blub-interpreter-args
  '()                                                ; bash
  ;; '("--noprompt" "--noreadline" "--nomultiline")  ; ruby
  "Blub interpreter command or executable arguments.")

(defvar org-babel-blub-remove-prompt-command
  "PROMPT_COMMAND=;PS1=;PS2=;"  ; bash
  ;; "import sys;sys.ps1='';sys.ps2='';"  ; python
  "Command(s) to remove interpreter prompt.")

(defvar org-babel-blub-output-start-delimiter
  "echo \"start_%s\""       ; bash
  ;; "puts \"start_%s\""    ; ruby
  ;; "print(\"start_%s\")"  ; python
  "Format expression for writing the start delimiter to standard
output in blub.")

(defvar org-babel-blub-output-end-delimiter
  "echo \"end_%s\""       ; bash
  ;; "puts \"end_%s\""    ; ruby
  ;; "print(\"end_%s\")"  ; python
  "Format expression for writing the end delimiter to standard
output in blub.")

;; this removes the need for `org-babel-comint-buffer-livep'
(defun org-babel-blub-get-session (session)
  "Return SESSION buffer, making process buffer if none exists."
  (cond ((not (get-buffer-process session))
         (apply
          #'make-comint-in-buffer
          `(,session                     ; process name
            ,session                     ; buffer name
            ,org-babel-blub-interpreter  ; program
            nil                          ; start file
            ,@org-babel-blub-interpreter-args))
         (org-babel-comint-wait-for-output session)
         (if org-babel-blub-remove-prompt-command
             (my-org-babel-comint-send-string
              session
              org-babel-blub-remove-prompt-command))

         ;; Needed for Emacs 23 since the marker is initially
         ;; undefined and the filter functions try to use it without
         ;; checking.
         (with-current-buffer session
           (set-marker comint-last-output-start (point)))

         ;; return shell buffer
         (get-buffer session))
        (t session)))

(defun org-babel-execute:blub (body params)
  "Execute BODY of Blub code with org-babel."
  (let* ((session (cdr (assq :session params))))
    (cond ((and session
                (not (string= session "none"))
                (org-babel-blub-get-session session))
           (if (not my-org-babel-comint--async-uuid)
               (my-org-babel-comint-send-to-session-async session body)
             ;; TODO prompt user to just go ahead with it (clobbering
             ;; the "existing" process)
             (error "Block already running.  Call `(setq my-org-babel-comint--async-uuid nil)' to run a new process")))
          (t (my-org-babel-eval-async org-babel-blub-interpreter (org-trim body))))))

(provide 'ob-blub)
;;; ob-blub.el ends here