emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: Matt <matt@excalamus.com>
To: "Ihor Radchenko" <yantar92@posteo.net>
Cc: "emacs-orgmode" <emacs-orgmode@gnu.org>
Subject: Re: [PATCH] Async evaluation in ob-shell
Date: Fri, 03 Mar 2023 12:53:18 -0500	[thread overview]
Message-ID: <186a89b5a24.cc4913fe1885630.21323979015166045@excalamus.com> (raw)
In-Reply-To: <87bkl96g6e.fsf@localhost>

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]


 ---- On Fri, 03 Mar 2023 09:52:09 -0500  Ihor Radchenko  wrote --- 
 > I tried the patch, and I am getting failed tests:
 > 
 > 1 unexpected results:
 >    FAILED  test-ob-shell/session-async-evaluation  ((should (string= ": 1\n: 2\n" (buffer-substring-no-properties (point) (point-max)))) :form (string= ": 1\n: 2\n" ": 1\n: 2\n: org_babel_sh_prompt>\n") :value nil :explanation (arrays-of-different-length 8 31 ": 1\n: 2\n" ": 1\n: 2\n: org_babel_sh_prompt>\n" first-mismatch-at 8))

Sorry for missing that.  The issue is that when I replaced `org-babel-sh-prompt' with `comint-prompt-regexp', the regexp no longer matches the output and grabs the next prompt.  It looks like the reason is `comint-prompt-regexp' is set to "^org_babel_sh_prompt>  *" (with two spaces between the '>' and '*').  Attached is a revised patch which removes one of the spaces by changing how `org-babel-sh-initiate-session' sets the `comint-prompt-regexp'.  Another place this could be done is in the defvar for `org-babel-sh-prompt' itself (which ends with a space).  However, I think it's customary to leave a trailing space for prompts?

 > > +                  (let ((uuid (org-id-uuid)))
 > 
 > Do you need to use `org-id-uuid' here? ob-python directly uses `md5'.
 > If you still prefer org-id-uuid, we probably need to move it to
 > org-macs.el

I just need a random string.  `md5' would work for that.  However, might it be better to update ob-R and ob-python to use `org-id-uuid'?  Both of those manually declare the randomness.  It's conceivable that someone may delete or mistype the number (100000000), resulting in a lower entropy.  An md5 is also not a uuid, strictly speaking.  Of course, the chance of collision is still basically zero and the cost of collision about the same.  Using `org-id-uuid' would only provide a consistent way to do things.

[-- Attachment #2: 0003-ob-shell-Add-async-evaluation.patch --]
[-- Type: application/octet-stream, Size: 7720 bytes --]

From 941bbadb922051f404943c58c8f0c6482623f876 Mon Sep 17 00:00:00 2001
From: Matthew Trzcinski <matt@excalamus.com>
Date: Fri, 3 Mar 2023 12:40:15 -0500
Subject: [PATCH] ob-shell: Add async evaluation

* ob-shell.el (org-babel-sh-initiate-session): Remove extra space from
`comint-prompt-regexp'.
(org-babel-sh-evaluate): Add condition for async within
session.  Allow :async header argument to be either t or blank.

* test-ob-shell.el: Add const regexp for uuids.
(test-ob-shell/session-async-valid-header-arg-values): Check that
:async header works for both t and blank values.
(test-ob-shell/session-async-inserts-uuid-before-results-are-returned):
(test-ob-shell/session-async-evaluation): Check that asynchronously
evaluated results are eventually placed in the buffer.

Link: https://list.orgmode.org/186283d230a.129f5feb61660123.3289004102603503414@excalamus.com/
---
 lisp/ob-shell.el              | 56 ++++++++++++++++++++++++++---------
 testing/lisp/test-ob-shell.el | 56 +++++++++++++++++++++++++++++++++++
 2 files changed, 98 insertions(+), 14 deletions(-)

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 9e7b45a89..db5d2aabc 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -262,19 +262,29 @@ var of the same value."
              (format org-babel-prompt-command org-babel-sh-prompt))
             (setq-local comint-prompt-regexp
                         (concat "^" (regexp-quote org-babel-sh-prompt)
-                                " *"))
+                                "*"))
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
 	    (set-marker comint-last-output-start (point))
 	    (get-buffer (current-buffer)))))))
 
+(defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'"
+  "Session output delimiter template.
+See `org-babel-comint-async-indicator'.")
+
+(defun ob-shell-async-chunk-callback (string)
+  "Filter applied to results before insertion.
+See `org-babel-comint-async-chunk-callback'."
+  (replace-regexp-in-string comint-prompt-regexp "" string))
+
 (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
 If RESULT-TYPE equals `output' then return a list of the outputs
 of the statements in BODY, if RESULT-TYPE equals `value' then
 return the value of the last statement in BODY."
   (let* ((shebang (cdr (assq :shebang params)))
+         (async (org-babel-comint-use-async params))
 	 (results-params (cdr (assq :result-params params)))
 	 (value-is-exit-status
 	  (or (and
@@ -306,19 +316,37 @@ return the value of the last statement in BODY."
                                 (concat (file-local-name script-file)  " " cmdline)))))
 		(buffer-string))))
 	   (session			; session evaluation
-	    (mapconcat
-	     #'org-babel-sh-strip-weird-long-prompt
-	     (mapcar
-	      #'org-trim
-	      (butlast ; Remove eoe indicator
-	       (org-babel-comint-with-output
-		   (session org-babel-sh-eoe-output t body)
-                 (insert (org-trim body) "\n"
-                         org-babel-sh-eoe-indicator)
-		 (comint-send-input nil t))
-               ;; Remove `org-babel-sh-eoe-indicator' output line.
-	       1))
-	     "\n"))
+            (if async
+                (progn
+                  (let ((uuid (org-id-uuid)))
+                    (org-babel-comint-async-register
+                     session
+                     (current-buffer)
+                     "ob_comint_async_shell_\\(.+?\\)_\\(.+\\)"
+                     'ob-shell-async-chunk-callback
+                     nil)
+                    (org-babel-comint-async-delete-dangling-and-eval
+                        session
+                      (insert (format ob-shell-async-indicator "start" uuid))
+                      (comint-send-input nil t)
+                      (insert (org-trim body))
+                      (comint-send-input nil t)
+                      (insert (format ob-shell-async-indicator "end" uuid))
+                      (comint-send-input nil t))
+                    uuid))
+	      (mapconcat
+	       #'org-babel-sh-strip-weird-long-prompt
+	       (mapcar
+	        #'org-trim
+	        (butlast ; Remove eoe indicator
+	         (org-babel-comint-with-output
+		     (session org-babel-sh-eoe-output t body)
+                   (insert (org-trim body) "\n"
+                           org-babel-sh-eoe-indicator)
+		   (comint-send-input nil t))
+                 ;; Remove `org-babel-sh-eoe-indicator' output line.
+	         1))
+	       "\n")))
 	   ;; External shell script, with or without a predefined
 	   ;; shebang.
 	   ((org-string-nw-p shebang)
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 8366f9dbe..c56a76acf 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -33,6 +33,9 @@
 
 (org-test-for-executable "sh")
 
+(defconst test-ob-shell/uuid-regex
+  "[0-9a-fA-F]\\{8\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{12\\}")
+
 \f
 ;;; Code:
 (ert-deftest test-ob-shell/dont-insert-spaces-on-expanded-bodies ()
@@ -75,6 +78,59 @@ the body of the tangled block does."
     (if (should (equal '((1) (2)) result))
         (kill-buffer session-name))))
 
+(ert-deftest test-ob-shell/session-async-valid-header-arg-values ()
+  "Test that session runs asynchronously for certain :async values."
+  (let ((session-name "test-ob-shell/session-async-valid-header-arg-values")
+        (kill-buffer-query-functions nil))
+    (dolist (arg-val '("t" ""))
+     (org-test-with-temp-text
+         (concat "#+begin_src sh :session " session-name " :async " arg-val "
+echo 1<point>
+#+end_src")
+       (if (should
+            (string-match
+             test-ob-shell/uuid-regex
+             (org-trim (org-babel-execute-src-block))))
+           (kill-buffer session-name))))))
+
+(ert-deftest test-ob-shell/session-async-inserts-uuid-before-results-are-returned ()
+  "Test that a uuid placeholder is inserted before results are inserted."
+  (let ((session-name "test-ob-shell/session-async-inserts-uuid-before-results-are-returned")
+        (kill-buffer-query-functions nil))
+    (org-test-with-temp-text
+        (concat "#+begin_src sh :session " session-name " :async t
+echo 1<point>
+#+end_src")
+      (if (should
+           (string-match
+            test-ob-shell/uuid-regex
+            (org-trim (org-babel-execute-src-block))))
+          (kill-buffer session-name)))))
+
+(ert-deftest test-ob-shell/session-async-evaluation ()
+  "Test the async evaluation process."
+  (let* ((session-name "test-ob-shell/session-async-evaluation")
+         (kill-buffer-query-functions nil)
+         (start-time (current-time))
+         (wait-time (time-add start-time 3))
+         uuid-placeholder)
+    (org-test-with-temp-text
+        (concat "#+begin_src sh :session " session-name " :async t
+echo 1
+echo 2<point>
+#+end_src")
+      (setq uuid-placeholder (org-trim (org-babel-execute-src-block)))
+      (catch 'too-long
+        (while (string-match uuid-placeholder (buffer-string))
+          (progn
+            (sleep-for 0.01)
+            (when (time-less-p wait-time (current-time))
+              (throw 'too-long (ert-fail "Took too long to get result from callback"))))))
+    (search-forward "#+results")
+    (beginning-of-line 2)
+    (if (should (string= ": 1\n: 2\n" (buffer-substring-no-properties (point) (point-max))))
+          (kill-buffer session-name)))))
+
 (ert-deftest test-ob-shell/generic-uses-no-arrays ()
   "Test generic serialization of array into a single string."
   (org-test-with-temp-text
-- 
2.39.1


  reply	other threads:[~2023-03-03 17:54 UTC|newest]

Thread overview: 72+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-02-06 19:39 [PATCH] Async evaluation in ob-shell Matt
2023-02-07 11:40 ` Ihor Radchenko
2023-02-09  4:33   ` Matt
2023-02-09 11:24     ` Ihor Radchenko
2023-02-10 22:19       ` Matt
2023-02-11 11:44         ` Ihor Radchenko
2023-02-12 19:32           ` Matt
2023-02-15 15:08             ` Ihor Radchenko
2023-02-16  4:02               ` Matt
2023-02-17 10:44                 ` Ihor Radchenko
2023-02-19 23:14                   ` Matt
2023-02-20 11:24                     ` Ihor Radchenko
2023-02-20 17:24                       ` Matt
2023-02-22 10:30                         ` Ihor Radchenko
2023-03-02  1:36                           ` Matt
2023-03-03 14:52                             ` Ihor Radchenko
2023-03-03 17:53                               ` Matt [this message]
2023-03-05 12:15                                 ` Ihor Radchenko
2023-03-06  6:45                                   ` Matt
2023-03-07 12:45                                     ` Ihor Radchenko
2023-03-09 17:36                                       ` Matt
2023-03-10  1:52                                         ` Max Nikulin
2023-03-12 16:28                                         ` Jack Kamm
2023-03-18 10:48                                         ` Ihor Radchenko
2023-03-21 20:29                                           ` Matt
2023-03-22 12:12                                             ` Ihor Radchenko
2023-03-23 11:50                                             ` Ihor Radchenko
2023-03-23 19:35                                               ` Matt
2023-03-24  9:13                                                 ` Ihor Radchenko
2023-03-28  2:53                                                   ` Matt
2023-03-28 10:06                                                     ` Ihor Radchenko
2023-04-17 15:31                                                   ` Matt
2023-04-17 18:55                                                     ` Ihor Radchenko
2023-04-17 18:56                                                       ` Matt
2023-04-17 19:05                                                         ` Ihor Radchenko
2023-03-23  3:25       ` [SUGGESTION] ob-shell async result output should not contains shell prompt Christopher M. Miles
2023-03-23  4:21         ` Matt
2023-03-23 11:12           ` Christopher M. Miles
2023-03-23 16:23             ` Matt
2023-03-24 11:20               ` Ihor Radchenko
2023-03-23 16:26             ` Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt) Matt
2023-03-24  1:53               ` Remove "shell" as a supported Babel language within ob-shell.el Christopher M. Miles
2023-03-24 11:38               ` Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt) Ihor Radchenko
2023-03-25  5:47                 ` Samuel Wales
2023-03-25 18:07                   ` Ihor Radchenko
2023-03-28  2:33                 ` Matt
2023-02-11 20:56 ` [PATCH] Async evaluation in ob-shell jackkamm
2023-02-12 19:02   ` Matt
2023-02-13  3:16     ` Jack Kamm
2023-02-13 17:07       ` [BUG] shell sessions started outside of Babel broken Matt
2023-02-15  6:19         ` Jack Kamm
2023-02-16 12:53           ` Ihor Radchenko
2023-02-19 15:04             ` Jack Kamm
2023-02-20 11:22               ` Ihor Radchenko
2023-02-21  5:23                 ` Jack Kamm
2023-02-22 10:38                   ` Ihor Radchenko
2023-03-25 16:55               ` Jack Kamm
2023-03-25 16:59                 ` [PATCH] Fix externally started sessions with ob-python Jack Kamm
2023-02-13 20:11       ` [BUG] conda doesn't work in ob-shell sessions Matt
2023-02-15  6:21         ` Jack Kamm
2024-01-18 11:55           ` Ihor Radchenko
2024-01-21 22:48             ` Jack Kamm
2024-01-22  3:42               ` Jack Kamm
2024-01-22 11:59                 ` Ihor Radchenko
2024-01-23  6:09                   ` Jack Kamm
2024-01-24 15:22                     ` Ihor Radchenko
2024-01-25 19:14                       ` Matt
2024-01-25 20:36                         ` Ihor Radchenko
2024-01-26  0:42                       ` Jack Kamm
2024-01-27 10:25                         ` Matt
2024-02-09 16:37                           ` Ihor Radchenko
2024-01-23 18:51               ` Suhail Singh

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=186a89b5a24.cc4913fe1885630.21323979015166045@excalamus.com \
    --to=matt@excalamus.com \
    --cc=emacs-orgmode@gnu.org \
    --cc=yantar92@posteo.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).