From: Matt <matt@excalamus.com>
To: "Ihor Radchenko" <yantar92@posteo.net>
Cc: "emacs-orgmode" <emacs-orgmode@gnu.org>
Subject: Re: [PATCH] Async evaluation in ob-shell
Date: Tue, 21 Mar 2023 16:29:20 -0400 [thread overview]
Message-ID: <18705dca9b8.f4f7e70c165671.6021704484862511260@excalamus.com> (raw)
In-Reply-To: <873562z66c.fsf@localhost>
[-- Attachment #1: Type: text/plain, Size: 5076 bytes --]
> Matt matt@excalamus.com> writes:
>
> I see only two options to fix it: remove a space from the concat expression (which I did in my latest patch) or remove a space from `org-babel-sh-prompt'.
Unfortunately, I was mistaken and the second option (removing the space from `org-babel-sh-prompt') doesn't fix the issue. The TLDR is that the code in `org-babel-comint-async-filter' which grabs the region between the indicators (incorrectly) fails to include the prompt's trailing space.
#+begin_longwinded_explanation
I'll first explain why removing the space from `org-babel-sh-prompt' doesn't fix the issue because it well also highlight the underlying problem.
If we remove the space from the `org-babel-sh-prompt', then `comint-prompt-regexp' becomes "^org_babel_sh_prompt> *" (with one space). This would work if the string passed to the `ob-shell-async-chunk-callback' stayed the same. It doesn't (this is where my reasoning and testing failed). Changing the `org-babel-sh-prompt' to "org_babel_sh_prompt>" (without a space) causes the following string to be passed to the callback:
"org_babel_sh_prompt>1
org_babel_sh_prompt>2
org_babel_sh_prompt"
Note that the final prompt doesn't have a ">" and therefore the `comint-prompt-regexp' (which becomes "^org_babel_sh_prompt> * (with one space)) used in the callback fails to match it. When we remove the space from the `org-babel-sh-prompt', the session buffer looks like this:
"sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt>";PS2=
org_babel_sh_prompt>echo 'ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8'
echo 1
echo 2
echo 'ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8'
ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8
org_babel_sh_prompt>1
org_babel_sh_prompt>2
org_babel_sh_prompt>ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8
org_babel_sh_prompt>"
The `org-babel-comint-async-filter' is what calls the `ob-shell-async-chunk-callback' (ob-comint.el:284). It monitors for the end indicator. When that appears, it passes the region between the beginning of the end indicator **less 1** and the character after the end of the start indicator to the callback. For a clean run of `test-ob-shell/session-async-evaluation', the beginning of the end indicator is at 361 and the character after the end of the start indicator is at 298. This is the string I gave above which is missing the ">".
In order to make the second option work, we'd need to change the "less 1" part of `org-babel-comint-async-filter' from (- (match-beginning 0) 1) to (match-beginning 0). It turns out that's actually all we need to do.
When `org-babel-sh-prompt' is "org_babel_sh_prompt> " (with one space), then the session buffer looks like:
"sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> echo 'ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26'
echo 1
echo 2
echo 'ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26'
ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26
org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt> ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26
org_babel_sh_prompt> "
The region passed to the callback is then defined as 366 to 300, or
"org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt>" (<-- no space)
This looks okay at first glance. However, **the last line is not a valid prompt**. A prompt must end in a space! When the `org-babel-sh-prompt' is set to "org_babel_sh_prompt> " (with one space), the `comint-prompt-regexp' is "^org_babel_sh_prompt> *" (with two spaces). This means that the `comint-prompt-regexp' matches on a trailing space which the **region passed to the callback doesn't have**. Therefore, the match fails.
Instead, if we modify the `org-babel-comint-async-filter' like
modified lisp/ob-comint.el
@@ -273,7 +273,7 @@ STRING contains the output originally inserted into the comint buffer."
(res-str-raw
(buffer-substring
;; move point to beginning of indicator
- (- (match-beginning 0) 1)
+ (match-beginning 0)
;; find the matching start indicator
(cl-loop
do (re-search-backward indicator)
then the region passed to the callback will be from 367 to 300, or
"org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt> " (<-- with one space)
The `comint-prompt-regexp' will now match the last prompt in the region.
With this change, the `org-babel-sh-prompt' keeps the trailing space (like it should), the `comint-prompt-regexp' becomes "^org_babel_sh_prompt> *" (with two spaces, requiring a prompt to have a trailing space like it should), the `ob-shell-async-chunk-callback' can use `comint-prompt-regexp' without modification, and the tests all pass.
#+end_longwinded_explanation
I've attached an updated diff. If everyone is satisfied with this, I'll do a proper commit and then handle moving the uuid code like we talked about earlier in the thread.
[-- Attachment #2: 0004-ob-shell-Add-async-evaluation.diff --]
[-- Type: application/octet-stream, Size: 6818 bytes --]
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 54bf5127e..86c2bf7a7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -273,7 +273,7 @@ STRING contains the output originally inserted into the comint buffer."
(res-str-raw
(buffer-substring
;; move point to beginning of indicator
- (- (match-beginning 0) 1)
+ (match-beginning 0)
;; find the matching start indicator
(cl-loop
do (re-search-backward indicator)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 9e7b45a89..eab8ea935 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -269,12 +269,22 @@ var of the same value."
(set-marker comint-last-output-start (point))
(get-buffer (current-buffer)))))))
+(defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'"
+ "Session output delimiter template.
+See `org-babel-comint-async-indicator'.")
+
+(defun ob-shell-async-chunk-callback (string)
+ "Filter applied to results before insertion.
+See `org-babel-comint-async-chunk-callback'."
+ (replace-regexp-in-string comint-prompt-regexp "" string))
+
(defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
"Pass BODY to the Shell process in BUFFER.
If RESULT-TYPE equals `output' then return a list of the outputs
of the statements in BODY, if RESULT-TYPE equals `value' then
return the value of the last statement in BODY."
(let* ((shebang (cdr (assq :shebang params)))
+ (async (org-babel-comint-use-async params))
(results-params (cdr (assq :result-params params)))
(value-is-exit-status
(or (and
@@ -306,19 +316,38 @@ return the value of the last statement in BODY."
(concat (file-local-name script-file) " " cmdline)))))
(buffer-string))))
(session ; session evaluation
- (mapconcat
- #'org-babel-sh-strip-weird-long-prompt
- (mapcar
- #'org-trim
- (butlast ; Remove eoe indicator
- (org-babel-comint-with-output
- (session org-babel-sh-eoe-output t body)
- (insert (org-trim body) "\n"
- org-babel-sh-eoe-indicator)
- (comint-send-input nil t))
- ;; Remove `org-babel-sh-eoe-indicator' output line.
- 1))
- "\n"))
+ (if async
+ (progn
+ (let ((uuid (org-id-uuid)))
+ (org-babel-comint-async-register
+ session
+ (current-buffer)
+ "ob_comint_async_shell_\\(.+?\\)_\\(.+\\)"
+ ;; "ob_comint_async_shell_\\(.+\\)_\\(.+\\)"
+ 'ob-shell-async-chunk-callback
+ nil)
+ (org-babel-comint-async-delete-dangling-and-eval
+ session
+ (insert (format ob-shell-async-indicator "start" uuid))
+ (comint-send-input nil t)
+ (insert (org-trim body))
+ (comint-send-input nil t)
+ (insert (format ob-shell-async-indicator "end" uuid))
+ (comint-send-input nil t))
+ uuid))
+ (mapconcat
+ #'org-babel-sh-strip-weird-long-prompt
+ (mapcar
+ #'org-trim
+ (butlast ; Remove eoe indicator
+ (org-babel-comint-with-output
+ (session org-babel-sh-eoe-output t body)
+ (insert (org-trim body) "\n"
+ org-babel-sh-eoe-indicator)
+ (comint-send-input nil t))
+ ;; Remove `org-babel-sh-eoe-indicator' output line.
+ 1))
+ "\n")))
;; External shell script, with or without a predefined
;; shebang.
((org-string-nw-p shebang)
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 8366f9dbe..c56a76acf 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -33,6 +33,9 @@
(org-test-for-executable "sh")
+(defconst test-ob-shell/uuid-regex
+ "[0-9a-fA-F]\\{8\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{12\\}")
+
\f
;;; Code:
(ert-deftest test-ob-shell/dont-insert-spaces-on-expanded-bodies ()
@@ -75,6 +78,59 @@ the body of the tangled block does."
(if (should (equal '((1) (2)) result))
(kill-buffer session-name))))
+(ert-deftest test-ob-shell/session-async-valid-header-arg-values ()
+ "Test that session runs asynchronously for certain :async values."
+ (let ((session-name "test-ob-shell/session-async-valid-header-arg-values")
+ (kill-buffer-query-functions nil))
+ (dolist (arg-val '("t" ""))
+ (org-test-with-temp-text
+ (concat "#+begin_src sh :session " session-name " :async " arg-val "
+echo 1<point>
+#+end_src")
+ (if (should
+ (string-match
+ test-ob-shell/uuid-regex
+ (org-trim (org-babel-execute-src-block))))
+ (kill-buffer session-name))))))
+
+(ert-deftest test-ob-shell/session-async-inserts-uuid-before-results-are-returned ()
+ "Test that a uuid placeholder is inserted before results are inserted."
+ (let ((session-name "test-ob-shell/session-async-inserts-uuid-before-results-are-returned")
+ (kill-buffer-query-functions nil))
+ (org-test-with-temp-text
+ (concat "#+begin_src sh :session " session-name " :async t
+echo 1<point>
+#+end_src")
+ (if (should
+ (string-match
+ test-ob-shell/uuid-regex
+ (org-trim (org-babel-execute-src-block))))
+ (kill-buffer session-name)))))
+
+(ert-deftest test-ob-shell/session-async-evaluation ()
+ "Test the async evaluation process."
+ (let* ((session-name "test-ob-shell/session-async-evaluation")
+ (kill-buffer-query-functions nil)
+ (start-time (current-time))
+ (wait-time (time-add start-time 3))
+ uuid-placeholder)
+ (org-test-with-temp-text
+ (concat "#+begin_src sh :session " session-name " :async t
+echo 1
+echo 2<point>
+#+end_src")
+ (setq uuid-placeholder (org-trim (org-babel-execute-src-block)))
+ (catch 'too-long
+ (while (string-match uuid-placeholder (buffer-string))
+ (progn
+ (sleep-for 0.01)
+ (when (time-less-p wait-time (current-time))
+ (throw 'too-long (ert-fail "Took too long to get result from callback"))))))
+ (search-forward "#+results")
+ (beginning-of-line 2)
+ (if (should (string= ": 1\n: 2\n" (buffer-substring-no-properties (point) (point-max))))
+ (kill-buffer session-name)))))
+
(ert-deftest test-ob-shell/generic-uses-no-arrays ()
"Test generic serialization of array into a single string."
(org-test-with-temp-text
next prev parent reply other threads:[~2023-03-21 20:30 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-06 19:39 [PATCH] Async evaluation in ob-shell Matt
2023-02-07 11:40 ` Ihor Radchenko
2023-02-09 4:33 ` Matt
2023-02-09 11:24 ` Ihor Radchenko
2023-02-10 22:19 ` Matt
2023-02-11 11:44 ` Ihor Radchenko
2023-02-12 19:32 ` Matt
2023-02-15 15:08 ` Ihor Radchenko
2023-02-16 4:02 ` Matt
2023-02-17 10:44 ` Ihor Radchenko
2023-02-19 23:14 ` Matt
2023-02-20 11:24 ` Ihor Radchenko
2023-02-20 17:24 ` Matt
2023-02-22 10:30 ` Ihor Radchenko
2023-03-02 1:36 ` Matt
2023-03-03 14:52 ` Ihor Radchenko
2023-03-03 17:53 ` Matt
2023-03-05 12:15 ` Ihor Radchenko
2023-03-06 6:45 ` Matt
2023-03-07 12:45 ` Ihor Radchenko
2023-03-09 17:36 ` Matt
2023-03-10 1:52 ` Max Nikulin
2023-03-12 16:28 ` Jack Kamm
2023-03-18 10:48 ` Ihor Radchenko
2023-03-21 20:29 ` Matt [this message]
2023-03-22 12:12 ` Ihor Radchenko
2023-03-23 11:50 ` Ihor Radchenko
2023-03-23 19:35 ` Matt
2023-03-24 9:13 ` Ihor Radchenko
2023-03-28 2:53 ` Matt
2023-03-28 10:06 ` Ihor Radchenko
2023-04-17 15:31 ` Matt
2023-04-17 18:55 ` Ihor Radchenko
2023-04-17 18:56 ` Matt
2023-04-17 19:05 ` Ihor Radchenko
2023-03-23 3:25 ` [SUGGESTION] ob-shell async result output should not contains shell prompt Christopher M. Miles
2023-03-23 4:21 ` Matt
2023-03-23 11:12 ` Christopher M. Miles
2023-03-23 16:23 ` Matt
2023-03-24 11:20 ` Ihor Radchenko
2023-03-23 16:26 ` Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt) Matt
2023-03-24 1:53 ` Remove "shell" as a supported Babel language within ob-shell.el Christopher M. Miles
2023-03-24 11:38 ` Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt) Ihor Radchenko
2023-03-25 5:47 ` Samuel Wales
2023-03-25 18:07 ` Ihor Radchenko
2023-03-28 2:33 ` Matt
2023-02-11 20:56 ` [PATCH] Async evaluation in ob-shell jackkamm
2023-02-12 19:02 ` Matt
2023-02-13 3:16 ` Jack Kamm
2023-02-13 17:07 ` [BUG] shell sessions started outside of Babel broken Matt
2023-02-15 6:19 ` Jack Kamm
2023-02-16 12:53 ` Ihor Radchenko
2023-02-19 15:04 ` Jack Kamm
2023-02-20 11:22 ` Ihor Radchenko
2023-02-21 5:23 ` Jack Kamm
2023-02-22 10:38 ` Ihor Radchenko
2023-03-25 16:55 ` Jack Kamm
2023-03-25 16:59 ` [PATCH] Fix externally started sessions with ob-python Jack Kamm
2023-02-13 20:11 ` [BUG] conda doesn't work in ob-shell sessions Matt
2023-02-15 6:21 ` Jack Kamm
2024-01-18 11:55 ` Ihor Radchenko
2024-01-21 22:48 ` Jack Kamm
2024-01-22 3:42 ` Jack Kamm
2024-01-22 11:59 ` Ihor Radchenko
2024-01-23 6:09 ` Jack Kamm
2024-01-24 15:22 ` Ihor Radchenko
2024-01-25 19:14 ` Matt
2024-01-25 20:36 ` Ihor Radchenko
2024-01-26 0:42 ` Jack Kamm
2024-01-27 10:25 ` Matt
2024-02-09 16:37 ` Ihor Radchenko
2024-01-23 18:51 ` Suhail Singh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=18705dca9b8.f4f7e70c165671.6021704484862511260@excalamus.com \
--to=matt@excalamus.com \
--cc=emacs-orgmode@gnu.org \
--cc=yantar92@posteo.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).