Re: babel output seems to drop anything before % (in session)

[Nick Savage <nick@nicksavage.ca>]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

So I have a patch written that doesn't completely fix the problem, but 
maybe makes enough progress that someone else can figure it out.

The issue is that comint-prompt-regexp is reading the "% " as a prompt, 
and taking everyone off before it. I've added another parameter to 
org-babel-comint-with-output in the "meta" to pass our own regexp to 
replace comint-prompt-regexp, which works except when it doesn't. The 
regexp I've added is just "\n" now, so the newline characters are removed.

The tests that Daniele added as a patch don't quite pass though with 
this. The issue is that something in the way the output is posted in the 
output buffer includes the prompt occasionally is included in a line and 
occasionally not. It seems the first time the block of code is executed, 
it is included (and therefore needs to be removed) and each other time 
it is not, so it is only the first time that it is run is not working 
properly.

This is obviously way too fragile to actually merge, but I was hoping 
the work I've done so far is enough to help someone else make progress. 
I'll probably take another stab at this tomorrow (since it's bugging 
me), but thought I'd share what I have for now.

Cheers,
Nick

On 5/6/21 8:24 AM, Ihor Radchenko wrote:
> "Nicholas Savage" <nick@nicksavage.ca> writes:
>
>> I can confirm this too on the latest master.
>>
>> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point.
> I confirm as well. I also saw an anomaly in the comint buffer. Note that
> all the output lines, except "five 0% six" are after the shell prompt.
> As I remember, the code expects the result to be exactly at the prompt
> line. So, for some reason the first command ("echo five 0% six") of the
> second block does not get inserted at the empty line.
>
> echo one 0% two
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two
> echo tree 0% four
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six
> five 0% six
> echo seven 0% eight
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
>
> Best,
> Ihor

[-- Attachment #2: 0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.patch --]
[-- Type: text/x-patch, Size: 3692 bytes --]

From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001
From: Nicholas Savage <nick@nicksavage.ca>
Date: Thu, 6 May 2021 19:17:33 -0400
Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly
 truncated on special characters.

* lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta
optional argument for providing a custom prompt regexp.
lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new
argument to prevent incorrect truncation on special characters.

If shell output included special characters that also occasionally are
included in shell prompts, such as "#" or "%", a regexp was tripping
up on them and cutting out part of the line. As ob-shell already cuts
out the shell prompts, this was not necessary and instead we can just
use \n as the separator. Original functionality was retained for other
ob-* files in case this was necessary.
---
 lisp/ob-comint.el | 20 ++++++++++++--------
 lisp/ob-shell.el  |  2 +-
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 18d4f3c93..27ad6efd7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and
 
 (defmacro org-babel-comint-with-output (meta &rest body)
   "Evaluate BODY in BUFFER and return process output.
-Will wait until EOE-INDICATOR appears in the output, then return
-all process output.  If REMOVE-ECHO and FULL-BODY are present and
-non-nil, then strip echo'd body from the returned output.  META
-should be a list containing the following where the last two
-elements are optional.
+Will wait until EOE-INDICATOR appears in the output, then return all
+process output.  If REMOVE-ECHO and FULL-BODY are present and non-nil,
+then strip echo'd body from the returned output.  PROMPT-REGEXP is a
+filter that, if provided, overrides the default regexp that tries to
+filter out the shell prompt.  META should be a list containing the
+following where the last three elements are optional.
 
- (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY)
+ (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY PROMPT-REGEXP)
 
 This macro ensures that the filter is removed in case of an error
 or user `keyboard-quit' during execution of body."
@@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body."
   (let ((buffer (nth 0 meta))
 	(eoe-indicator (nth 1 meta))
 	(remove-echo (nth 2 meta))
-	(full-body (nth 3 meta)))
+	(full-body (nth 3 meta))
+        (prompt-regexp (if (nth 4 meta)
+                           (nth 4 meta)
+                         comint-prompt-regexp)))
     `(org-babel-comint-in-buffer ,buffer
        (let* ((string-buffer "")
 	      (comint-output-filter-functions
@@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body."
 		      "\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
 		     string-buffer))
 	   (setq string-buffer (substring string-buffer (match-end 0))))
-	 (split-string string-buffer comint-prompt-regexp)))))
+	 (split-string string-buffer ,prompt-regexp)))))
 (def-edebug-spec org-babel-comint-with-output (sexp body))
 
 (defun org-babel-comint-input-command (buffer cmd)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 3eed0c164..9ec0425cb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,7 +263,7 @@ return the value of the last statement in BODY."
 	      #'org-trim
 	      (butlast
 	       (org-babel-comint-with-output
-		   (session org-babel-sh-eoe-output t body)
+		   (session org-babel-sh-eoe-output t body "\n")
 		 (dolist (line (append (split-string (org-trim body) "\n")
 				       (list org-babel-sh-eoe-indicator)))
 		   (insert line)
-- 
2.20.1