emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: Nick Savage <nick@nicksavage.ca>
To: Ihor Radchenko <yantar92@gmail.com>
Cc: "Emanuel Berg via General discussions about Org-mode."
Subject: Re: babel output seems to drop anything before % (in session)
Date: Thu, 6 May 2021 19:44:32 -0400	[thread overview]
Message-ID: <5780a7fb-17b5-6bde-f3a1-ef741af57734@nicksavage.ca> (raw)
In-Reply-To: <87wnsc2gp2.fsf@localhost>

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

So I have a patch written that doesn't completely fix the problem, but 
maybe makes enough progress that someone else can figure it out.

The issue is that comint-prompt-regexp is reading the "% " as a prompt, 
and taking everyone off before it. I've added another parameter to 
org-babel-comint-with-output in the "meta" to pass our own regexp to 
replace comint-prompt-regexp, which works except when it doesn't. The 
regexp I've added is just "\n" now, so the newline characters are removed.

The tests that Daniele added as a patch don't quite pass though with 
this. The issue is that something in the way the output is posted in the 
output buffer includes the prompt occasionally is included in a line and 
occasionally not. It seems the first time the block of code is executed, 
it is included (and therefore needs to be removed) and each other time 
it is not, so it is only the first time that it is run is not working 

This is obviously way too fragile to actually merge, but I was hoping 
the work I've done so far is enough to help someone else make progress. 
I'll probably take another stab at this tomorrow (since it's bugging 
me), but thought I'd share what I have for now.


On 5/6/21 8:24 AM, Ihor Radchenko wrote:
> "Nicholas Savage" <nick@nicksavage.ca> writes:
>> I can confirm this too on the latest master.
>> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point.
> I confirm as well. I also saw an anomaly in the comint buffer. Note that
> all the output lines, except "five 0% six" are after the shell prompt.
> As I remember, the code expects the result to be exactly at the prompt
> line. So, for some reason the first command ("echo five 0% six") of the
> second block does not get inserted at the empty line.
> echo one 0% two
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two
> echo tree 0% four
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six
> five 0% six
> echo seven 0% eight
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
> Best,
> Ihor

[-- Attachment #2: 0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.patch --]
[-- Type: text/x-patch, Size: 3692 bytes --]

From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001
From: Nicholas Savage <nick@nicksavage.ca>
Date: Thu, 6 May 2021 19:17:33 -0400
Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly
 truncated on special characters.

* lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta
optional argument for providing a custom prompt regexp.
lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new
argument to prevent incorrect truncation on special characters.

If shell output included special characters that also occasionally are
included in shell prompts, such as "#" or "%", a regexp was tripping
up on them and cutting out part of the line. As ob-shell already cuts
out the shell prompts, this was not necessary and instead we can just
use \n as the separator. Original functionality was retained for other
ob-* files in case this was necessary.
 lisp/ob-comint.el | 20 ++++++++++++--------
 lisp/ob-shell.el  |  2 +-
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 18d4f3c93..27ad6efd7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and
 (defmacro org-babel-comint-with-output (meta &rest body)
   "Evaluate BODY in BUFFER and return process output.
-Will wait until EOE-INDICATOR appears in the output, then return
-all process output.  If REMOVE-ECHO and FULL-BODY are present and
-non-nil, then strip echo'd body from the returned output.  META
-should be a list containing the following where the last two
-elements are optional.
+Will wait until EOE-INDICATOR appears in the output, then return all
+process output.  If REMOVE-ECHO and FULL-BODY are present and non-nil,
+then strip echo'd body from the returned output.  PROMPT-REGEXP is a
+filter that, if provided, overrides the default regexp that tries to
+filter out the shell prompt.  META should be a list containing the
+following where the last three elements are optional.
 This macro ensures that the filter is removed in case of an error
 or user `keyboard-quit' during execution of body."
@@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body."
   (let ((buffer (nth 0 meta))
 	(eoe-indicator (nth 1 meta))
 	(remove-echo (nth 2 meta))
-	(full-body (nth 3 meta)))
+	(full-body (nth 3 meta))
+        (prompt-regexp (if (nth 4 meta)
+                           (nth 4 meta)
+                         comint-prompt-regexp)))
     `(org-babel-comint-in-buffer ,buffer
        (let* ((string-buffer "")
@@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body."
 		      "\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
 	   (setq string-buffer (substring string-buffer (match-end 0))))
-	 (split-string string-buffer comint-prompt-regexp)))))
+	 (split-string string-buffer ,prompt-regexp)))))
 (def-edebug-spec org-babel-comint-with-output (sexp body))
 (defun org-babel-comint-input-command (buffer cmd)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 3eed0c164..9ec0425cb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,7 +263,7 @@ return the value of the last statement in BODY."
-		   (session org-babel-sh-eoe-output t body)
+		   (session org-babel-sh-eoe-output t body "\n")
 		 (dolist (line (append (split-string (org-trim body) "\n")
 				       (list org-babel-sh-eoe-indicator)))
 		   (insert line)

  reply	other threads:[~2021-05-06 23:45 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-05 15:15 babel output seems to drop anything before % (in session) Daniele Pizzolli
2021-05-06  2:35 ` John Corless
2021-05-06 11:39   ` Nicholas Savage
2021-05-06 12:24     ` Ihor Radchenko
2021-05-06 23:44       ` Nick Savage [this message]
2021-05-15 20:47         ` Bastien
2021-06-09  3:41           ` Kaushal Modi
2021-06-09  3:54         ` Kaushal Modi
2021-06-09  4:01           ` Kaushal Modi
2021-11-09 21:21             ` Kaushal Modi
2021-11-10  4:13               ` Ihor Radchenko
2022-06-07  0:01                 ` Felix Freeman via General discussions about Org-mode.
2022-06-07 21:50                   ` Felix Freeman via General discussions about Org-mode.
2022-06-07 22:55                     ` Felix Freeman via General discussions about Org-mode.
2022-06-18  3:20                   ` [PATCH] " Ihor Radchenko
2022-08-26 12:51                     ` Ihor Radchenko
2022-09-25  1:44                       ` Ihor Radchenko
2022-09-25  2:02                         ` Bastien Guerry
2022-07-16 10:04 ` Ihor Radchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5780a7fb-17b5-6bde-f3a1-ef741af57734@nicksavage.ca \
    --to=nick@nicksavage.ca \
    --cc=emacs-orgmode@gnu.org \
    --cc=yantar92@gmail.com \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).