* babel output seems to drop anything before % (in session) @ 2021-05-05 15:15 Daniele Pizzolli 2021-05-06 2:35 ` John Corless 2022-07-16 10:04 ` Ihor Radchenko 0 siblings, 2 replies; 19+ messages in thread From: Daniele Pizzolli @ 2021-05-05 15:15 UTC (permalink / raw) To: Emacs Org mode mailing list [-- Attachment #1: Type: text/plain, Size: 645 bytes --] Hello, Try to execute a few times the following and see the output corrupted in the line: #+BEGIN_EXAMPLE 1 packets transmitted, 1 received, 0% packet loss, time 0ms #+END_EXAMPLE #+PROPERTY: header-args:shell #+PROPERTY: header-args:shell+ :results output verbatim :wrap src text :session test #+NAME: ping #+BEGIN_SRC shell ping -c 1 127.0.0.1 #+END_SRC I tried to write a test (now with the :expected-result :failed), that hit the problem both in main and master (applies fine to both). I was not able to fix it, but hopefully the tests are still useful for somebody more knowledgeable! Thanks for you suggestion/fix! Best, Daniele [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --] [-- Type: text/x-diff, Size: 1944 bytes --] From 897ae50cf055d459d741c90f8de94dd56a8eb3d9 Mon Sep 17 00:00:00 2001 From: Daniele Pizzolli <dan+git@toel.it> Date: Wed, 5 May 2021 17:00:28 +0200 Subject: [PATCH 1/1] Add disabled test for % in babel shell output --- testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index 0aadb355d..a42cc3d77 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -106,6 +106,35 @@ ob-comint.el, which was not previously tested." "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC" (org-trim (org-babel-execute-src-block)))))) +(ert-deftest ob-shell/percent-simple () + "Test percent in output." + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + +(ert-deftest ob-shell/percent-session () + "Test percent in output for session. + +At the second iteration the string before % is dropped." + :expected-result :failed + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% six\nseven 0% eight" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + (provide 'test-ob-shell) ;;; test-ob-shell.el ends here -- 2.29.2 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-05 15:15 babel output seems to drop anything before % (in session) Daniele Pizzolli @ 2021-05-06 2:35 ` John Corless 2021-05-06 11:39 ` Nicholas Savage 2022-07-16 10:04 ` Ihor Radchenko 1 sibling, 1 reply; 19+ messages in thread From: John Corless @ 2021-05-06 2:35 UTC (permalink / raw) To: Daniele Pizzolli; +Cc: Emacs Org mode mailing list [-- Attachment #1: Type: text/plain, Size: 1264 bytes --] Confirmed Daniele, I was able to reproduce the behavior you described. Using the test case... #+BEGIN_SRC shell script :results output ping -c 1 127.0.0.1 #+END_SRC ... the results output matches what I get in a bash shell. But if you add the :session to the header args like this... #+BEGIN_SRC shell script :results output :session test ping -c 1 127.0.0.1 #+END_SRC ... the output line that includes "0% packet loss" gets truncated to after the "%" sign. I tested with 9.4.5. John On Wed, May 5, 2021 at 10:19 AM Daniele Pizzolli <dan@toel.it> wrote: > > Hello, > > Try to execute a few times the following and see the output corrupted in > the line: > > #+BEGIN_EXAMPLE > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > #+END_EXAMPLE > > #+PROPERTY: header-args:shell > #+PROPERTY: header-args:shell+ :results output verbatim :wrap src text > :session test > > #+NAME: ping > #+BEGIN_SRC shell > ping -c 1 127.0.0.1 > #+END_SRC > > I tried to write a test (now with the :expected-result :failed), that > hit the problem both in main and master (applies fine to both). > > I was not able to fix it, but hopefully the tests are still useful for > somebody more knowledgeable! > > Thanks for you suggestion/fix! > > Best, > Daniele > > [-- Attachment #2: Type: text/html, Size: 2875 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-06 2:35 ` John Corless @ 2021-05-06 11:39 ` Nicholas Savage 2021-05-06 12:24 ` Ihor Radchenko 0 siblings, 1 reply; 19+ messages in thread From: Nicholas Savage @ 2021-05-06 11:39 UTC (permalink / raw) To: Emanuel Berg via General discussions about Org-mode. [-- Attachment #1: Type: text/plain, Size: 1622 bytes --] I can confirm this too on the latest master. I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point. On Wed, May 5, 2021, at 22:35, John Corless wrote: > Confirmed > > Daniele, > > I was able to reproduce the behavior you described. Using the test case... > > #+BEGIN_SRC shell script :results output > ping -c 1 127.0.0.1 > #+END_SRC > > ... the results output matches what I get in a bash shell. But if you add the :session to the header args like this... > > #+BEGIN_SRC shell script :results output :session test > ping -c 1 127.0.0.1 > #+END_SRC > > ... the output line that includes "0% packet loss" gets truncated to after the "%" sign. > > I tested with 9.4.5. > > John > > > On Wed, May 5, 2021 at 10:19 AM Daniele Pizzolli <dan@toel.it> wrote: >> >> Hello, >> >> Try to execute a few times the following and see the output corrupted in >> the line: >> >> #+BEGIN_EXAMPLE >> 1 packets transmitted, 1 received, 0% packet loss, time 0ms >> #+END_EXAMPLE >> >> #+PROPERTY: header-args:shell >> #+PROPERTY: header-args:shell+ :results output verbatim :wrap src text :session test >> >> #+NAME: ping >> #+BEGIN_SRC shell >> ping -c 1 127.0.0.1 >> #+END_SRC >> >> I tried to write a test (now with the :expected-result :failed), that >> hit the problem both in main and master (applies fine to both). >> >> I was not able to fix it, but hopefully the tests are still useful for >> somebody more knowledgeable! >> >> Thanks for you suggestion/fix! >> >> Best, >> Daniele >> [-- Attachment #2: Type: text/html, Size: 3965 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-06 11:39 ` Nicholas Savage @ 2021-05-06 12:24 ` Ihor Radchenko 2021-05-06 23:44 ` Nick Savage 0 siblings, 1 reply; 19+ messages in thread From: Ihor Radchenko @ 2021-05-06 12:24 UTC (permalink / raw) To: Nicholas Savage; +Cc: Emanuel Berg via General discussions about Org-mode. "Nicholas Savage" <nick@nicksavage.ca> writes: > I can confirm this too on the latest master. > > I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point. I confirm as well. I also saw an anomaly in the comint buffer. Note that all the output lines, except "five 0% six" are after the shell prompt. As I remember, the code expects the result to be exactly at the prompt line. So, for some reason the first command ("echo five 0% six") of the second block does not get inserted at the empty line. echo one 0% two yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two echo tree 0% four yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four echo 'org_babel_sh_eoe' yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six five 0% six echo seven 0% eight yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight echo 'org_babel_sh_eoe' yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ Best, Ihor ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-06 12:24 ` Ihor Radchenko @ 2021-05-06 23:44 ` Nick Savage 2021-05-15 20:47 ` Bastien 2021-06-09 3:54 ` Kaushal Modi 0 siblings, 2 replies; 19+ messages in thread From: Nick Savage @ 2021-05-06 23:44 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Emanuel Berg via General discussions about Org-mode. [-- Attachment #1: Type: text/plain, Size: 2654 bytes --] So I have a patch written that doesn't completely fix the problem, but maybe makes enough progress that someone else can figure it out. The issue is that comint-prompt-regexp is reading the "% " as a prompt, and taking everyone off before it. I've added another parameter to org-babel-comint-with-output in the "meta" to pass our own regexp to replace comint-prompt-regexp, which works except when it doesn't. The regexp I've added is just "\n" now, so the newline characters are removed. The tests that Daniele added as a patch don't quite pass though with this. The issue is that something in the way the output is posted in the output buffer includes the prompt occasionally is included in a line and occasionally not. It seems the first time the block of code is executed, it is included (and therefore needs to be removed) and each other time it is not, so it is only the first time that it is run is not working properly. This is obviously way too fragile to actually merge, but I was hoping the work I've done so far is enough to help someone else make progress. I'll probably take another stab at this tomorrow (since it's bugging me), but thought I'd share what I have for now. Cheers, Nick On 5/6/21 8:24 AM, Ihor Radchenko wrote: > "Nicholas Savage" <nick@nicksavage.ca> writes: > >> I can confirm this too on the latest master. >> >> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point. > I confirm as well. I also saw an anomaly in the comint buffer. Note that > all the output lines, except "five 0% six" are after the shell prompt. > As I remember, the code expects the result to be exactly at the prompt > line. So, for some reason the first command ("echo five 0% six") of the > second block does not get inserted at the empty line. > > echo one 0% two > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two > echo tree 0% four > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four > echo 'org_babel_sh_eoe' > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six > five 0% six > echo seven 0% eight > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight > echo 'org_babel_sh_eoe' > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ > > Best, > Ihor [-- Attachment #2: 0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.patch --] [-- Type: text/x-patch, Size: 3692 bytes --] From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001 From: Nicholas Savage <nick@nicksavage.ca> Date: Thu, 6 May 2021 19:17:33 -0400 Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly truncated on special characters. * lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta optional argument for providing a custom prompt regexp. lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new argument to prevent incorrect truncation on special characters. If shell output included special characters that also occasionally are included in shell prompts, such as "#" or "%", a regexp was tripping up on them and cutting out part of the line. As ob-shell already cuts out the shell prompts, this was not necessary and instead we can just use \n as the separator. Original functionality was retained for other ob-* files in case this was necessary. --- lisp/ob-comint.el | 20 ++++++++++++-------- lisp/ob-shell.el | 2 +- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 18d4f3c93..27ad6efd7 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. -Will wait until EOE-INDICATOR appears in the output, then return -all process output. If REMOVE-ECHO and FULL-BODY are present and -non-nil, then strip echo'd body from the returned output. META -should be a list containing the following where the last two -elements are optional. +Will wait until EOE-INDICATOR appears in the output, then return all +process output. If REMOVE-ECHO and FULL-BODY are present and non-nil, +then strip echo'd body from the returned output. PROMPT-REGEXP is a +filter that, if provided, overrides the default regexp that tries to +filter out the shell prompt. META should be a list containing the +following where the last three elements are optional. - (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY) + (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY PROMPT-REGEXP) This macro ensures that the filter is removed in case of an error or user `keyboard-quit' during execution of body." @@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body." (let ((buffer (nth 0 meta)) (eoe-indicator (nth 1 meta)) (remove-echo (nth 2 meta)) - (full-body (nth 3 meta))) + (full-body (nth 3 meta)) + (prompt-regexp (if (nth 4 meta) + (nth 4 meta) + comint-prompt-regexp))) `(org-babel-comint-in-buffer ,buffer (let* ((string-buffer "") (comint-output-filter-functions @@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body." "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) string-buffer)) (setq string-buffer (substring string-buffer (match-end 0)))) - (split-string string-buffer comint-prompt-regexp))))) + (split-string string-buffer ,prompt-regexp))))) (def-edebug-spec org-babel-comint-with-output (sexp body)) (defun org-babel-comint-input-command (buffer cmd) diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index 3eed0c164..9ec0425cb 100644 --- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -263,7 +263,7 @@ return the value of the last statement in BODY." #'org-trim (butlast (org-babel-comint-with-output - (session org-babel-sh-eoe-output t body) + (session org-babel-sh-eoe-output t body "\n") (dolist (line (append (split-string (org-trim body) "\n") (list org-babel-sh-eoe-indicator))) (insert line) -- 2.20.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-06 23:44 ` Nick Savage @ 2021-05-15 20:47 ` Bastien 2021-06-09 3:41 ` Kaushal Modi 2021-06-09 3:54 ` Kaushal Modi 1 sibling, 1 reply; 19+ messages in thread From: Bastien @ 2021-05-15 20:47 UTC (permalink / raw) To: Nick Savage Cc: Emanuel Berg via General discussions about Org-mode., Ihor Radchenko Hi Nick, Nick Savage <nick@nicksavage.ca> writes: > So I have a patch written that doesn't completely fix the problem, but > maybe makes enough progress that someone else can figure it out. I hope you can continue to make progress on this (annoying) one. Let us know! -- Bastien ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-15 20:47 ` Bastien @ 2021-06-09 3:41 ` Kaushal Modi 0 siblings, 0 replies; 19+ messages in thread From: Kaushal Modi @ 2021-06-09 3:41 UTC (permalink / raw) To: Nick Savage; +Cc: Bastien, emacs-org list, Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 1023 bytes --] Today I ended up on possibly a related issue with ob-shell results. Observations: 1. Issue triggered when using :session 2. Output text is OK for the the first shell src block's output. But for the second block, the output before '>' gets truncated. Here's a minimal reproducible example: ===== #+property: header-args:shell :exports both :results output :session *ob-shell-session-issue* #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : 1 > 0 #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : 0 ===== The root cause seems to be similar, with detection of shell prompt: '>', '%', etc. when session is enabled. -- Kaushal Modi On Sat, May 15, 2021 at 4:48 PM Bastien <bzg@gnu.org> wrote: > Hi Nick, > > Nick Savage <nick@nicksavage.ca> writes: > > > So I have a patch written that doesn't completely fix the problem, but > > maybe makes enough progress that someone else can figure it out. > > I hope you can continue to make progress on this (annoying) one. > > Let us know! > > -- > Bastien > > [-- Attachment #2: Type: text/html, Size: 1859 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-06 23:44 ` Nick Savage 2021-05-15 20:47 ` Bastien @ 2021-06-09 3:54 ` Kaushal Modi 2021-06-09 4:01 ` Kaushal Modi 1 sibling, 1 reply; 19+ messages in thread From: Kaushal Modi @ 2021-06-09 3:54 UTC (permalink / raw) To: Nick Savage Cc: Emanuel Berg via General discussions about Org-mode., Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 1906 bytes --] On Thu, May 6, 2021 at 7:45 PM Nick Savage <nick@nicksavage.ca> wrote: > So I have a patch written that doesn't completely fix the problem, but > maybe makes enough progress that someone else can figure it out. > > The issue is that comint-prompt-regexp is reading the "% " as a prompt, > and taking everyone off before it. I've added another parameter to > org-babel-comint-with-output in the "meta" to pass our own regexp to > replace comint-prompt-regexp, which works except when it doesn't. The > regexp I've added is just "\n" now, so the newline characters are removed. > > The tests that Daniele added as a patch don't quite pass though with > this. The issue is that something in the way the output is posted in the > output buffer includes the prompt occasionally is included in a line and > occasionally not. It seems the first time the block of code is executed, > it is included (and therefore needs to be removed) and each other time > it is not, so it is only the first time that it is run is not working > properly. > > This is obviously way too fragile to actually merge, but I was hoping > the work I've done so far is enough to help someone else make progress. > I'll probably take another stab at this tomorrow (since it's bugging > me), but thought I'd share what I have for now. > > Cheers, > Nick > Thanks! I tried out your patch and it does fix the truncation of results output. Though, it retains the shell prompt in the first eval block and removes it only from the second eval block onwards. Here's a MWE: ===== #+property: header-args:shell :exports both :results output :session *ob-shell-session-issue* #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : sh-4.2$ 1 > 0 #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : 1 > 0 #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : 1 > 0 #+begin_src shell echo "1 > 0" #+end_src #+RESULTS: : : 1 > 0 ===== [-- Attachment #2: Type: text/html, Size: 2623 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-06-09 3:54 ` Kaushal Modi @ 2021-06-09 4:01 ` Kaushal Modi 2021-11-09 21:21 ` Kaushal Modi 0 siblings, 1 reply; 19+ messages in thread From: Kaushal Modi @ 2021-06-09 4:01 UTC (permalink / raw) To: Nick Savage Cc: Emanuel Berg via General discussions about Org-mode., Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 971 bytes --] Hi Nick, On Tue, Jun 8, 2021 at 11:54 PM Kaushal Modi <kaushal.modi@gmail.com> wrote: > > Thanks! I tried out your patch and it does fix the truncation of results > output. Though, it retains the shell prompt in the first eval block and > removes it only from the second eval block onwards. > I have one more feedback about the patch. I see that now indentation is stripped away from ob-shell results. I am documenting git commands in ob-shell. In the terminal, `git log' outputs: ===== commit ba72672c499af8b713437d8c05eb5979969954dd (HEAD -> main) Author: Foo Bar <foo.bar@fakeemail.com> Date: Tue Jun 8 23:56:48 2021 -0400 first commit ===== But in shell src block results, I see that that indentation is gone: ===== #+begin_src shell git log #+end_src #+RESULTS: : : commit 34924761f042eab59567ec150a73c730ab2c929c (HEAD -> main) : Author: Foo Bar <foo.bar@fakeemail.com> : Date: Tue Jun 8 23:56:48 2021 -0400 : : first commit ===== Kaushal Modi [-- Attachment #2: Type: text/html, Size: 1747 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-06-09 4:01 ` Kaushal Modi @ 2021-11-09 21:21 ` Kaushal Modi 2021-11-10 4:13 ` Ihor Radchenko 0 siblings, 1 reply; 19+ messages in thread From: Kaushal Modi @ 2021-11-09 21:21 UTC (permalink / raw) To: Nick Savage Cc: Emanuel Berg via General discussions about Org-mode., Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 283 bytes --] Hello all, I am following up on this issue with ob-shell and sessions. I had posted the recipe to reproduce the issue few months back, and it still applies; ref: https://list.orgmode.org/CAFyQvY2AeToQc2G=e+DE4VOTeTFbkvisENipYpW1Y-feu1ov1g@mail.gmail.com/ Thanks! -- Kaushal Modi [-- Attachment #2: Type: text/html, Size: 677 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-11-09 21:21 ` Kaushal Modi @ 2021-11-10 4:13 ` Ihor Radchenko 2022-06-07 0:01 ` Felix Freeman via General discussions about Org-mode. 0 siblings, 1 reply; 19+ messages in thread From: Ihor Radchenko @ 2021-11-10 4:13 UTC (permalink / raw) To: Kaushal Modi Cc: Nick Savage, Emanuel Berg via General discussions about Org-mode. Kaushal Modi <kaushal.modi@gmail.com> writes: > Hello all, > > I am following up on this issue with ob-shell and sessions. > > I had posted the recipe to reproduce the issue few months back, and it > still applies; ref: > https://list.orgmode.org/CAFyQvY2AeToQc2G=e+DE4VOTeTFbkvisENipYpW1Y-feu1ov1g@mail.gmail.com/ FYI, I attempted to fix this several times, but the underlying code is not trivial. Hopefully someone™ more familiar with comint can make a working patch. If you encounter this issue frequently, you may be the most motivated person here to spend time studying the code. Ideas how to approach the problem would be appreciated. Best, Ihor ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-11-10 4:13 ` Ihor Radchenko @ 2022-06-07 0:01 ` Felix Freeman via General discussions about Org-mode. 2022-06-07 21:50 ` Felix Freeman via General discussions about Org-mode. 2022-06-18 3:20 ` [PATCH] " Ihor Radchenko 0 siblings, 2 replies; 19+ messages in thread From: Felix Freeman via General discussions about Org-mode. @ 2022-06-07 0:01 UTC (permalink / raw) To: yantar92; +Cc: emacs-orgmode, kaushal.modi, nick Today I stumbed upon the same bug. Sadly I have no idea how to fix it, but thought it would be nice to post about it here so the bug doesn't get lost in the sands of time. Felix Freeman ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2022-06-07 0:01 ` Felix Freeman via General discussions about Org-mode. @ 2022-06-07 21:50 ` Felix Freeman via General discussions about Org-mode. 2022-06-07 22:55 ` Felix Freeman via General discussions about Org-mode. 2022-06-18 3:20 ` [PATCH] " Ihor Radchenko 1 sibling, 1 reply; 19+ messages in thread From: Felix Freeman via General discussions about Org-mode. @ 2022-06-07 21:50 UTC (permalink / raw) To: emacs-orgmode; +Cc: kaushal.modi, nick, yantar92 I've been able to bypass the error by adding the following to my .emacs (require 'ob-shell) (defun org-babel-sh-strip-weird-long-prompt (string) "Remove prompt cruft from a string of shell output." (while (string-match shell-prompt-pattern string) (setq string (substring string (match-end 0)))) string) (setq shell-prompt-pattern "^sh-[[:digit:]].[[:digit:]]\$ ") Of course, this is is just a dirty hack not suitable for a release. It seems logical that `org-babel-sh-strip-weird-long-prompt` on `ob-shell.el` respects `shell-prompt-pattern` from `shell.el`. It's not like this on the released source code. But to be honest still the idea ripping a dynamic string from the shell output seems pretty hackish to me. I'm not really aware of the internals of `comint.el`, but I think that instead it should set the shell $PS1 to an empty value and read the thing afterwards, or - if this changes things too much - at least set $PS1 to a very unique and distinctive value in order to avoid these kind of bugs. Felix Freeman ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2022-06-07 21:50 ` Felix Freeman via General discussions about Org-mode. @ 2022-06-07 22:55 ` Felix Freeman via General discussions about Org-mode. 0 siblings, 0 replies; 19+ messages in thread From: Felix Freeman via General discussions about Org-mode. @ 2022-06-07 22:55 UTC (permalink / raw) To: emacs-orgmode; +Cc: kaushal.modi, nick, yantar92 > (setq shell-prompt-pattern "^sh-[[:digit:]].[[:digit:]]\$ ") Updated this bit for something better written for emacs and suitable for my use cases: bash and SSH over TRAMP. I'll leave it here in case anyone else needs it in the future: (setq shell-prompt-pattern "^\\(sh-[[:digit:]]\\.[[:digit:]]\\|/ssh:[^:]+:[^$]+\\)$ ") Notice it requires the patch to org-babel-sh-strip-weird-long-prompt posted earlier. I hope I got it right this time ^_^' Felix Freeman ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH] babel output seems to drop anything before % (in session) 2022-06-07 0:01 ` Felix Freeman via General discussions about Org-mode. 2022-06-07 21:50 ` Felix Freeman via General discussions about Org-mode. @ 2022-06-18 3:20 ` Ihor Radchenko 2022-08-26 12:51 ` Ihor Radchenko 1 sibling, 1 reply; 19+ messages in thread From: Ihor Radchenko @ 2022-06-18 3:20 UTC (permalink / raw) To: Felix Freeman; +Cc: emacs-orgmode, kaushal.modi, nick [-- Attachment #1: Type: text/plain, Size: 788 bytes --] "Felix Freeman" <libsys@hacktivista.org> writes: > Today I stumbed upon the same bug. > > Sadly I have no idea how to fix it, but thought it would be nice to post > about it here so the bug doesn't get lost in the sands of time. I finally managed to get some working fix. At least it kind of works on my system using bash. The idea is changing shell prompt to something unique when we initialize the session. See the attached patch. Please test this on "sh" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh" and "posh". Especially on fish, csh, and posh (which I believe refers to PowerShell). Also, doing PS1="blabla > " in bash may not always work. At least it does not work with https://github.com/nojhan/liquidprompt So, more reliable ways to change prompt are welcome. Best, Ihor [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --] [-- Type: text/x-patch, Size: 2022 bytes --] From 141f810a658d652e1cb3a147c5f71659f62df86f Mon Sep 17 00:00:00 2001 Message-Id: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> From: Daniele Pizzolli <dan+git@toel.it> Date: Wed, 5 May 2021 17:00:28 +0200 Subject: [PATCH 1/2] Add disabled test for % in babel shell output --- testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index 2f346f699..e81058ac7 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -106,6 +106,35 @@ (ert-deftest ob-shell/simple-list () "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC" (org-trim (org-babel-execute-src-block)))))) +(ert-deftest ob-shell/percent-simple () + "Test percent in output." + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + +(ert-deftest ob-shell/percent-session () + "Test percent in output for session. + +At the second iteration the string before % is dropped." + :expected-result :failed + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% six\nseven 0% eight" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + (provide 'test-ob-shell) ;;; test-ob-shell.el ends here -- 2.35.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-ob-shell-Fix-output-containing-strings-matching-comi.patch --] [-- Type: text/x-patch, Size: 6579 bytes --] From 4f7a1369e72b22836cb74931c7c86e845155a729 Mon Sep 17 00:00:00 2001 Message-Id: <4f7a1369e72b22836cb74931c7c86e845155a729.1655522152.git.yantar92@gmail.com> In-Reply-To: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> References: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> From: Ihor Radchenko <yantar92@gmail.com> Date: Sat, 18 Jun 2022 11:11:12 +0800 Subject: [PATCH 2/2] ob-shell: Fix output containing strings matching `comint-prompt-regexp' * lisp/ob-comint.el (org-babel-comint-wait-for-output): Do not rely on `face-at-point' returning non-nil. * lisp/ob-shell.el (org-babel-shell-set-prompt-commands): New constant holding shell-specific commands to change prompt. (org-babel-prompt-command): New variable holding command to be user to set distinguishable prompt. (org-babel-shell-initialize): Set `org-babel-prompt-command' according to shell name. (org-babel-sh-prompt): New variable holding default shell prompt. (org-babel-sh-initiate-session): Change the default prompt to `org-babel-sh-prompt' and alter `comint-prompt-regexp' to match it tightly. * testing/lisp/test-ob-shell.el (ob-shell/percent-session): Do not expect failure in the fixed test. Fixes https://list.orgmode.org/CKK9TULBP2BG.2UITT31YJV03J@laptop/T/#mc8e3ca2f5f1b9a94040a68b4c6201234b209041c --- lisp/ob-comint.el | 4 +--- lisp/ob-shell.el | 42 ++++++++++++++++++++++++++++++----- testing/lisp/test-ob-shell.el | 4 ++-- 3 files changed, 39 insertions(+), 11 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 427aba341..1d3e24607 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -124,9 +124,7 @@ (defun org-babel-comint-wait-for-output (buffer) (while (progn (goto-char comint-last-input-end) (not (and (re-search-forward comint-prompt-regexp nil t) - (goto-char (match-beginning 0)) - (string= (face-name (face-at-point)) - "comint-highlight-prompt")))) + (goto-char (match-beginning 0))))) (accept-process-output (get-buffer-process buffer))))) (defun org-babel-comint-eval-invisibly-and-wait-for-file diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index c25941a44..f80783653 100644 --- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -42,6 +42,23 @@ (declare-function orgtbl-to-generic "org-table" (table params)) (defvar org-babel-default-header-args:shell '()) (defvar org-babel-shell-names) +(defconst org-babel-shell-set-prompt-commands + '(("fish" . "function fish_prompt\n\techo \"%s\"\nend") + ("csh" . "set prompt=\"%s\"") + ("posh" . "function prompt { \"%s\" }") + (t . "PS1=\"%s\"")) + "Alist assigning shells with their prompt setting command. + +Each element of the alist associates a shell type from +`org-babel-shell-names' with a template used to create a command to +change the default prompt. The template is an argument to `format' +that will be called with a single additional argument: prompt string. + +The fallback association template is defined in (t . \"template\") +alist element.") + +(defvar org-babel-prompt-command) + (defun org-babel-shell-initialize () "Define execution functions associated to shell names. This function has to be called whenever `org-babel-shell-names' @@ -51,7 +68,10 @@ (defun org-babel-shell-initialize () (eval `(defun ,(intern (concat "org-babel-execute:" name)) (body params) ,(format "Execute a block of %s commands with Babel." name) - (let ((shell-file-name ,name)) + (let ((shell-file-name ,name) + (org-babel-prompt-command + (or (alist-get ,name org-babel-shell-set-prompt-commands) + (alist-get t org-babel-shell-set-prompt-commands)))) (org-babel-execute:shell body params)))) (eval `(defalias ',(intern (concat "org-babel-variable-assignments:" name)) 'org-babel-variable-assignments:shell @@ -206,6 +226,13 @@ (defun org-babel-sh-var-to-string (var &optional sep hline) (mapconcat echo-var var "\n")) (t (funcall echo-var var))))) +(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'" + "String to indicate that evaluation has completed.") +(defvar org-babel-sh-eoe-output "org_babel_sh_eoe" + "String to indicate that evaluation has completed.") +(defvar org-babel-sh-prompt "org_babel_sh_prompt> " + "String to set prompt in session shell.") + (defun org-babel-sh-initiate-session (&optional session _params) "Initiate a session named SESSION according to PARAMS." (when (and session (not (string= session "none"))) @@ -213,17 +240,20 @@ (defun org-babel-sh-initiate-session (&optional session _params) (or (org-babel-comint-buffer-livep session) (progn (shell session) + ;; Set unique prompt for easier analysis of the output. + (org-babel-comint-wait-for-output (current-buffer)) + (org-babel-comint-input-command + (current-buffer) + (format org-babel-prompt-command org-babel-sh-prompt)) + (setq-local comint-prompt-regexp + (concat "^" (regexp-quote org-babel-sh-prompt) + " *")) ;; Needed for Emacs 23 since the marker is initially ;; undefined and the filter functions try to use it without ;; checking. (set-marker comint-last-output-start (point)) (get-buffer (current-buffer))))))) -(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'" - "String to indicate that evaluation has completed.") -(defvar org-babel-sh-eoe-output "org_babel_sh_eoe" - "String to indicate that evaluation has completed.") - (defun org-babel-sh-evaluate (session body &optional params stdin cmdline) "Pass BODY to the Shell process in BUFFER. If RESULT-TYPE equals `output' then return a list of the outputs diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index e81058ac7..2bfc29228 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -122,8 +122,8 @@ (ert-deftest ob-shell/percent-simple () (ert-deftest ob-shell/percent-session () "Test percent in output for session. -At the second iteration the string before % is dropped." - :expected-result :failed +Percent is matching the default value of `comint-prompt-regexp', which +can cause issues with analysis." (should (equal "one 0% two\ntree 0% four" (org-test-with-temp-text -- 2.35.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH] babel output seems to drop anything before % (in session) 2022-06-18 3:20 ` [PATCH] " Ihor Radchenko @ 2022-08-26 12:51 ` Ihor Radchenko 2022-09-25 1:44 ` Ihor Radchenko 0 siblings, 1 reply; 19+ messages in thread From: Ihor Radchenko @ 2022-08-26 12:51 UTC (permalink / raw) To: Felix Freeman; +Cc: emacs-orgmode, kaushal.modi, nick Ihor Radchenko <yantar92@gmail.com> writes: > I finally managed to get some working fix. > At least it kind of works on my system using bash. > The idea is changing shell prompt to something unique when we initialize > the session. > > See the attached patch. Fixed. Applied onto main via a35d16368. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a I did not do anything with Daniele's patch because it requires FSF copyright. -- Ihor Radchenko, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH] babel output seems to drop anything before % (in session) 2022-08-26 12:51 ` Ihor Radchenko @ 2022-09-25 1:44 ` Ihor Radchenko 2022-09-25 2:02 ` Bastien Guerry 0 siblings, 1 reply; 19+ messages in thread From: Ihor Radchenko @ 2022-09-25 1:44 UTC (permalink / raw) To: Felix Freeman, Bastien; +Cc: emacs-orgmode, kaushal.modi, nick Ihor Radchenko <yantar92@gmail.com> writes: > Applied onto main via a35d16368. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a > > I did not do anything with Daniele's patch because it requires FSF > copyright. Bastien, If I recall correctly, our tests are not all contributed by people with FSF assignments even for non-tiny contributions. May you clarify if we are following the FSF copyright assignment rules for test? -- Ihor Radchenko, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH] babel output seems to drop anything before % (in session) 2022-09-25 1:44 ` Ihor Radchenko @ 2022-09-25 2:02 ` Bastien Guerry 0 siblings, 0 replies; 19+ messages in thread From: Bastien Guerry @ 2022-09-25 2:02 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Felix Freeman, emacs-orgmode, kaushal.modi, nick Ihor Radchenko <yantar92@gmail.com> writes: > May you clarify if we are following the FSF copyright assignment rules > for test? Yes, we should. Tests are just source code. -- Bastien ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: babel output seems to drop anything before % (in session) 2021-05-05 15:15 babel output seems to drop anything before % (in session) Daniele Pizzolli 2021-05-06 2:35 ` John Corless @ 2022-07-16 10:04 ` Ihor Radchenko 1 sibling, 0 replies; 19+ messages in thread From: Ihor Radchenko @ 2022-07-16 10:04 UTC (permalink / raw) To: Daniele Pizzolli; +Cc: Emacs Org mode mailing list Daniele Pizzolli <dan@toel.it> writes: > From 897ae50cf055d459d741c90f8de94dd56a8eb3d9 Mon Sep 17 00:00:00 2001 > From: Daniele Pizzolli <dan+git@toel.it> > Date: Wed, 5 May 2021 17:00:28 +0200 > Subject: [PATCH 1/1] Add disabled test for % in babel shell output Daniele, your patch is over 15LOC. Would you mind doing FSF paperwork as described in https://orgmode.org/worg/org-contribute.html#copyright ? Best, Ihor ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2022-09-25 2:03 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-05 15:15 babel output seems to drop anything before % (in session) Daniele Pizzolli 2021-05-06 2:35 ` John Corless 2021-05-06 11:39 ` Nicholas Savage 2021-05-06 12:24 ` Ihor Radchenko 2021-05-06 23:44 ` Nick Savage 2021-05-15 20:47 ` Bastien 2021-06-09 3:41 ` Kaushal Modi 2021-06-09 3:54 ` Kaushal Modi 2021-06-09 4:01 ` Kaushal Modi 2021-11-09 21:21 ` Kaushal Modi 2021-11-10 4:13 ` Ihor Radchenko 2022-06-07 0:01 ` Felix Freeman via General discussions about Org-mode. 2022-06-07 21:50 ` Felix Freeman via General discussions about Org-mode. 2022-06-07 22:55 ` Felix Freeman via General discussions about Org-mode. 2022-06-18 3:20 ` [PATCH] " Ihor Radchenko 2022-08-26 12:51 ` Ihor Radchenko 2022-09-25 1:44 ` Ihor Radchenko 2022-09-25 2:02 ` Bastien Guerry 2022-07-16 10:04 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).