org-babel-execute-src-block filters characters from :session *shell* output

org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

Hi,

While using:

	Org mode version 9.7.3 (9.7.3-2f1844 @ /home/mec/.dotfiles/emacs/.emacs.d/elpa/org-9.7.3/)
	GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo

Let me walk you through the issue:

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: >d
: <e

Results look as expected. So far so good.

Now create a shell buffer (which will be named "*shell*")

#+begin_src elisp
(shell)
#+end_src

and try it again

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: d
: <e

Huh? what happened to the ">" before the "d"?

Let's try it again.

First kill the *shell* buffer:

#+begin_src elisp
(let ((kill-buffer-query-functions nil)) (kill-buffer "*shell*"))
#+end_src

#+RESULTS:
: t


and try it again:

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: >d
: <e

The ">" is back!

What is going on.

I'm pretty sure that `org-babel-comint--prompt-filter` is implicated which has had some recent changes (https://list.orgmode.org/87ttmn9fg0.fsf@localhost/T/) related to prompt filtering.

Any workarounds / patches most welcome!

Thanks!

~ Malcolm

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> Let me walk you through the issue:
>
> #+begin_src sh
> printf 'a\nb\nc\n>d\n<e\n'
> #+end_src
>
> #+RESULTS:
> : a
> : b
> : c
> : >d
> : <e
>
> Results look as expected. So far so good.
>
> Now create a shell buffer (which will be named "*shell*")
>
> #+begin_src elisp
> (shell)
> #+end_src
>
> and try it again
>
> #+begin_src sh
> printf 'a\nb\nc\n>d\n<e\n'
> #+end_src
>
> #+RESULTS:
> : a
> : b
> : c
> : d
> : <e
>
> Huh? what happened to the ">" before the "d"?

I cannot reproduce.
May it be that you are setting global :session header argument?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>"Cook, Malcolm" <mailto:MEC@stowers.org> writes:
>
>> Let me walk you through the issue:
>>
>> #+begin_src sh
>> printf 'a\nb\nc\n>d\n<e\n'
>> #+end_src
>>
>> #+RESULTS:
>> : a
>> : b
>> : c
>> : >d
>> : <e
>>
>> Results look as expected. So far so good.
>>
>> Now create a shell buffer (which will be named "*shell*")
>>
>> #+begin_src elisp
>> (shell)
>> #+end_src
>>
>> and try it again
>>
>> #+begin_src sh
>> printf 'a\nb\nc\n>d\n<e\n'
>> #+end_src
>>
>> #+RESULTS:
>> : a
>> : b
>> : c
>> : d
>> : <e
>>
>> Huh? what happened to the ">" before the "d"?
>
>I cannot reproduce.
>May it be that you are setting global :session header argument?

Yes, of course, so sorry, there is

#+PROPERTY: header-args:sh :session *shell* :results output

You should now be able to reproduce.

Thanks, please let me know if you have a fix, workaround, or lesson for me.

~Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

>>> #+begin_src sh
>>> printf 'a\nb\nc\n>d\n<e\n'
>>> #+end_src
>>>
>>> #+RESULTS:
>>> : a
>>> : b
>>> : c
>>> : d
>>> : <e
>>>
>>> Huh? what happened to the ">" before the "d"?
>>
>>I cannot reproduce.
>>May it be that you are setting global :session header argument?
>
> Yes, of course, so sorry, there is
>
> #+PROPERTY: header-args:sh :session *shell* :results output
>
> You should now be able to reproduce.
>
> Thanks, please let me know if you have a fix, workaround, or lesson for me.

In theory, we may provide a workaround for this kind of problem. But it
would have pros and cons.

The underlying cause is limitation of Emacs API for interactive shells -
we cannot easily distinguish command output from prompt and other extra
staff your shell/other interactive command spits into the buffer.
So, we have to either filter output the prompts ourselves to get the
command output reliably or redirect output to files, where nothing
litters the actual output with prompts.

Redirecting output works fine when we do not use sessions and do not
care what is displayed in the *shell* buffer for users. For sessions, we
use more complex approach - filter out anything that looks like prompt.

The problem with bash is that prompt is ">", and, as you showed in your
example, having ">" in the commands output breaks everything.

Again, we know this problem, and we normally force the prompt to be
something more unique when creating sessions. But we do not touch the
prompt when you point Org to an existing interactive shell buffer - that
could be a surprise if Org mode changes the prompts by force in the
shells you use interactively.

TL;DR: It is complicated. When we pull one string out, several more
       get entangled.

As a practical workaround, just do not use *shell* session names and
session names that are the same as shell buffers you create manually.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 15/06/2024 20:19, Ihor Radchenko wrote:
> The underlying cause is limitation of Emacs API for interactive shells -
> we cannot easily distinguish command output from prompt and other extra
> staff your shell/other interactive command spits into the buffer.
> So, we have to either filter output the prompts ourselves to get the
> command output reliably or redirect output to files, where nothing
> litters the actual output with prompts.

Some shells support "semantic shell" that allows terminal applications 
e.g. to copy whole command output. It is based on escape sequences.

- 
https://docs.kde.org/stable5/en/konsole/konsole/semantic-shell-integration.html
- 
https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
- https://github.com/tmux/tmux/issues/3064

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Some shells support "semantic shell" that allows terminal applications 
> e.g. to copy whole command output. It is based on escape sequences.
>
> - 
> https://docs.kde.org/stable5/en/konsole/konsole/semantic-shell-integration.html
> - 
> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md

I am looking at this gitlab link, and they way it is implemented is
simply setting PROMPT in a way that adds these escape sequences to the
original PROMPT value.

In other words, it is a bit more advanced version of our approach with
setting prompt to a unique string - they surround the prompt with unique
chars instead.

In theory, we can do the same thing, adding some kind of invisible
unicode character around the prompt. We may even do it in the existing
prompts. Although one downside will be that the unicode character we add
may be carried along if the user copies the comint buffer contents.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>>>> #+begin_src sh
>>>> printf 'a\nb\nc\n>d\n<e\n'
>>>> #+end_src
>>>>
>>>> #+RESULTS:
>>>> : a
>>>> : b
>>>> : c
>>>> : d
>>>> : <e
>>>>
>>>> Huh? what happened to the ">" before the "d"?
>>>
>>>I cannot reproduce.
>>>May it be that you are setting global :session header argument?
>>
>> Yes, of course, so sorry, there is
>>
>> #+PROPERTY: header-args:sh :session *shell* :results output
>>
>> You should now be able to reproduce.
>>
>> Thanks, please let me know if you have a fix, workaround, or lesson for me.
>
>In theory, we may provide a workaround for this kind of problem. But it
>would have pros and cons.
>
>The underlying cause is limitation of Emacs API for interactive shells -
>we cannot easily distinguish command output from prompt and other extra
>staff your shell/other interactive command spits into the buffer.
>So, we have to either filter output the prompts ourselves to get the
>command output reliably or redirect output to files, where nothing
>litters the actual output with prompts.

Yeah, I've noticed this also especially with Org as it relates to R and ESS.

>
>Redirecting output works fine when we do not use sessions and do not
>care what is displayed in the *shell* buffer for users. For sessions, we
>use more complex approach - filter out anything that looks like prompt.
>
>The problem with bash is that prompt is ">", and, as you showed in your
>example, having ">" in the commands output breaks everything.
>
>Again, we know this problem, and we normally force the prompt to be
>something more unique when creating sessions. But we do not touch the
>prompt when you point Org to an existing interactive shell buffer - that
>could be a surprise if Org mode changes the prompts by force in the
>shells you use interactively.
>
>TL;DR: It is complicated. When we pull one string out, several more
>get entangled.

Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process?  I would expect such to be significantly more complicated, but perhaps better factored?  Not that I'm offering or capable of such a re-write 😉

>As a practical workaround, just do not use *shell* session names and
>session names that are the same as shell buffers you create manually.

Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?

Thanks so much for your careful explanation and pointers, and of course all the work you do for Org ecosystem.  Perhaps time for my 1st trip to liberapay... 😉

~ Malcolm

>
>-- 
>Ihor Radchenko // yantar92,
>Org mode contributor,
>Learn more about Org mode at <https://orgmode.org>.
>Support Org development at <https://liberapay.com/org-mode>,
>or support my work at <https://liberapay.com/yantar92>
>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

>>TL;DR: It is complicated. When we pull one string out, several more
>>get entangled.
>
> Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process?  I would expect such to be significantly more complicated, but perhaps better factored?  Not that I'm offering or capable of such a re-write 😉

comint buffer is doing exactly this - it is sending input (user
comments) to the underlying shell process and receiving the output from
that process, putting it back into the comit buffer.

Unfortunately, there is no simple way to distinguish real output, shell
echoing the submitted command, and shell prompt - shells do not provide
any such information. The best they can provide is splitting between
stdout and stderr. Alas.

>>As a practical workaround, just do not use *shell* session names and
>>session names that are the same as shell buffers you create manually.
>
> Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?

That's also an option.
What you need to fiddle with is `comint-prompt-regexp'.

> Thanks so much for your careful explanation and pointers, and of course all the work you do for Org ecosystem.  Perhaps time for my 1st trip to liberapay... 😉

Thanks for the kind words.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>>>TL;DR: It is complicated. When we pull one string out, several more
>>>get entangled.
>>
>> Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process? I would expect such to be significantly more complicated, but perhaps better factored? Not that I'm offering or capable of such a re-write 😉
>
>comint buffer is doing exactly this - it is sending input (user
>comments) to the underlying shell process and receiving the output from
>that process, putting it back into the comit buffer.
>
>Unfortunately, there is no simple way to distinguish real output, shell
>echoing the submitted command, and shell prompt - shells do not provide
>any such information. The best they can provide is splitting between
>stdout and stderr. Alas.
>
>>>As a practical workaround, just do not use *shell* session names and
>>>session names that are the same as shell buffers you create manually.
>>
>> Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?
>
>That's also an option.
>What you need to fiddle with is `comint-prompt-regexp'.

This!  

Since my (bash) shell prompt is a (more or less) constant string (e.g. "myname@myhost> ").

So, my workaround is to:

    (setq comint-prompt-regexp "myname@myhost> ")

Then the filtering works perfectly.

Of course if I change my name, this will fail.  Or, more likely, connect to a different host within the shell.

Or if I change PS1 😉

It would be useful to automate this a little.

The variable needs to be set buffer-local to the shell buffer.

And it could possibly somehow ask the process for the value of PS1.

Any more TIPS on doing this?

Or perhaps advice that I shouldn't want to ...??? 😉

Cheers,

~ Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

> So, my workaround is to:
> 
>     (setq comint-prompt-regexp "myname@myhost> ")
> 
> Then the filtering works perfectly.
> 
> Of course if I change my name, this will fail.  Or, more likely, connect to a
> different host within the shell.
> 
> Or if I change PS1 😉
> 
> It would be useful to automate this a little.
> 
> The variable needs to be set buffer-local to the shell buffer.
> 
> And it could possibly somehow ask the process for the value of PS1.

FWIW: the following gets at it...

(with-current-buffer "*shell*"  (setq comint-prompt-regexp   (buffer-substring-no-properties (pos-bol) (pos-eol))))

> 
> Any more TIPS on doing this?
> 
> Or perhaps advice that I shouldn't want to ...??? 😉
> 
> Cheers,
> 
> ~ Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> Since my (bash) shell prompt is a (more or less) constant string (e.g. "myname@myhost> ").
>
> So, my workaround is to:
>
>     (setq comint-prompt-regexp "myname@myhost> ")
>
> Then the filtering works perfectly.
>
> Of course if I change my name, this will fail.  Or, more likely, connect to a different host within the shell.
>
> Or if I change PS1 😉
>
> It would be useful to automate this a little.

There is internal buffer-local variable called
`org-babel-comint-prompt-regexp-old'. If you set it to the default
`comint-prompt-regexp', Org will attempt to use it if no prompt is
detected for longer than `org-babel-comint-fallback-regexp-threshold'
seconds.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>