org alters output of bash code blocks when run with :session

org alters output of bash code blocks when run with :session

[Tyler Smith]

Hi,

(originally posted to emacs.stackexchange[1] with no answers, )

I am running a short bash command from an org mode code block, using the program `fold` to wrap it at 60 characters. If I run the code without a session argument, the results display as expected. However, when I set the session argument (which I have to do in the context of the long literate analysis I'm doing), some of the output characters don't appear.

Steps to reproduce:

    emacs -Q

Open the following file and execute each code block in turn:

+------------------------
    #+BEGIN_SRC elisp setup
    (org-babel-do-load-languages
     'org-babel-load-languages
     '((shell . t)))
    #+END_SRC
    
    #+RESULTS:
    
    #+BEGIN_SRC bash no session :results replace verbatim
     echo 'CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFDGGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGEGC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7' | fold -w 60
    #+END_SRC
    
    #+RESULTS:
    : CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
    : GGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFD
    : GGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGE
    : GC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7
    
    #+BEGIN_SRC bash session :results replace verbatim :session my-session
     echo 'CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFDGGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGEGC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7' | fold -w 60
    #+END_SRC
    
    #+RESULTS:
    : CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
    : GGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFD
    : FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGE
    : 5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7
+------------------------

Notice the difference between the two bash results blocks. In the first, the input string is folded at column 60, without any other changes. In the second, a string of characters has been deleted: 

- `GGGFCF:,FF9>` from the beginning of the third line 
- `GC5>` from the beginning of the fourth line

The fact that the deleted strings both start at the beginning of a line and end with a `>` suggests that the session thinks they should be redirected somehow, and in the process is dropping them completely. However, when I look at the session buffer, the output appears as expected, with no missing characters.

How do I force the babel session to leave the output alone, and print it verbatim into the org file?


Thanks!

Tyler

GNU Emacs 26.0.90
Org mode version 9.1.4

[1] https://emacs.stackexchange.com/questions/38964/org-alters-output-of-bash-code-blocks-when-run-with-session

-- 
plantarum.ca

Re: org alters output of bash code blocks when run with :session

[Nicolas Goaziou]

Hello,

Tyler Smith <tyler@plantarum.ca> writes:

> I am running a short bash command from an org mode code block, using
> the program `fold` to wrap it at 60 characters. If I run the code
> without a session argument, the results display as expected. However,
> when I set the session argument (which I have to do in the context of
> the long literate analysis I'm doing), some of the output characters
> don't appear.
>
> Steps to reproduce:
>
>     emacs -Q
>
> Open the following file and execute each code block in turn:
>
> +------------------------
>
>     #+BEGIN_SRC elisp setup
>     (org-babel-do-load-languages
>      'org-babel-load-languages
>      '((shell . t)))
>     #+END_SRC
>
>     
>     #+RESULTS:
>     
>
>     #+BEGIN_SRC bash no session :results replace verbatim
>      echo 'CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFDGGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGEGC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7' | fold -w 60
>     #+END_SRC
>
>     
>     #+RESULTS:
>     : CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
>     : GGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFD
>     : GGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGE
>     : GC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7
>     
>
>     #+BEGIN_SRC bash session :results replace verbatim :session my-session
>      echo 'CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFDGGGFCF:,FF9>FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGEGC5>5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7' | fold -w 60
>     #+END_SRC
>
>     
>     #+RESULTS:
>     : CCCC8,,CEFGDF,,,CC,CF:,,C8CFGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
>     : GGGGGGCFGGGG@GGGGGGGGGGGGGGGGGGGGGGGGGGGGG+FF:B3CDFGGGFGGCFD
>     : FEC,F<FEG*?FFFFFGGG*CFEEECGGGGC+>FDGGDGFDFFEEFGE
>     : 5?C*CGGF+3:*<7*2>+*<9CGCD::C=*@DCFCFFGG<C9<)7*9.5<@BA*7
> +------------------------
>
> Notice the difference between the two bash results blocks. In the
> first, the input string is folded at column 60, without any other
> changes. In the second, a string of characters has been deleted:
>
> - `GGGFCF:,FF9>` from the beginning of the third line 
> - `GC5>` from the beginning of the fourth line
>
> The fact that the deleted strings both start at the beginning of
> a line and end with a `>` suggests that the session thinks they should
> be redirected somehow, and in the process is dropping them completely.
> However, when I look at the session buffer, the output appears as
> expected, with no missing characters.
>
> How do I force the babel session to leave the output alone, and print
> it verbatim into the org file?

It may be related to other problems related to prompt in shell sessions.
See for example
<http://lists.gnu.org/r/emacs-orgmode/2018-02/msg00081.html>,
<http://lists.gnu.org/r/emacs-orgmode/2018-01/msg00415.html>, and for
step forward a solution
<http://lists.gnu.org/archive/html/emacs-orgmode/2017-04/msg00147.html>.

Patches welcome!

Regards,

-- 
Nicolas Goaziou

Re: org alters output of bash code blocks when run with :session

[Tyler Smith]

On Mon, Feb 26, 2018, at 6:27 AM, Nicolas Goaziou wrote:
> Tyler Smith <tyler@plantarum.ca> writes:
> >
> > How do I force the babel session to leave the output alone, and print
> > it verbatim into the org file?
> 
> It may be related to other problems related to prompt in shell sessions.
> See for example
> <http://lists.gnu.org/r/emacs-orgmode/2018-02/msg00081.html>,
> <http://lists.gnu.org/r/emacs-orgmode/2018-01/msg00415.html>, and for
> step forward a solution
> <http://lists.gnu.org/archive/html/emacs-orgmode/2017-04/msg00147.html>.
> 
> Patches welcome!

It looks like ob-shell sets up sessions just by calling `shell`, which means all the user's customizations for shell-mode are automatically applied. This includes setting the prompt (e.g., from .emacs.d/init_bash.sh), as well as the value of `comint-prompt-regexp`.

I can't see any straightforward way to modify ob-shell sessions without also changing regular shell-mode. So I wonder if the easiest way to address this would be to create a new mode derived from shell-mode, ('ob-shell-mode')? That would give us a way to modify the prompt code inside babel without interfering with shell-mode.

If that sounds sensible I can put together a patch for ob-shell.el. Let me know what you think.

Best,

Tyler

Re: org alters output of bash code blocks when run with :session

[Nicolas Goaziou]

Hello,

Tyler Smith <tyler@plantarum.ca> writes:

> I can't see any straightforward way to modify ob-shell sessions
> without also changing regular shell-mode. So I wonder if the easiest
> way to address this would be to create a new mode derived from
> shell-mode, ('ob-shell-mode')? That would give us a way to modify the
> prompt code inside babel without interfering with shell-mode.

Creating a new mode doesn't sound like the easiest way.

What about the following suggestion, from a link I posted in this
thread:

    Thanks for the examples. Anything goes, of course, but I'm hoping for a
    tighter integration. Maybe a :prompt param to sh source blocks, which
    sets the prompt in a session, and which also manipulates
    `comint-prompt-regexp' or `shell-prompt-pattern' local in the session buffer.

    Or a mechanism like in Tramp, where the shell prompt is set to a random
    string not expected in the shell (see `tramp-end-of-output'), and
    `comint-prompt-regexp' or `shell-prompt-pattern' are adapted
    automatically local in the session buffer.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738