Re: bash source code block: problem after ssh commands

[Matt <matt@excalamus.com>]

 ---- On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 

 > This has nothing to do with Emacs comint and this is also not a bug in
 > Emacs 

Ihor, there were two claims made in the original report.  I was referring to Claim 2.  That deals with M-x shell and therefore comint-mode.

Regarding Claim 1:

- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
  + M-x org-version
  + M-x emacs-version

I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ /home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).

* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with contents "foo" as well as give "bar" as the result.

    #+begin_src bash :results output
    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
    echo "bar"
    #+end_src

The reported behavior is that "foo_file" is created with "foo" (with "foo" is not stated, but implied) and "bar" is *not* given as the result.

** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like xfce4-terminal executes the ssh line as expected and outputs the result of the second line.  It was noted that this does not happen with M-x shell.

* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result.  The result is "bar" for me.

The exact "expected behavior" for a shell block is a little fuzzy.  According to my analysis (given below), what Alain reports (remote file and no "bar") is the "expected" behavior.  What I see (no remote file and "bar") is actually "unexpected".

I used the following to test the claim:

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

I am unable to reproduce the reported behavior (of "bar" not returning).  Instead, I get an ssh-askpass permission denied error, foo_file is not created, and "bar" is given as the result.  I do not see anywhere in the thread that the original claim was reproduced.

The thread preceded something like follows.

Leo Butler suggested two work arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Russell Adams suggested another work around:

- add -n to the ssh command

Ihor identified that a non-session call does something like the following command:

    bash -c bash /tmp/temp-file-with-source-block-code.sh

where

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

The second line (significantly different from the original report) pipes the echo result to stdout and to a file, bar.txt.  Writing to a file allows us to confirm if that line was executed.

Ihor suggested that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not run the second line because an interactive password prompt is displayed by ssh.  The reasoning is that the prompt hangs the process while waiting for input and the second line never runs.  Indeed, running the command does not produce /tmp/bar.txt.

Ihor is correct about prompts messing with shell blocks (this is not the first time he's seen this).  However, the way it's stated does not demonstrate it. This is because Emacs does *not* make a call like

    bash -c bash /tmp/temp-file-with-source-block-code.sh

Alain responded by pointing out that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not execute the first line.  This is true.  Consider calling

    bash -c bash /tmp/two-lines.sh

where

    ------ /tmp/two-lines.sh ------
    echo "first" > /tmp/first.txt
    echo "second" > /tmm/second.txt
    -------------------------------

Neither first.txt or second.txt are created.

Max Nikulin interjected with a helpful reminder that Bash scripting is a snakepit of footguns.  (What Max said is more than that and interesting.  I skip it here because it depends on the form of the call.)

Before trying to untangle what a given Bash command does, we need to be sure what command is actually called.  Unfortunately, there's not a clear Bash command corresponding to how Emacs makes the call.

What happens goes something like this:

1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object implemented in C
3. DEFUN ("call-process") forwards this information to the C function call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess

A lot of cleaning and setup happens which is dependent on the system (GNU, Window, Darwin, etc.).  There's a call to openp which looks for the executable.  An emacs_pipe is set up (I assume for writing to stdout and stderr).  I'm unable to give a definitive answer as to what precisely emacs_spawn calls.  Do any of the args "get quotes?"  I can't say.

Bruno Barbier commented that his understanding of process-file is that it gets commands from stdin.  Maybe that's what the call to emacs_pipe is doing?

He gives the example:

#+begin_quote
(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash
#+end_quote

He then proposes an experiment to close stdin.  To do this, he calls

    #+begin_src shell :results output
    exec 0>&-
    echo OK
    #+end_src

He claims that "exec 0<&-" closes stdin.  I believe there is a typo.  It's not clear if it has a negative effect, though.  According to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]],

#+begin_quote
Closing File Descriptors

n<&-

    Close input file descriptor n.
0<&-, <&-

    Close stdin.
n>&-

    Close output file descriptor n.
1>&-, >&-

    Close stdout.
#+end_quote

What Bruno writes corresponds to "closing output file descriptor 0".  I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor".  I had no luck finding this information in man bash or info bash.

Rerunning the experiment according to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], the result is the same: "OK" is *not* printed.

    #+begin_src shell :results output
    exec 0<&-
    echo OK
    #+end_src

Doing the following echoes OK for either direction of the redirection:

  -- /tmp/exec-OKlt.sh ---
  exec 0<&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKlt.sh
  #+end_example

  -- /tmp/exec-OKgt.sh ---
  exec 0>&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKgt.sh
  #+end_example

The INFILE passed to process-file looks like,

    #+begin_src emacs-lisp
    (process-file
       "bash"
       "/tmp/two-lines.sh"
       '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
       nil
       "-c" "bash")
    #+end_src

So, the call Emacs makes is probably more close to:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

What this exactly does is unclear to me.  It appears to pass the contents of /tmp/two-lines.sh to a subshell process.  That is, it seems to behave like "bash /tmp/two-lines.sh" is run in a subprocess.

Running this in xfce4-terminal, I get what I expect:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

Each line is echoed to file so nothing is written to the console.  However, both files are created with the expected text.  Both lines executed.

If I update two-lines to output to std,

    ------ /tmp/two-lines-tee.sh ------
    echo "first"  | tee /tmp/first.txt
    echo "second" | tee /tmp/second.txt
    -----------------------------------

I see "first" and "second" echoed to the console:

    ahab@pequod /tmp$ cat two-lines-tee.sh | bash -c bash
    first
    second

Running the following, neither give an output to the console:

#+begin_example
ahab@pequod /tmp$ cat exec-OKlt.sh | bash -c bash
ahab@pequod /tmp$ cat exec-OKgt.sh | bash -c bash
#+end_example

This is what we see in Org.  I'll be honest, though, I don't really know what to expect with exec 0>&- and exec 0<&-.  When I call them in the terminal, it kills the terminal.

The surprising bit is that running this in xfce4-terminal

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

    #+begin_example
    cat /tmp/temp-file-with-source-block-code.sh | bash -c bash
    #+end_example

does *not* echo bar (and does not create /tmp/bar.txt) yet it creates foo_file.  I get prompted for my password and then the second line doesn't execute.  Nothing prints to the console and no bar.txt is created.

This is the behavior Alain reports happening in Org (that I am unable to reproduce).  That is, the *reported behavior is the expected behavior* (assuming my analysis is correct).  However, according to the behavior I see when I run the block (fails to create the remote file and echoes "bar"), Org does the "wrong thing".  I can't account for this.

Anyway, Ihor's main point stands: a prompt does not work with non-session shell blocks.  The following returns exit code 1 (which means fail):

    #+begin_src bash :results output
    read -p "What? "
    #+end_src

As far as I can tell, though, that's not what prevents "bar" from being returned.  As far as I can reproduce, calling

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

*does* give "bar" for results even though it shouldn't.

** Comment 2.
The second claim has nothing to do with Org Babel.  I was able to confirm it and provide the steps to reproduce.  I think it would make sense to report it upstream and let them decide if it's expected behavior.  I'm still happy to do that, but I need to step away from the keyboard :)