* bash source code block: problem after ssh commands
@ 2023-10-25 11:17 Alain.Cochard
2023-10-25 15:12 ` Leo Butler
` (2 more replies)
0 siblings, 3 replies; 54+ messages in thread
From: Alain.Cochard @ 2023-10-25 11:17 UTC (permalink / raw)
To: emacs-orgmode
Hello.
For me, 'C-c C-c' on the following group
#+begin_src bash :results output
echo "foo"
echo "bar"
#+end_src
works as I expect: 'foo' and 'bar' are echo'ed. But it fails on this
one:
#+begin_src bash :results output
ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src
The file 'foo_file' is created on the remote machine, but 'bar' is not
echo'ed. I have tried to insert other commands between the ssh and
echo commands, or append semicolons at the end of the commands,
without success.
By contrast, it works with this one:
#+begin_src bash :results output
ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
#+end_src
Perhaps it is worth noting that I observe the same behavior if I
simply copy/yank the commands (I mean: the two commands yank'ed at
once) in an emacs terminal (obtained with 'M-x shell'). However,
everything works as I expect if I copy/yank the commands (all at once)
in a "regular" X terminal (xfce in my case).
I wish I could understand what is going on.
Thanks
--
EOST (École et Observatoire des Sciences de la Terre)
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44
F-67084 Strasbourg Cedex, France | [ slot available for rent ]
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: bash source code block: problem after ssh commands 2023-10-25 11:17 bash source code block: problem after ssh commands Alain.Cochard @ 2023-10-25 15:12 ` Leo Butler 2023-10-25 16:14 ` Alain.Cochard 2023-10-26 8:44 ` Ihor Radchenko 2023-10-26 14:44 ` Russell Adams 2023-11-18 8:09 ` Max Nikulin 2 siblings, 2 replies; 54+ messages in thread From: Leo Butler @ 2023-10-25 15:12 UTC (permalink / raw) To: Alain.Cochard@unistra.fr; +Cc: emacs-orgmode@gnu.org On Wed, Oct 25 2023, Alain.Cochard@unistra.fr wrote: > Hello. > > For me, 'C-c C-c' on the following group > > #+begin_src bash :results output > echo "foo" > echo "bar" > #+end_src > > > works as I expect: 'foo' and 'bar' are echo'ed. But it fails on this > one: > > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" > echo "bar" > #+end_src > > > The file 'foo_file' is created on the remote machine, but 'bar' is not > echo'ed. I have tried to insert other commands between the ssh and > echo commands, or append semicolons at the end of the commands, > without success. > > By contrast, it works with this one: > > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar" > #+end_src > > Perhaps it is worth noting that I observe the same behavior if I > simply copy/yank the commands (I mean: the two commands yank'ed at > once) in an emacs terminal (obtained with 'M-x shell'). However, > everything works as I expect if I copy/yank the commands (all at once) > in a "regular" X terminal (xfce in my case). It looks like an issue in comint-mode and the way that it is handling the temporary ssh session. This is not an issue with org-mode. Here are a couple work-arounds: - add the -f to the ssh command - add a semi-colon and line continuation to the first line. Working examples: #+begin_src bash :results output ssh -f SSH-HOST 'echo foo>/tmp/foo_file' echo $(uname -a) |tee /tmp/uname.txt #+end_src #+RESULTS: : Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux #+begin_src bash :results output ssh pearce-120 'echo foo>/tmp/foo_file' &&\ echo $(uname -a) | tee /tmp/uname.txt #+end_src #+RESULTS: : Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux Not working: #+begin_src bash :results output ssh SSH-HOST 'echo foo>/tmp/foo_file' echo $(uname -a) |tee /tmp/uname1.txt #+end_src If you look at /tmp after running the third example, you see that that second command line has not been executed. Leo ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 15:12 ` Leo Butler @ 2023-10-25 16:14 ` Alain.Cochard 2023-10-25 16:47 ` Leo Butler 2023-10-25 16:59 ` yaxp 2023-10-26 8:44 ` Ihor Radchenko 1 sibling, 2 replies; 54+ messages in thread From: Alain.Cochard @ 2023-10-25 16:14 UTC (permalink / raw) To: Leo Butler; +Cc: Alain.Cochard@unistra.fr, emacs-orgmode@gnu.org Thanks much for the detailed explanations. > It looks like an issue in comint-mode and the way that it is > handling the temporary ssh session. This is not an issue with > org-mode. So it is an issue with emacs, right? -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 16:14 ` Alain.Cochard @ 2023-10-25 16:47 ` Leo Butler 2023-10-25 16:59 ` yaxp 1 sibling, 0 replies; 54+ messages in thread From: Leo Butler @ 2023-10-25 16:47 UTC (permalink / raw) To: Alain.Cochard@unistra.fr; +Cc: emacs-orgmode@gnu.org On Wed, Oct 25 2023, Alain.Cochard@unistra.fr wrote: > Thanks much for the detailed explanations. > > > It looks like an issue in comint-mode and the way that it is > > handling the temporary ssh session. This is not an issue with > > org-mode. > > So it is an issue with emacs, right? Yes, that is my understanding. Perhaps you can report it to the emacs-devel mailing list. Leo ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 16:14 ` Alain.Cochard 2023-10-25 16:47 ` Leo Butler @ 2023-10-25 16:59 ` yaxp 1 sibling, 0 replies; 54+ messages in thread From: yaxp @ 2023-10-25 16:59 UTC (permalink / raw) To: emacs-orgmode > So it is an issue with emacs, right? comint-mode -- (yaxp me) => t ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 15:12 ` Leo Butler 2023-10-25 16:14 ` Alain.Cochard @ 2023-10-26 8:44 ` Ihor Radchenko 2023-10-26 13:23 ` Alain.Cochard 1 sibling, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-10-26 8:44 UTC (permalink / raw) To: Leo Butler; +Cc: Alain.Cochard@unistra.fr, emacs-orgmode@gnu.org Leo Butler <Leo.Butler@umanitoba.ca> writes: > It looks like an issue in comint-mode and the way that it is handling > the temporary ssh session. This is not an issue with org-mode. > ... > Not working: > > #+begin_src bash :results output > ssh SSH-HOST 'echo foo>/tmp/foo_file' > echo $(uname -a) |tee /tmp/uname1.txt > #+end_src By default (no session), the above command should not use comint-mode. So, something else might be going on here. I currently do not have a setup to test ssh commands, so I would appreciate someone trying to dig why `org-babel-eval' apparently not returning the expected result. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-26 8:44 ` Ihor Radchenko @ 2023-10-26 13:23 ` Alain.Cochard 2023-10-26 13:44 ` Ihor Radchenko 0 siblings, 1 reply; 54+ messages in thread From: Alain.Cochard @ 2023-10-26 13:23 UTC (permalink / raw) To: Ihor Radchenko Cc: Leo Butler, Alain.Cochard@unistra.fr, emacs-orgmode@gnu.org Ihor Radchenko writes on Thu 26 Oct 2023 08:44: > I currently do not have a setup to test ssh commands, Sorry if that's irrelevant: I realized that one can ssh to the *same* machine. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-26 13:23 ` Alain.Cochard @ 2023-10-26 13:44 ` Ihor Radchenko 2023-10-27 18:26 ` Alain.Cochard 0 siblings, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-10-26 13:44 UTC (permalink / raw) To: alain.cochard; +Cc: Leo Butler, emacs-orgmode@gnu.org Alain.Cochard@unistra.fr writes: > Ihor Radchenko writes on Thu 26 Oct 2023 08:44: > > > I currently do not have a setup to test ssh commands, > > Sorry if that's irrelevant: I realized that one can ssh to the *same* > machine. You are indeed right. I can now reproduce the problem locally. It boils down to (setq exit-status (process-file shell-file-name input-file (if error-file (list t error-file) t) nil shell-command-switch command)) that is an equivalent of bash -c bash /path/to/file-containing-the-source-code.sh ----- file-containing-the-source-code.sh ---- ssh localhost 'echo foo>/tmp/foo_file' echo $(uname -a) |tee /tmp/uname1.txt --------------------------------------------- If one tries to evaluate the above, the second line is not produced. It has nothing to do with Emacs itself. However, if I try (process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not produced. This time, it is Emacs problem. The problem is caused by interactive password prompt displayed by ssh. I guess we may try to report this upstream, though I am not sure if this is something that is supported to start with. From Org perspective, it is not the first time when interactive shell programs are causing problems. I'd say that we have no technical ability to support them reliably and one has to use ob-screen or similar libraries to properly interact with such code blocks. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-26 13:44 ` Ihor Radchenko @ 2023-10-27 18:26 ` Alain.Cochard 2023-10-28 5:22 ` Max Nikulin 2023-10-30 10:50 ` Bruno Barbier 0 siblings, 2 replies; 54+ messages in thread From: Alain.Cochard @ 2023-10-27 18:26 UTC (permalink / raw) To: Ihor Radchenko; +Cc: alain.cochard, Leo Butler, emacs-orgmode@gnu.org Ihor Radchenko writes on Thu 26 Oct 2023 13:44: > I can now reproduce the problem locally. > > It boils down to > > (setq exit-status > (process-file shell-file-name input-file > (if error-file > (list t error-file) > t) > nil shell-command-switch command)) > > that is an equivalent of > > bash -c bash /path/to/file-containing-the-source-code.sh > > ----- file-containing-the-source-code.sh ---- > ssh localhost 'echo foo>/tmp/foo_file' > echo $(uname -a) |tee /tmp/uname1.txt > --------------------------------------------- > If one tries to evaluate the above, the second line is not produced. It > has nothing to do with Emacs itself. I am confused about what you specifically do to "evaluate the above". To start with, I have to use quotes to make your command be performed: bash -c "bash /path/to/file-containing-the-source-code.sh" Without the quotes, the 1st line is not executed either. NB: I get the same behavior with simply bash /path/to/file-containing-the-source-code.sh or, after making the script executable with 'chmod +x': /path/to/file-containing-the-source-code.sh But most importantly, the second line *is* produced, either if I use an SSH key for passwordless access or if I enter the password manually. > However, if I try > > (process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not > produced. Here too, it is not clear to me how you "try". The way I know is to highlight (process-file "bash" "/path/to/file-containing-the-source-code.sh") and do M-x eval-region In this case, indeed, the 1st line is executed but not the 2nd one (again, whether or not I use an SSH key). -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-27 18:26 ` Alain.Cochard @ 2023-10-28 5:22 ` Max Nikulin 2023-10-30 10:50 ` Bruno Barbier 1 sibling, 0 replies; 54+ messages in thread From: Max Nikulin @ 2023-10-28 5:22 UTC (permalink / raw) To: alain.cochard; +Cc: Leo Butler, emacs-orgmode@gnu.org On 28/10/2023 01:26, Alain.Cochard@unistra.fr wrote: > bash -c "bash /path/to/file-containing-the-source-code.sh" > > Without the quotes, the 1st line is not executed either. NB: I get > the same behavior with simply Without quotes it means: execute shell script (just "bash" this case) with "$0" set to "/path/to/file-containing-the-source-code.sh". "/path/..." is ignored because shell script does not contain "$0". It is a way to safely pass arguments to shell, however "$0" is confusing: sh -c 'echo "arg0: $0"; echo args: "$@"' arg0 arg1 arg2 arg0: arg0 args: arg1 arg2 In this example "arg0: $0"; echo args: "$@"' has the same role as "bash" in the cited command. If you need to pass another variable into "sh -c" the use var="/bin/sh" sh -c 'ls -l "$1"' argv0-for-sh "$var" Unsafe variant ("var" contain quotes): sh -c "ls -l '$var'". The same warning is applicable to "emacs --eval "(func \"$var\")". > or, after making the script executable with 'chmod +x': > > /path/to/file-containing-the-source-code.sh Always add a shebang ("#!/bin/sh", "#!/bin/bash", etc.) to executable scripts. A chance to face a bug is a shell is higher when it is left up to the shell and the kernel what to do with files without shebangs. Sorry, I have not tried to debug the original issue. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-27 18:26 ` Alain.Cochard 2023-10-28 5:22 ` Max Nikulin @ 2023-10-30 10:50 ` Bruno Barbier 2023-11-06 13:32 ` Ihor Radchenko 1 sibling, 1 reply; 54+ messages in thread From: Bruno Barbier @ 2023-10-30 10:50 UTC (permalink / raw) To: Alain.Cochard, Ihor Radchenko Cc: alain.cochard, Leo Butler, emacs-orgmode@gnu.org Hi, Alain.Cochard@unistra.fr writes: > Ihor Radchenko writes on Thu 26 Oct 2023 13:44: > > > I can now reproduce the problem locally. > > > > It boils down to > > > > (setq exit-status > > (process-file shell-file-name input-file > > (if error-file > > (list t error-file) > > t) > > nil shell-command-switch command)) > > > > that is an equivalent of > > > > bash -c bash /path/to/file-containing-the-source-code.sh > > ... > I am confused about what you specifically do to "evaluate the above". > To start with, I have to use quotes to make your command be performed: > > bash -c "bash /path/to/file-containing-the-source-code.sh" > ... > > But most importantly, the second line *is* produced, either if I use > an SSH key for passwordless access or if I enter the password > manually. IIUC, the elisp expression: (process-file "bash" "/tmp/test.sh") is more equivalent to: cat /tmp/test.sh | bash i.e. the shell is getting the commands from stdin. Thus, any command that uses stdin might change what gets executed or not. I'm able to reproduce using the following minimal script, without passwords nor SSH (where the cryptic first line closes stdin). #+begin_src shell :results output exec 0>&- echo OK #+end_src The result is "OK" only when commenting out the first line; else, the echo command is not executed (because stdin has been closed). Here is an other example, where the second echo is eaten by the script itself: #+begin_src shell :results output echo 1 read -p "Next command? " NEXT_COMMAND echo 2 echo 3 #+end_src #+RESULTS: : 1 : 3 Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-30 10:50 ` Bruno Barbier @ 2023-11-06 13:32 ` Ihor Radchenko 2023-11-06 18:25 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-06 13:32 UTC (permalink / raw) To: Bruno Barbier; +Cc: Alain.Cochard, Leo Butler, emacs-orgmode@gnu.org Bruno Barbier <brubar.cs@gmail.com> writes: > IIUC, the elisp expression: > > (process-file "bash" "/tmp/test.sh") > > is more equivalent to: > > cat /tmp/test.sh | bash > > i.e. the shell is getting the commands from stdin. Thus, any command > that uses stdin might change what gets executed or not. Looking at `org-babel-sh-evaluate', it should be enough to specify :shebang, :cmdline, or :stdin header argument to force using script rather than channel source block as input to bash: #+begin_src shell :results output :cmdline bash exec 0>&- echo OK #+end_src #+RESULTS: : OK #+begin_src shell :results output :cmdline bash echo 1 read -p "Next command? " NEXT_COMMAND echo 2 echo 3 #+end_src #+RESULTS: : 1 : 2 : 3 I am wondering about the possible downsides of using script approach instead of stdin redirection. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-06 13:32 ` Ihor Radchenko @ 2023-11-06 18:25 ` Matt 2023-11-07 8:55 ` Ihor Radchenko 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-06 18:25 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode@gnu.org ---- On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > I am wondering about the possible downsides of using script approach > instead of stdin redirection. I'm curious to hear more about what you're thinking. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-06 18:25 ` Matt @ 2023-11-07 8:55 ` Ihor Radchenko 2023-11-08 19:41 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-07 8:55 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode@gnu.org Matt <matt@excalamus.com> writes: > ---- On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > > I am wondering about the possible downsides of using script approach > > instead of stdin redirection. > > I'm curious to hear more about what you're thinking. I am thinking to change the (t (org-babel-eval shell-file-name (org-trim body))) clause in `org-babel-sh-evaluate' to something that uses a script file. It will clearly solve the discussed problem, possibly at the cost of small overhead to write the script file to disk. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-07 8:55 ` Ihor Radchenko @ 2023-11-08 19:41 ` Matt 2023-11-09 12:14 ` Ihor Radchenko 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-08 19:41 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode@gnu.org ---- On Tue, 07 Nov 2023 09:53:46 +0100 Ihor Radchenko wrote --- > Matt matt@excalamus.com> writes: > > > ---- On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > > > I am wondering about the possible downsides of using script approach > > > instead of stdin redirection. > > > > I'm curious to hear more about what you're thinking. > > I am thinking to change the > (t (org-babel-eval shell-file-name (org-trim body))) > clause in `org-babel-sh-evaluate' to something that uses a script file. > > It will clearly solve the discussed problem, possibly at the cost of > small overhead to write the script file to disk. Thanks for clarifying. I've been away from the codebase for a bit and, now that the FSF paperwork is signed (still need to get Craig a copy), I'm reviewing =ob-shell= with an eye for what could be cleaned or improved. I feel like =org-babel-sh-evaluate= could use some attention. I'm open to your suggestion. The good is that using a script is how :shebang and :cmdline are processed currently (like you pointed out) so there's precedence and experience with it. Also, it would make all non-session execution use the same model (script versus comint). I like how that would create a clean separation. For bad, nothing jumps out to me as obviously a problem. Let me "think out loud" for a moment. We'd need to write to disk. Like you say, this incurs overhead opening, writing, and closing the file. It's not like we'd forget to close it, though. Nor is running out of space or inodes our problem. Writing requires permission. That's not an issue with /tmp. Then, it needs to execute. Aside from permission, any code we insert needs to be correct. For example, a shebang would need to point to the correct application and any arguments would need to correspond to the implementation being called. I doubt we'd need anything beyond /bin/<shell>. FWIW, it looks like there's been at least one instance where :shebang's formatting was questioned (https://yhetil.org/orgmode/CA+A2iZZ1vMmKiUf4Fem1AU7CA1m9GQap+BkvrOsz+0BxRt6rRA@mail.gmail.com/). We'd also need to control for what environment the script runs in. That was another issue I saw raised in the list (https://yhetil.org/orgmode/87609ug5ae.fsf@luisa.c0t0d0s0.de/). Of course, we'd need to read the stdout and stderr. This is handled by =process-file=. Any step I missed or some kind of failure I didn't consider? ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-08 19:41 ` Matt @ 2023-11-09 12:14 ` Ihor Radchenko 2023-11-09 17:48 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-09 12:14 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode@gnu.org Matt <matt@excalamus.com> writes: > ... Aside from permission, any code we > insert needs to be correct. For example, a shebang would need to point > to the correct application and any arguments would need to correspond > to the implementation being called. I doubt we'd need anything beyond > /bin/<shell>. FWIW, it looks like there's been at least one instance > where :shebang's formatting was questioned > (https://yhetil.org/orgmode/CA+A2iZZ1vMmKiUf4Fem1AU7CA1m9GQap+BkvrOsz+0BxRt6rRA@mail.gmail.com/). It is a problem, but I am not sure how useful it is to solve the line numbering there. A bigger problem is that errors *Org Babel Error Output* are not clickable: Try #+begin_src bash cdf #+end_src and you will get ------ *Org Babel Error Output* --------- bash: line 1: cdf: command not found [ Babel evaluation exited with code 127 ] ----------------------------------------- Clicking on "line 1" will yield file opening dialogue. However, there is yet another problem revealed when we try #+begin_src bash :cmdline bash cdf #+end_src #+RESULTS: : /tmp/babel-47DarV/sh-script-vkbRfJ: line 2: cdf: command not found There is no *Org Babel Error Output* buffer at all. So, it looks like we may need an alternative `org-babel-eval' function that works with script files rather than input and arranges stdout/stderr separation. > We'd also need to control for what environment the script runs in. > That was another issue I saw raised in the list > (https://yhetil.org/orgmode/87609ug5ae.fsf@luisa.c0t0d0s0.de/). Of > course, we'd need to read the stdout and stderr. This is handled by > =process-file=. Any step I missed or some kind of failure I didn't > consider? Valid concern. We should reuse `process-file' in this scenario as well, but utilizing its ARGS argument + arrange to generate the temporary script file in the appropriate context - see `make-nearby-temp-file'. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-09 12:14 ` Ihor Radchenko @ 2023-11-09 17:48 ` Matt 2023-11-15 16:32 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-09 17:48 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode@gnu.org ---- On Thu, 09 Nov 2023 13:13:36 +0100 Ihor Radchenko wrote --- > A bigger problem is that errors *Org Babel Error > Output* are not clickable: Interesting. > So, it looks like we may need an alternative `org-babel-eval' function > that works with script files rather than input and arranges > stdout/stderr separation. Yes, it seems like a separate handler may be in order. > We should reuse `process-file' in this scenario as well, > but utilizing its ARGS argument + arrange to generate the temporary > script file in the appropriate context - see `make-nearby-temp-file'. I was unaware of `make-nearby-temp-file'. Thank you. It's helpful for this situation. I'm not in a position to address these points right now. My main focus is re-familiarizing myself with the =ob-shell= and =ob-comint= source. When I was implementing async shell evaluatation earlier this year, I remember having the impression that execution paths, especially `org-babel-comint-with-output', could probably be simplified. It seems to me that everything we're talking about in this thread aligns with that objective. I've made a note of your points and have every intention to return to them when I get to writing code. However, don't let me stop you if this is something you were interested in doing. I'll do what I can to assist. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-09 17:48 ` Matt @ 2023-11-15 16:32 ` Matt 2023-11-15 18:04 ` Matt 2023-11-16 9:32 ` Ihor Radchenko 0 siblings, 2 replies; 54+ messages in thread From: Matt @ 2023-11-15 16:32 UTC (permalink / raw) To: emacs-orgmode@gnu.org I was poking around, learning how sessions are started. Basically, `shell` creates the comint buffer using `make-comint-in-buffer`. What I find interesting is that `make-comint-in-buffer` can also create a comint buffer from a network stream: (let ((buff "*localhost-process-buffer*")) (switch-to-buffer (make-comint-in-buffer "localhost-process" buff "ssh" nil (format "%s@localhost" (getenv "USER"))))) So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that. Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`. `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used. Therefore, we could provide header arguments to pass parameters and the destination to ssh. I haven't made a judgment yet about whether any of this is good or bad. I thought it was interesting and figured I'd share. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-15 16:32 ` Matt @ 2023-11-15 18:04 ` Matt 2023-11-16 9:32 ` Ihor Radchenko 1 sibling, 0 replies; 54+ messages in thread From: Matt @ 2023-11-15 18:04 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode@gnu.org ---- On Wed, 15 Nov 2023 17:32:19 +0100 Matt wrote --- > (let ((buff "*localhost-process-buffer*")) > (switch-to-buffer > (make-comint-in-buffer > "localhost-process" > buff > "ssh" > nil > (format "%s@localhost" (getenv "USER"))))) It looks like the way this is called actually uses `start-file-process` rather than `open-network-stream`. It's not clear to me how to call it so that `open-network-stream` is used. However, that's more a question for the Emacs mailing list and, honestly, not one that I think I care about right now. Posting this just to correct what I said before. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-15 16:32 ` Matt 2023-11-15 18:04 ` Matt @ 2023-11-16 9:32 ` Ihor Radchenko 2023-11-16 19:03 ` Matt 1 sibling, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-16 9:32 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode@gnu.org Matt <matt@excalamus.com> writes: > So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that. Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`. `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used. Therefore, we could provide header arguments to pass parameters and the destination to ssh. Or we can make use of TRAMP as we usually do to access remote environment. IMHO, it is more reliable as TRAMP takes care about arranging all the Elisp FS interaction to work on remote system. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-16 9:32 ` Ihor Radchenko @ 2023-11-16 19:03 ` Matt 2023-11-16 19:46 ` Alain.Cochard 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-16 19:03 UTC (permalink / raw) To: Ihor Radchenko; +Cc: alaincochard, emacs-orgmode@gnu.org ---- On Thu, 16 Nov 2023 10:30:59 +0100 Ihor Radchenko wrote --- > Or we can make use of TRAMP as we usually do to access remote > environment. IMHO, it is more reliable as TRAMP takes care about > arranging all the Elisp FS interaction to work on remote system. Agreed. Also, I wasn't aware TRAMP worked with shell block evaluation. That's nice. Sure enough, these both work: #+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote* echo foo>foo_file echo "bar" #+end_src #+begin_src bash :results output :dir /ssh:user@localhost:/home/user echo foo>foo_file echo "bar" #+end_src Of course, both calls happen on the remote. I wonder, would this have helped Alain? I'm admittedly a little confused about the original report, #+name: original-report #+begin_src bash :results output ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src Clearly, the ssh line is intended to happen on the remote server. Was the echo "bar" intended to happen locally? PS: I've updated the WORG page for ob-shell with a :dir section (https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54) ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-16 19:03 ` Matt @ 2023-11-16 19:46 ` Alain.Cochard 2023-11-16 20:54 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Alain.Cochard @ 2023-11-16 19:46 UTC (permalink / raw) To: Matt; +Cc: Ihor Radchenko, alaincochard, emacs-orgmode@gnu.org Matt writes on Thu 16 Nov 2023 20:03: > > ---- On Thu, 16 Nov 2023 10:30:59 +0100 Ihor Radchenko wrote --- > > > Or we can make use of TRAMP as we usually do to access remote > > environment. IMHO, it is more reliable as TRAMP takes care about > > arranging all the Elisp FS interaction to work on remote system. > > Agreed. Also, I wasn't aware TRAMP worked with shell block > evaluation. That's nice. > > Sure enough, these both work: > > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote* > echo foo>foo_file > echo "bar" > #+end_src > > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user > echo foo>foo_file > echo "bar" > #+end_src > > Of course, both calls happen on the remote. > > I wonder, would this have helped Alain? No (see below). > I'm admittedly a little confused about the original report, > > #+name: original-report > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" > echo "bar" > #+end_src > > Clearly, the ssh line is intended to happen on the remote server. > Was the echo "bar" intended to happen locally? Yes. (So I think that explains the 'no' above.) Then I'm confused about your confusion... Anyway, this gives me the opportunity to come back to the question of whether or not there is a problem with emacs itself (like some people here thought), and if some message should be sent to some emacs list. Again my argument was that the 2 commands copy/yank'ed in an emacs terminal don't work as expected, while they do if the same is done in an X terminal. I guess it is not important for Org, because workarounds have be proposed, but I think if there is a problem with emacs it should be reported, right? Thank you. -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-16 19:46 ` Alain.Cochard @ 2023-11-16 20:54 ` Matt 2023-11-17 9:22 ` Ihor Radchenko 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-16 20:54 UTC (permalink / raw) To: alaincochard; +Cc: Ihor Radchenko, emacs-orgmode@gnu.org alaincochard alain.cochard@unistra.fr writes: > Yes. (So I think that explains the 'no' above.) Thank you for clarifying. > Anyway, this gives me the opportunity to come back to the question of > whether or not there is a problem with emacs itself (like some people > here thought), and if some message should be sent to some emacs list. > Again my argument was that the 2 commands copy/yank'ed in an emacs > terminal don't work as expected, while they do if the same is done in > an X terminal. Okay, I follow you now. Yes, I agree with what others have said, it's related to Emacs (probably comint-mode). It happens with M-x shell and *not* with M-x eshell. It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste. M-x with term-line-mode doesn' t print "bar". > I guess it is not important for Org, because workarounds have be > proposed, but I think if there is a problem with emacs it should be > reported, right? I think it's worth reporting. I'm headed to bed and can submit it tomorrow for you if you're not comfortable with doing it or don't have the time. Here are the steps I followed to reproduce with M-x shell: 1. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 2. Paste them into a shell (tested with xterm or xfce4-terminal) and press return 3. After entering the correct password, "bar" is echoed 4. Paste them into the buffer associated with M-x shell 5. After entering the correct password, "bar" is NOT echoed Here are the steps I followed to reproduce with M-x term: 1. M-x term 2. C-c C-j to switch to term-line-mode 3. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 4. Paste them into the buffer associated with M-x term 5. After entering the correct password, "bar" is NOT echoed ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-16 20:54 ` Matt @ 2023-11-17 9:22 ` Ihor Radchenko 2023-11-17 9:55 ` Alain.Cochard 2023-11-17 22:07 ` Matt 0 siblings, 2 replies; 54+ messages in thread From: Ihor Radchenko @ 2023-11-17 9:22 UTC (permalink / raw) To: Matt; +Cc: alaincochard, emacs-orgmode@gnu.org Matt <matt@excalamus.com> writes: > Thank you for clarifying. > > > Anyway, this gives me the opportunity to come back to the question of > > whether or not there is a problem with emacs itself (like some people > > here thought), and if some message should be sent to some emacs list. > > Again my argument was that the 2 commands copy/yank'ed in an emacs > > terminal don't work as expected, while they do if the same is done in > > an X terminal. > > Okay, I follow you now. > > Yes, I agree with what others have said, it's related to Emacs (probably comint-mode). It happens with M-x shell and *not* with M-x eshell. It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste. M-x with term-line-mode doesn' t print "bar". I think that I need to clarify here. We are talking about two different things: 1. Bash src block without session 2. Bash src block with session The original report used bash src blocks _without_ session. In such scenario, comint (and M-x shell) is not relevant. Org simply uses `process-file' with INFILE argument. This is equivalent of someone opening __X shell__, and literally typing ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" In the scenario described in the report, when ssh asks for password, it is equivalent to $ ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" Password: <typing 'echo "bar"' here> This has nothing to do with Emacs comint and this is also not a bug in Emacs - we use INFILE argument, that is equivalent to the above as _per docstring_. So, Org mode is simply not using `process-file' function as users expect - instead of executing a bash script, we emulate interactive user input to bash. (Side note: for someone aware about these details, it is possible to do something like #+begin_src bash ssh remote-server; <<super-secret-password-derived-from-library-of-babel-block-stored-in-safe-private-place>> #+end_src or, as a demo #+begin_src bash read x; value2 echo "We just read \"$x\""; #+end_src #+RESULTS: : We just read "value2" <end of side note>) Another case is (2), when we do use comint is session is what we discussed in https://list.orgmode.org/orgmode/CAL1eYuJntGbXY6A794qM7PTbXH3DiU1aF6OayS7CLf3kOBsbig@mail.gmail.com/ I guess we can try to report this as a bug, especially since it also manifests itself when comint is used interactively. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 9:22 ` Ihor Radchenko @ 2023-11-17 9:55 ` Alain.Cochard 2023-11-17 10:17 ` Ihor Radchenko 2023-11-17 22:07 ` Matt 1 sibling, 1 reply; 54+ messages in thread From: Alain.Cochard @ 2023-11-17 9:55 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Matt, alaincochard, emacs-orgmode@gnu.org Ihor Radchenko writes on Fri 17 Nov 2023 09:22: > I think that I need to clarify here. Thank you. Well, thank you for _trying_: most of the discussion in this thread is way beyond my pay grade. At the most basic user level (i.e., non lisp aware), why is it not necessarily a bug if "something" does the expected in an X terminal but not in an emacs terminal? I think Matt and I (and others) are on the same page here, and he already essentially drafted the bug report I would have sent (only better). What also confuses me is that it seems to me that, in the minimum working example, you consider the ssh command and the read command as equivalent. But I don't even enter the password when using ssh... -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 9:55 ` Alain.Cochard @ 2023-11-17 10:17 ` Ihor Radchenko 2023-11-17 15:32 ` Leo Butler ` (2 more replies) 0 siblings, 3 replies; 54+ messages in thread From: Ihor Radchenko @ 2023-11-17 10:17 UTC (permalink / raw) To: Alain.Cochard; +Cc: Matt, emacs-orgmode@gnu.org Alain.Cochard@unistra.fr writes: > Ihor Radchenko writes on Fri 17 Nov 2023 09:22: > > > I think that I need to clarify here. > > Thank you. > > Well, thank you for _trying_: most of the discussion in this thread is > way beyond my pay grade. My reply was to Matthew, the current ob-shell maintainer. If you want, you need to examine ob-shell (`org-babel-sh-evaluate') and ob-eval (`org-babel-eval') code, and try to step through the code using debugger to understand technical details in thread better. > At the most basic user level (i.e., non lisp aware), why is it not > necessarily a bug if "something" does the expected in an X terminal > but not in an emacs terminal? I think Matt and I (and others) are on > the same page here, and he already essentially drafted the bug report > I would have sent (only better). WRT M-x shell, feel free to submit a bug report. I mostly pointed that the problem with M-x shell is not the problem you originally ran to. It is a different problem (also, we ran into it in the past). > What also confuses me is that it seems to me that, in the minimum > working example, you consider the ssh command and the read command as > equivalent. But I don't even enter the password when using ssh... I was only able to reproduce your problem with ssh asking a password. We are discussing the reproduced case. If you see problems with #+begin_src bash :results output ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src even when ssh does not ask for a password, please provide more detailed reproducer that we can replicate locally without guessing your ssh config. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 10:17 ` Ihor Radchenko @ 2023-11-17 15:32 ` Leo Butler 2023-11-17 15:47 ` Bruno Barbier 2023-11-18 8:04 ` bash source code block: problem after ssh commands Max Nikulin 2 siblings, 0 replies; 54+ messages in thread From: Leo Butler @ 2023-11-17 15:32 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Alain.Cochard@unistra.fr, Matt, emacs-orgmode@gnu.org On Fri, Nov 17 2023, Ihor Radchenko <yantar92@posteo.net> wrote: > I was only able to reproduce your problem with ssh asking a password. > We are discussing the reproduced case. > > If you see problems with > > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" > echo "bar" > #+end_src > > even when ssh does not ask for a password, please provide more detailed > reproducer that we can replicate locally without guessing your ssh config. I think you are confused, Ihor. The bug was confirmed here: https://list.orgmode.org/orgmode/87bkcmlor9.fsf@t14.reltub.ca/ I mis-identified the culprit; you corrected me by pointing out the bug is in `process-file'. And you confirmed the bug: https://list.orgmode.org/orgmode/87fs1xbis1.fsf@localhost/ Russell Adams and I suggested ssh-specific work-arounds that prevent ssh from grabbing stdin (either -n or -f). Leo ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 10:17 ` Ihor Radchenko 2023-11-17 15:32 ` Leo Butler @ 2023-11-17 15:47 ` Bruno Barbier 2023-11-18 10:37 ` Ihor Radchenko 2023-11-19 4:17 ` Non-emacs shell (Re: bash source code block: problem after ssh commands) Max Nikulin 2023-11-18 8:04 ` bash source code block: problem after ssh commands Max Nikulin 2 siblings, 2 replies; 54+ messages in thread From: Bruno Barbier @ 2023-11-17 15:47 UTC (permalink / raw) To: Ihor Radchenko, Alain.Cochard; +Cc: Matt, emacs-orgmode@gnu.org Hi Matt, Ihor, Alain, Ihor Radchenko <yantar92@posteo.net> writes: > Alain.Cochard@unistra.fr writes: > >> At the most basic user level (i.e., non lisp aware), why is it not >> necessarily a bug if "something" does the expected in an X terminal >> but not in an emacs terminal? I think Matt and I (and others) are on >> the same page here, and he already essentially drafted the bug report >> I would have sent (only better). > > WRT M-x shell, feel free to submit a bug report. I mostly pointed that > the problem with M-x shell is not the problem you originally ran to. It > is a different problem (also, we ran into it in the past). FWIW, M-x shell differs from what a plain terminal is doing (xterm, in my case), but, I do prefer 'M-x shell' behavior: it allows me to copy multiple lines, getting the same results as when I type them manually, or copy them line by line. My xterm doesn't seem to allow me to do that. >> What also confuses me is that it seems to me that, in the minimum >> working example, you consider the ssh command and the read command as >> equivalent. But I don't even enter the password when using ssh... I've introduced the 'read' example, as a simpler way to modify the standard input and get the same kind of "unexpected" results, without relying on SSH. It seems that I have only caused confusion, sorry about that. IIUC, the OP example is not working because SSH is modifying the standard input (with or without passwords). > I was only able to reproduce your problem with ssh asking a password. > We are discussing the reproduced case. > > If you see problems with > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" > echo "bar" > #+end_src > > even when ssh does not ask for a password, please provide more detailed > reproducer that we can replicate locally without guessing your ssh config. > I'm not the OP, but, my SSH is configured to work without passwords and SSH is still consuming lines from standard input: #+begin_src bash :results output ssh phone echo "remote" echo "local" #+end_src #+RESULTS: : remote It looks like it is a known SSH "feature" (see https://unix.stackexchange.com/a/688024): #+begin_src bash :results output seq 1000000 | (ssh phone sleep 1; wc -l) #+end_src #+RESULTS: : 675173 Same block, but asking SSH to not use stdin (using '-n' as mentionned in this thread): #+begin_src bash :results output seq 1000000 | (ssh -n phone sleep 1; wc -l) #+end_src #+RESULTS: : 1000000 IMHO, what ob-shell is doing today seems a valid way of evaluating source blocks (and it seems to have been like that for a long time). It should probably be documented somewhere, so that users know how to write their source blocks, or switch to another way, like adding a :cmdline parameter as mentionned in this thread. Hoping I didn't increase the confusion again, :-) Bruno > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 15:47 ` Bruno Barbier @ 2023-11-18 10:37 ` Ihor Radchenko 2023-11-21 19:01 ` Bruno Barbier 2023-11-19 4:17 ` Non-emacs shell (Re: bash source code block: problem after ssh commands) Max Nikulin 1 sibling, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-18 10:37 UTC (permalink / raw) To: Bruno Barbier; +Cc: Alain.Cochard, Matt, emacs-orgmode@gnu.org Bruno Barbier <brubar.cs@gmail.com> writes: >> WRT M-x shell, feel free to submit a bug report. I mostly pointed that >> the problem with M-x shell is not the problem you originally ran to. It >> is a different problem (also, we ran into it in the past). > > FWIW, M-x shell differs from what a plain terminal is doing (xterm, in > my case), but, I do prefer 'M-x shell' behavior: it allows me to copy > multiple lines, getting the same results as when I type them manually, > or copy them line by line. My xterm doesn't seem to allow me to do that. The behavior of M-x shell can indeed be made use of. However, this particular difference with xterm, AFAIU, is not documented - unaware users may be surprised. The situation is worse with Org shell blocks - users naturally expect script-like behavior (even for :session), but run into edge cases like this and get confused. We should either document the caveats, or, preferably, make the behavior more consistent with expectations. At least, by default. That's why I think that filing a bug report makes sense from Org mode project point of view. > It looks like it is a known SSH "feature" (see > https://unix.stackexchange.com/a/688024): > > #+begin_src bash :results output > seq 1000000 | (ssh phone sleep 1; wc -l) > #+end_src > > #+RESULTS: > : 675173 > ... > IMHO, what ob-shell is doing today seems a valid way of evaluating > source blocks (and it seems to have been like that for a long time). It > should probably be documented somewhere, so that users know how to write > their source blocks, or switch to another way, like adding a :cmdline > parameter as mentionned in this thread. And, as Max demonstrated, this ssh feature only works with bash, but not necessarily other shells. Moreover, "-n" option that disables the above may depend on ssh configuration (see StdinNull in man 5 ssh_config). So, I believe that it only leads to confusion, even if we try to document it. __By default__, Org should produce more expected behavior - what users would get from running a script file rather than from redirecting stdin. We can optionally leave the stdin redirection as an option to be used by the users who understand the peculiarities. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-18 10:37 ` Ihor Radchenko @ 2023-11-21 19:01 ` Bruno Barbier 2023-11-22 17:06 ` Max Nikulin 0 siblings, 1 reply; 54+ messages in thread From: Bruno Barbier @ 2023-11-21 19:01 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Alain.Cochard, Matt, emacs-orgmode@gnu.org Ihor Radchenko <yantar92@posteo.net> writes: > Bruno Barbier <brubar.cs@gmail.com> writes: > >> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in >> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy >> multiple lines, getting the same results as when I type them manually, >> or copy them line by line. My xterm doesn't seem to allow me to do that. > > The behavior of M-x shell can indeed be made use of. > However, this particular difference with xterm, AFAIU, is not > documented - unaware users may be surprised. > The situation is worse with Org shell blocks - users naturally expect > script-like behavior (even for :session), but run into edge cases like > this and get confused. > > We should either document the caveats, or, preferably, make the behavior > more consistent with expectations. At least, by default. > That's why I think that filing a bug report makes sense from Org mode > project point of view. Thanks for the explanation. I see you got my point. We'll see what Emacs maintainers will say about the current behavior of M-x shell; filling the bug report was definitely a good idea anyway. Ihor Radchenko <yantar92@posteo.net> writes: > __By default__, Org should produce more expected behavior - what users > would get from running a script file rather than from redirecting stdin. > We can optionally leave the stdin redirection as an option to be used by > the users who understand the peculiarities. I agree that it would be simpler to switch to the script-like behavior by default on org side. About the interactive-like behavior, that would be nice to keep it, if some people rely on it in their existing documents (I personally don't). Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-21 19:01 ` Bruno Barbier @ 2023-11-22 17:06 ` Max Nikulin 0 siblings, 0 replies; 54+ messages in thread From: Max Nikulin @ 2023-11-22 17:06 UTC (permalink / raw) To: emacs-orgmode On 22/11/2023 02:01, Bruno Barbier wrote: > Ihor Radchenko writes: >> __By default__, Org should produce more expected behavior - what users >> would get from running a script file rather than from redirecting stdin. >> We can optionally leave the stdin redirection as an option to be used by >> the users who understand the peculiarities. > > I agree that it would be simpler to switch to the script-like behavior > by default on org side. About the interactive-like behavior, that > would be nice to keep it, if some people rely on it in their existing > documents (I personally don't). I do not like recommendations curl https://... | sudo bash but I regularly see them. So some users might expect behavior like namely stdin, not like interactive prompt commands. Perhaps even sessions may be switched to creation of a temporary file and executing "source tmpfile.sh". However it may be shell-specific (posh?). Likely it is possible to implement a header argument to explicitly control whether `process-file' should be used or the block should be executed as a temporary script file. Currently it is implicitly determined from other header arguments. P.S. https://mywiki.wooledge.org/BashFAQ/089 "I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!" A similar case, but only data is read from stdin, script is a regular file, no difference bash vs. dash. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Non-emacs shell (Re: bash source code block: problem after ssh commands) 2023-11-17 15:47 ` Bruno Barbier 2023-11-18 10:37 ` Ihor Radchenko @ 2023-11-19 4:17 ` Max Nikulin 2023-11-21 15:33 ` Bruno Barbier 1 sibling, 1 reply; 54+ messages in thread From: Max Nikulin @ 2023-11-19 4:17 UTC (permalink / raw) To: emacs-orgmode On 17/11/2023 22:47, Bruno Barbier wrote: > FWIW, M-x shell differs from what a plain terminal is doing (xterm, in > my case), but, I do prefer 'M-x shell' behavior: it allows me to copy > multiple lines, getting the same results as when I type them manually, > or copy them line by line. My xterm doesn't seem to allow me to do that. I am unsure what do you expect from xterm, but perhaps it is not responsibility of a terminal application. Multiple lines can be copied to regular BASH prompt (bracketed paste is enabled by default nowadays), however it may be inconvenient to edit. You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key binding) to start an editor and to paste multiple commands there. See also "fc" BASH built-in for editing and re-executing last commands. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Non-emacs shell (Re: bash source code block: problem after ssh commands) 2023-11-19 4:17 ` Non-emacs shell (Re: bash source code block: problem after ssh commands) Max Nikulin @ 2023-11-21 15:33 ` Bruno Barbier 0 siblings, 0 replies; 54+ messages in thread From: Bruno Barbier @ 2023-11-21 15:33 UTC (permalink / raw) To: Max Nikulin, emacs-orgmode Hi Max, Max Nikulin <manikulin@gmail.com> writes: > On 17/11/2023 22:47, Bruno Barbier wrote: >> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in >> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy >> multiple lines, getting the same results as when I type them manually, >> or copy them line by line. My xterm doesn't seem to allow me to do that. > > I am unsure what do you expect from xterm, but perhaps it is not > responsibility of a terminal application. It has been said in this thread that 'M-x shell' should be fixed to match the behavior that we see in a plain terminal, when we copy multiple lines. I just wanted to point out that I do prefer the way 'M-x shell' handles the copy of multiple lines. > Multiple lines can be copied to regular BASH prompt (bracketed paste is > enabled by default nowadays), however it may be inconvenient to edit. > You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key > binding) to start an editor and to paste multiple commands there. > See also "fc" BASH built-in for editing and re-executing last commands. Thanks Max! I didn't know that. I should definitely start using this when I'm stuck in a console, to safely copy/edit my commands using Emacs. Thanks, Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 10:17 ` Ihor Radchenko 2023-11-17 15:32 ` Leo Butler 2023-11-17 15:47 ` Bruno Barbier @ 2023-11-18 8:04 ` Max Nikulin 2023-11-18 10:43 ` Ihor Radchenko 2 siblings, 1 reply; 54+ messages in thread From: Max Nikulin @ 2023-11-18 8:04 UTC (permalink / raw) To: emacs-orgmode On 17/11/2023 17:17, Ihor Radchenko wrote: > I was only able to reproduce your problem with ssh asking a password. > We are discussing the reproduced case. I see bash vs. dash difference with public key authorization, so no need for password prompts. I have not figured out how to construct an example without ssh since this command *may* read stdin, but does not do it in a same way as e.g. cat(1). Perhaps a small program performing single non-blocking read will allow it. The following behavior observed for a regular shell prompt, Emacs is not involved. Debian 12 bookworm. cat ssh-script.sh ssh -p 2222 127.0.0.1 'echo foo>/tmp/foo' echo done Read commands from a script file: dash ssh-script.sh done bash ssh-script.sh done Read commands from stdin dash <ssh-script.sh done bash <ssh-script.sh # no output I have not expected this difference. dash reads a block from stdin (whole file in this case) and interprets commands. BASH reads just the ssh command and executes it. SSH reads "echo done" from stdin, so when control is returned to bash, stdin is exhausted and no commands remain to execute by BASH. SSH can not "unread" part of input not consumed by the remote command despite it might be possible in the case of the regular file as stdin. Actually bash reads the whole script file as well when called as it is shown above, but it calls lseek before executing ssh. To make difference more apparent (e.g. for strace), force creation of pipe(7) for which lseek is not supported cat ssh-script.sh | strace -o /tmp/bash.strace bash I am unsure if POSIX specifies exact behavior of shell when commands are read from stdin. I think, the suggested earlier "-n" ssh option (or </dev/null) should be used to make intentions clear: ssh should not read stdin. There is too much room for heuristics: interactive vs. non-interactive shell, a terminal vs. a regular file vs. a pipe as standard input. Be explicit to get reliable behavior. ssh -n user@host 'command' or tar cvf - . | ssh user@host 'tar xvf -' without "-n" when ssh needs stdin and it is explicitly specified. I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash vs. dash issue. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-18 8:04 ` bash source code block: problem after ssh commands Max Nikulin @ 2023-11-18 10:43 ` Ihor Radchenko 2023-11-18 16:18 ` Max Nikulin 0 siblings, 1 reply; 54+ messages in thread From: Ihor Radchenko @ 2023-11-18 10:43 UTC (permalink / raw) To: Max Nikulin; +Cc: emacs-orgmode Max Nikulin <manikulin@gmail.com> writes: > ... > I have not expected this difference. > > dash reads a block from stdin (whole file in this case) and interprets > commands. > > BASH reads just the ssh command and executes it. SSH reads "echo done" > from stdin, so when control is returned to bash, stdin is exhausted and > no commands remain to execute by BASH. SSH can not "unread" part of > input not consumed by the remote command despite it might be possible in > the case of the regular file as stdin. > > Actually bash reads the whole script file as well when called as it is > shown above, but it calls lseek before executing ssh. To make difference > more apparent (e.g. for strace), force creation of pipe(7) for which > lseek is not supported > ... > I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash > vs. dash issue. I still see it as a bug - what Org mode does to run the shell blocks is not what users expect. _By default_, we _should_ produce more expected behavior. The observed inconsistency between different shells just indicates that our approach with `process-file' should not be used as it leads to potentially confusing results (they are not confusing only to people who dig deeply into ssh, bash/dash/etc, and Org mode internals). I cannot find any clear motivation behind using `process-file' + INFILE in the git logs. I assume that it was used simply because it is easier compared to trying to create and run temporary script file on remote hosts. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-18 10:43 ` Ihor Radchenko @ 2023-11-18 16:18 ` Max Nikulin 0 siblings, 0 replies; 54+ messages in thread From: Max Nikulin @ 2023-11-18 16:18 UTC (permalink / raw) To: emacs-orgmode On 18/11/2023 17:43, Ihor Radchenko wrote: > > I still see it as a bug - what Org mode does to run the shell blocks is > not what users expect. _By default_, we _should_ produce more expected > behavior. A shell without unexpected behavior is neither POSIX nor a descendant shell like BASH https://mywiki.wooledge.org/BashPitfalls Behavior if interactive and non-interactive shells is not the same anyway. > I cannot find any clear motivation behind using `process-file' + INFILE > in the git logs. I assume that it was used simply because it is easier > compared to trying to create and run temporary script file on remote > hosts. From my point of view it is rather natural to use `process-file' since this function is available out of the box. I suspect some subtle issues may appear even if script file is forced. On the other hand ob-shell already has a (bit buggy) implementation relying on script files. It is used when :stdin or :cmdline is specified. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 9:22 ` Ihor Radchenko 2023-11-17 9:55 ` Alain.Cochard @ 2023-11-17 22:07 ` Matt 2023-11-18 3:11 ` Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) Max Nikulin ` (2 more replies) 1 sibling, 3 replies; 54+ messages in thread From: Matt @ 2023-11-17 22:07 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode@gnu.org ---- On Fri, 17 Nov 2023 10:20:28 +0100 Ihor Radchenko wrote --- > This has nothing to do with Emacs comint and this is also not a bug in > Emacs Ihor, there were two claims made in the original report. I was referring to Claim 2. That deals with M-x shell and therefore comint-mode. Regarding Claim 1: - Can anyone verify Claim 1? - Is anyone else unable to verify Claim 1 (like me)? - What versions are people using? + M-x org-version + M-x emacs-version I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ /home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0). * The original report has two claims: ** Claim 1. The following block is expected to write a remote file called "foo_file" with contents "foo" as well as give "bar" as the result. #+begin_src bash :results output ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src The reported behavior is that "foo_file" is created with "foo" (with "foo" is not stated, but implied) and "bar" is *not* given as the result. ** Claim 2. Copying and pasting the two lines from the first claim into a terminal like xfce4-terminal executes the ssh line as expected and outputs the result of the second line. It was noted that this does not happen with M-x shell. * Comments about the claims: ** Comment 1. tl;dr I can't reproduce the claim that "bar" is *not* the result. The result is "bar" for me. The exact "expected behavior" for a shell block is a little fuzzy. According to my analysis (given below), what Alain reports (remote file and no "bar") is the "expected" behavior. What I see (no remote file and "bar") is actually "unexpected". I used the following to test the claim: #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" #+end_src I am unable to reproduce the reported behavior (of "bar" not returning). Instead, I get an ssh-askpass permission denied error, foo_file is not created, and "bar" is given as the result. I do not see anywhere in the thread that the original claim was reproduced. The thread preceded something like follows. Leo Butler suggested two work arounds: - add the -f to the ssh command - add a semi-colon and line continuation to the first line. Russell Adams suggested another work around: - add -n to the ssh command Ihor identified that a non-session call does something like the following command: bash -c bash /tmp/temp-file-with-source-block-code.sh where ----- /tmp/temp-file-with-source-block-code ----- ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt ------------------------------------------------- The second line (significantly different from the original report) pipes the echo result to stdout and to a file, bar.txt. Writing to a file allows us to confirm if that line was executed. Ihor suggested that bash -c bash /tmp/temp-file-with-source-block-code.sh does not run the second line because an interactive password prompt is displayed by ssh. The reasoning is that the prompt hangs the process while waiting for input and the second line never runs. Indeed, running the command does not produce /tmp/bar.txt. Ihor is correct about prompts messing with shell blocks (this is not the first time he's seen this). However, the way it's stated does not demonstrate it. This is because Emacs does *not* make a call like bash -c bash /tmp/temp-file-with-source-block-code.sh Alain responded by pointing out that bash -c bash /tmp/temp-file-with-source-block-code.sh does not execute the first line. This is true. Consider calling bash -c bash /tmp/two-lines.sh where ------ /tmp/two-lines.sh ------ echo "first" > /tmp/first.txt echo "second" > /tmm/second.txt ------------------------------- Neither first.txt or second.txt are created. Max Nikulin interjected with a helpful reminder that Bash scripting is a snakepit of footguns. (What Max said is more than that and interesting. I skip it here because it depends on the form of the call.) Before trying to untangle what a given Bash command does, we need to be sure what command is actually called. Unfortunately, there's not a clear Bash command corresponding to how Emacs makes the call. What happens goes something like this: 1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path to a temp file containing the source block code, and ARGS ("-c" "bash") 2. This information is passed to DEFUN ("call-process"), a Lisp object implemented in C 3. DEFUN ("call-process") forwards this information to the C function call_process 4. call_process calls emacs_spawn 5. emacs_spawn creates a subprocess A lot of cleaning and setup happens which is dependent on the system (GNU, Window, Darwin, etc.). There's a call to openp which looks for the executable. An emacs_pipe is set up (I assume for writing to stdout and stderr). I'm unable to give a definitive answer as to what precisely emacs_spawn calls. Do any of the args "get quotes?" I can't say. Bruno Barbier commented that his understanding of process-file is that it gets commands from stdin. Maybe that's what the call to emacs_pipe is doing? He gives the example: #+begin_quote (process-file "bash" "/tmp/test.sh") is more equivalent to: cat /tmp/test.sh | bash #+end_quote He then proposes an experiment to close stdin. To do this, he calls #+begin_src shell :results output exec 0>&- echo OK #+end_src He claims that "exec 0<&-" closes stdin. I believe there is a typo. It's not clear if it has a negative effect, though. According to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], #+begin_quote Closing File Descriptors n<&- Close input file descriptor n. 0<&-, <&- Close stdin. n>&- Close output file descriptor n. 1>&-, >&- Close stdout. #+end_quote What Bruno writes corresponds to "closing output file descriptor 0". I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor". I had no luck finding this information in man bash or info bash. Rerunning the experiment according to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], the result is the same: "OK" is *not* printed. #+begin_src shell :results output exec 0<&- echo OK #+end_src Doing the following echoes OK for either direction of the redirection: -- /tmp/exec-OKlt.sh --- exec 0<&- echo OK ---------------------- #+begin_example bash /tmp/exec-OKlt.sh #+end_example -- /tmp/exec-OKgt.sh --- exec 0>&- echo OK ---------------------- #+begin_example bash /tmp/exec-OKgt.sh #+end_example The INFILE passed to process-file looks like, #+begin_src emacs-lisp (process-file "bash" "/tmp/two-lines.sh" '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH") nil "-c" "bash") #+end_src So, the call Emacs makes is probably more close to: #+begin_example cat /tmp/two-lines.sh | bash -c bash #+end_example What this exactly does is unclear to me. It appears to pass the contents of /tmp/two-lines.sh to a subshell process. That is, it seems to behave like "bash /tmp/two-lines.sh" is run in a subprocess. Running this in xfce4-terminal, I get what I expect: #+begin_example cat /tmp/two-lines.sh | bash -c bash #+end_example Each line is echoed to file so nothing is written to the console. However, both files are created with the expected text. Both lines executed. If I update two-lines to output to std, ------ /tmp/two-lines-tee.sh ------ echo "first" | tee /tmp/first.txt echo "second" | tee /tmp/second.txt ----------------------------------- I see "first" and "second" echoed to the console: ahab@pequod /tmp$ cat two-lines-tee.sh | bash -c bash first second Running the following, neither give an output to the console: #+begin_example ahab@pequod /tmp$ cat exec-OKlt.sh | bash -c bash ahab@pequod /tmp$ cat exec-OKgt.sh | bash -c bash #+end_example This is what we see in Org. I'll be honest, though, I don't really know what to expect with exec 0>&- and exec 0<&-. When I call them in the terminal, it kills the terminal. The surprising bit is that running this in xfce4-terminal ----- /tmp/temp-file-with-source-block-code ----- ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt ------------------------------------------------- #+begin_example cat /tmp/temp-file-with-source-block-code.sh | bash -c bash #+end_example does *not* echo bar (and does not create /tmp/bar.txt) yet it creates foo_file. I get prompted for my password and then the second line doesn't execute. Nothing prints to the console and no bar.txt is created. This is the behavior Alain reports happening in Org (that I am unable to reproduce). That is, the *reported behavior is the expected behavior* (assuming my analysis is correct). However, according to the behavior I see when I run the block (fails to create the remote file and echoes "bar"), Org does the "wrong thing". I can't account for this. Anyway, Ihor's main point stands: a prompt does not work with non-session shell blocks. The following returns exit code 1 (which means fail): #+begin_src bash :results output read -p "What? " #+end_src As far as I can tell, though, that's not what prevents "bar" from being returned. As far as I can reproduce, calling #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" #+end_src *does* give "bar" for results even though it shouldn't. ** Comment 2. The second claim has nothing to do with Org Babel. I was able to confirm it and provide the steps to reproduce. I think it would make sense to report it upstream and let them decide if it's expected behavior. I'm still happy to do that, but I need to step away from the keyboard :) ^ permalink raw reply [flat|nested] 54+ messages in thread
* Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-17 22:07 ` Matt @ 2023-11-18 3:11 ` Max Nikulin 2023-11-18 8:11 ` Matt 2023-11-18 8:19 ` bash source code block: problem after ssh commands Bruno Barbier 2023-11-18 15:51 ` Matt 2 siblings, 1 reply; 54+ messages in thread From: Max Nikulin @ 2023-11-18 3:11 UTC (permalink / raw) To: emacs-orgmode On 18/11/2023 05:07, Matt wrote: > Consider calling > > bash -c bash /tmp/two-lines.sh Matt, could you, please, describe what this command should do accordingly to your expectations? From my point of view it was a plain mistake in attempts to simulate the issue outside of Emacs. There is no point to concentrate on this command. I tried to explain that it is incorrect usage of "-c" shell option and what is the actual effect of this call, but I seems I failed. Hints: - Try this command in a regular shell prompt outside of Emacs - Try "echo $$" before and after this command - Try it with a script having a clear side effect and check whether it is produced, e.g. "date >>/tmp/touch-tmp-file.txt" ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 3:11 ` Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) Max Nikulin @ 2023-11-18 8:11 ` Matt 2023-11-18 8:29 ` Bruno Barbier 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-18 8:11 UTC (permalink / raw) To: Max Nikulin; +Cc: emacs-orgmode ---- On Sat, 18 Nov 2023 04:11:03 +0100 Max Nikulin wrote --- > > bash -c bash /tmp/two-lines.sh > > From my point of view it was a plain mistake in attempts to simulate > the issue outside of Emacs. There is no point to concentrate on this > command. I tried to explain that it is incorrect usage of "-c" shell > option and what is the actual effect of this call, but I seems I failed. As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs. This is why I've focused on the command. It acts as a guide for what is "correct." Maybe this is misguided for a reason I don't yet see? I concluded that the command is not "what Emacs does" and therefore isn't a valid source of "truth". I agree it's not worth concentrating on it further. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 8:11 ` Matt @ 2023-11-18 8:29 ` Bruno Barbier 2023-11-18 8:43 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Bruno Barbier @ 2023-11-18 8:29 UTC (permalink / raw) To: Matt, Max Nikulin; +Cc: emacs-orgmode Matt <matt@excalamus.com> writes: > ---- On Sat, 18 Nov 2023 04:11:03 +0100 Max Nikulin wrote --- > > > > bash -c bash /tmp/two-lines.sh > > > > From my point of view it was a plain mistake in attempts to simulate > > the issue outside of Emacs. There is no point to concentrate on this > > command. I tried to explain that it is incorrect usage of "-c" shell > > option and what is the actual effect of this call, but I seems I failed. > > As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs. This is why I've focused on the command. It acts as a guide for what is "correct." Maybe this is misguided for a reason I don't yet see? IIUC, what Max is saying is that you should not concentrate on *that specific command* because that command doesn't do what you think it does. To reproduce, I'm personally still using: cat /tmp/test.sh | bash which is, IIUC, what: (process-file "bash" "/tmp/test.sh") is doing, that is: The program’s input comes from file INFILE Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 8:29 ` Bruno Barbier @ 2023-11-18 8:43 ` Matt 2023-11-18 8:54 ` Bruno Barbier 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-18 8:43 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode ---- On Sat, 18 Nov 2023 09:29:56 +0100 Bruno Barbier > IIUC, what Max is saying is that you should not concentrate on > *that specific command* because that command doesn't do what you think > it does. Cool, it sounds like we're agreed (albeit for different reasons). > To reproduce, I'm personally still using: > > cat /tmp/test.sh | bash > > which is, IIUC, what: > > (process-file "bash" "/tmp/test.sh") Yes, agreed. I think that's more like what's happening. What about the ("-c" "bash") passed into process-file? The whole call looks like this: (process-file "bash" "/tmp/two-lines.sh" '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH") nil "-c" "bash") ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 8:43 ` Matt @ 2023-11-18 8:54 ` Bruno Barbier 2023-11-18 9:09 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Bruno Barbier @ 2023-11-18 8:54 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode Matt <matt@excalamus.com> writes: > ---- On Sat, 18 Nov 2023 09:29:56 +0100 Bruno Barbier > > > IIUC, what Max is saying is that you should not concentrate on > > *that specific command* because that command doesn't do what you think > > it does. > > Cool, it sounds like we're agreed (albeit for different reasons). Great :-) > > To reproduce, I'm personally still using: > > > > cat /tmp/test.sh | bash > > > > which is, IIUC, what: > > > > (process-file "bash" "/tmp/test.sh") > > Yes, agreed. I think that's more like what's happening. > > What about the ("-c" "bash") passed into process-file? Useless indirection when the command is "bash" is the same as 'shell-file-name', like in our case ? Maybe ... But, you're right. To be safe, from now on, I'll use: cat /tmp/test.sh | bash -c bash Thanks. Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 8:54 ` Bruno Barbier @ 2023-11-18 9:09 ` Matt 2023-11-18 9:11 ` Bruno Barbier 2023-11-18 10:47 ` Ihor Radchenko 0 siblings, 2 replies; 54+ messages in thread From: Matt @ 2023-11-18 9:09 UTC (permalink / raw) To: Bruno Barbier; +Cc: emacs-orgmode ---- On Sat, 18 Nov 2023 09:54:46 +0100 Bruno Barbier > But, you're right. To be safe, from now on, I'll use: > > cat /tmp/test.sh | bash -c bash It's still not clear to me if this is "what Emacs does". However, that's the best I could come up with. Evaluating the following #+name: /tmp/test.sh #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt #+end_src does exactly what cat /tmp/test.sh | bash -c bash does. Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line. So, I would say that what happens in Org is the "expected" behavior. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 9:09 ` Matt @ 2023-11-18 9:11 ` Bruno Barbier 2023-11-18 10:47 ` Ihor Radchenko 1 sibling, 0 replies; 54+ messages in thread From: Bruno Barbier @ 2023-11-18 9:11 UTC (permalink / raw) To: Matt; +Cc: emacs-orgmode Matt <matt@excalamus.com> writes: > ---- On Sat, 18 Nov 2023 09:54:46 +0100 Bruno Barbier > > It's still not clear to me if this is "what Emacs does". However, that's the best I could come up with. > > Evaluating the following > > #+name: /tmp/test.sh > #+begin_src bash :results output > ssh localhost "echo foo>foo_file" > echo "bar" | tee /tmp/bar.txt > #+end_src > > does exactly what > > cat /tmp/test.sh | bash -c bash > > does. Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line. > > So, I would say that what happens in Org is the "expected" behavior. Agreed. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) 2023-11-18 9:09 ` Matt 2023-11-18 9:11 ` Bruno Barbier @ 2023-11-18 10:47 ` Ihor Radchenko 1 sibling, 0 replies; 54+ messages in thread From: Ihor Radchenko @ 2023-11-18 10:47 UTC (permalink / raw) To: Matt; +Cc: Bruno Barbier, emacs-orgmode Matt <matt@excalamus.com> writes: > Evaluating the following > > #+name: /tmp/test.sh > #+begin_src bash :results output > ssh localhost "echo foo>foo_file" > echo "bar" | tee /tmp/bar.txt > #+end_src > > does exactly what > > cat /tmp/test.sh | bash -c bash > > does. Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line. > > So, I would say that what happens in Org is the "expected" behavior. Yup, it is expected. But only in a sense that "Emacs does what we asked for when calling `process-file'". From the user point of view, it is not expected at all. And, what is worse, may depend on (1) shell used; (2) ssh config (see StdinNull in man 5 ssh_config). I believe that we do need to change how we execute shell blocks _by default_ - to something more predictable, like creating and running a temporary script file. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92> ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 22:07 ` Matt 2023-11-18 3:11 ` Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) Max Nikulin @ 2023-11-18 8:19 ` Bruno Barbier 2023-11-18 9:02 ` Matt 2023-11-18 15:51 ` Matt 2 siblings, 1 reply; 54+ messages in thread From: Bruno Barbier @ 2023-11-18 8:19 UTC (permalink / raw) To: Matt, Ihor Radchenko; +Cc: emacs-orgmode@gnu.org Hi Matt, Thanks this summary and for working on this! Just a few comments/corrections about some specific points, hoping it might help. Matt <matt@excalamus.com> writes: > ---- On Fri, 17 Nov 2023 10:20:28 +0100 Ihor Radchenko wrote --- > > > This has nothing to do with Emacs comint and this is also not a bug in > > Emacs > > Ihor, there were two claims made in the original report. I was referring to Claim 2. That deals with M-x shell and therefore comint-mode. > > Regarding Claim 1: > > - Can anyone verify Claim 1? I do: the file is created and the command "echo bar" is NOT executed. Here is my code block and its results: #+begin_src bash :results output ssh phone "echo foo>foo_file" echo "bar" #+end_src #+RESULTS: No results (the echo command is NOT executed). The file "foo_file" is created on the remote; its content is "foo". #+begin_src bash :results output date ssh -n phone "ls -alh foo_file" ssh -n phone "cat foo_file" #+end_src #+RESULTS: : Sat Nov 18 08:33:59 CET 2023 : -rw------- 1 u0_a256 u0_a256 4 Nov 18 08:26 foo_file : foo > - What versions are people using? > + M-x org-version > + M-x emacs-version #+begin_src elisp (list emacs-version org-version) #+end_src #+RESULTS: | 30.0.50 | 9.7-pre | GNU/Linux gentoo > ... > * Comments about the claims: > ** Comment 1. > ... > I am unable to reproduce the reported behavior (of > "bar" not returning). Instead, I get an ssh-askpass permission denied > error, foo_file is not created, and "bar" is given as the result. I > do not see anywhere in the thread that the original claim was > reproduced. It seems your SSH failed to connect. In that case, I cannot swallow the second command; thus the command "echo bar" is executed. I can reproduce what you see on my side if I force the connection to fail: #+begin_src bash :results output ssh WRONG_REMOTE "echo foo>foo_file" echo "bar" #+end_src #+RESULTS: : bar > > The thread preceded something like follows. > > Leo Butler suggested two work arounds: > > - add the -f to the ssh command > - add a semi-colon and line continuation to the first line. > > Russell Adams suggested another work around: > > - add -n to the ssh command That's the one I use; the option -n is enough for me ('-n' = Redirects stdin from /dev/null). The option '-f' means SSH will go to background; I'm not sure I want that. > ... > ... > He then proposes an experiment to close stdin. To do this, he calls > > #+begin_src shell :results output > exec 0>&- > echo OK > #+end_src > > He claims that "exec 0<&-" closes stdin. I believe there is a typo. > ... You're right. Good catch, thanks! Although it seems to work either way on my side. #+begin_src shell :results output exec 0<&- echo OK #+end_src #+RESULTS: #+begin_src shell :results output exec 0>&- echo OK #+end_src #+RESULTS: > What Bruno writes corresponds to "closing output file descriptor 0". I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor". I had no luck finding this information in man bash or info bash. > My point was: the commands are read the standard input, thus, any command that modifies that standard input will modify what gets executed. > ... > This is what we see in Org. I'll be honest, though, I don't > really know what to expect with exec 0>&- and exec 0<&-. When I call > them in the terminal, it kills the terminal. Let's forget about 'exec 0<&-' (closing the standard input/outputs): this is bringing other corner cases. But, yes, I would expect a terminal to close itself automatically if its input is closed. > ... > As far as I can tell, though, that's not what prevents "bar" from being returned. As far as I can reproduce, calling > > #+begin_src bash :results output > ssh localhost "echo foo>foo_file" > echo "bar" > #+end_src > > *does* give "bar" for results even though it shouldn't. Does it echo bar when the SSH connection succeeds too ? Thanks again for working on this. Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-18 8:19 ` bash source code block: problem after ssh commands Bruno Barbier @ 2023-11-18 9:02 ` Matt 0 siblings, 0 replies; 54+ messages in thread From: Matt @ 2023-11-18 9:02 UTC (permalink / raw) To: Bruno Barbier; +Cc: Ihor Radchenko, emacs-orgmode@gnu.org To clarify a previous typo I made. Everything I've written was also done using: 1. emacs -q 2. C-x b "*scratch*" 3. M-x org-mode 4. Execute #+begin_src emacs-lisp :results none (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src This corresponds to Org mode version 9.6.6 (release_9.6.6 @ /gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/) I had previously, and incorrectly, given my version used with my full init (which uses a different Org version). ---- On Sat, 18 Nov 2023 09:19:13 +0100 Bruno Barbier > It seems your SSH failed to connect. In that case, I cannot swallow the > second command; thus the command "echo bar" is executed. > > I can reproduce what you see on my side if I force the connection to fail: > > #+begin_src bash :results output > ssh WRONG_REMOTE "echo foo>foo_file" > echo "bar" > #+end_src > > #+RESULTS: > : bar Thank you! That makes sense. I forgot that without set -e Bash keeps evaluating when errors happen. Yes, once it connects, I get the same behavior as reported. Here is how I connected without a password prompt: 1. ssh-keygen with id_babel, <enter>, <enter> 2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys 3. chmod og-wx ~/.ssh/authorized_keys 4. ssh-add .ssh/id_babel ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-17 22:07 ` Matt 2023-11-18 3:11 ` Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) Max Nikulin 2023-11-18 8:19 ` bash source code block: problem after ssh commands Bruno Barbier @ 2023-11-18 15:51 ` Matt 2 siblings, 0 replies; 54+ messages in thread From: Matt @ 2023-11-18 15:51 UTC (permalink / raw) To: emacs-orgmode@gnu.org ---- On Fri, 17 Nov 2023 23:07:57 +0100 Matt wrote --- > The second claim has nothing to do with Org Babel. I was able to confirm it and provide the steps to reproduce. I think it would make sense to report it upstream and let them decide if it's expected behavior. I'm still happy to do that, but I need to step away from the keyboard :) Submitted. https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 11:17 bash source code block: problem after ssh commands Alain.Cochard 2023-10-25 15:12 ` Leo Butler @ 2023-10-26 14:44 ` Russell Adams 2023-10-27 11:47 ` Alain.Cochard 2023-11-18 8:09 ` Max Nikulin 2 siblings, 1 reply; 54+ messages in thread From: Russell Adams @ 2023-10-26 14:44 UTC (permalink / raw) To: emacs-orgmode On Wed, Oct 25, 2023 at 01:17:42PM +0200, Alain.Cochard@unistra.fr wrote: > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" > echo "bar" > #+end_src I know that Ihor has already reproduced, but are you using an SSH key to connect, or entering a password? If entering a password, I'd expect there is some mixup. That I defer to Ihor. If you use an SSH key for passwordless access, try adding -n (ie: "ssh -n derp@host mycommand"). The "-n" flag helps prevent the remote command from interfering with your local terminal by redirecting /dev/null as the remote stdin. I have to use this commonly in systems administration, scripts, and tools like Ansible. Thanks. ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com https://www.adamsinfoserv.com/ ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-26 14:44 ` Russell Adams @ 2023-10-27 11:47 ` Alain.Cochard 2023-11-06 18:01 ` Matt 0 siblings, 1 reply; 54+ messages in thread From: Alain.Cochard @ 2023-10-27 11:47 UTC (permalink / raw) To: Russell Adams; +Cc: emacs-orgmode Russell Adams writes on Thu 26 Oct 2023 16:44: > [...] are you using an SSH key to connect, or entering a password? > If you use an SSH key for passwordless access, try adding -n (ie: > "ssh -n derp@host mycommand"). Yes, I use an SSH key, and yes, '-n' saves the day. Thanks. -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-27 11:47 ` Alain.Cochard @ 2023-11-06 18:01 ` Matt 2023-11-07 0:51 ` Alain.Cochard 0 siblings, 1 reply; 54+ messages in thread From: Matt @ 2023-11-06 18:01 UTC (permalink / raw) To: alaincochard; +Cc: emacs-orgmode ---- On Fri, 27 Oct 2023 13:47:24 +0200 Alain.Cochard@unistra.fr wrote --- > Russell Adams writes on Thu 26 Oct 2023 16:44: > > [...] are you using an SSH key to connect, or entering a password? > > If you use an SSH key for passwordless access, try adding -n (ie: > > > "ssh -n derp@host mycommand"). > > Yes, I use an SSH key, and yes, '-n' saves the day. Thanks. Hi Alain, I'm the supposed maintainer of ob-shell who's been missing in action for this whole discussion. I want to confirm, are you able to accomplish your task by using '-n'? Matt ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-06 18:01 ` Matt @ 2023-11-07 0:51 ` Alain.Cochard 0 siblings, 0 replies; 54+ messages in thread From: Alain.Cochard @ 2023-11-07 0:51 UTC (permalink / raw) To: Matt; +Cc: alaincochard, emacs-orgmode Matt writes on Mon 6 Nov 2023 19:01: > I want to confirm, are you able to accomplish your task by using > '-n'? Yes. Evaluating this block (with C-c C-c): #+begin_src bash :results output ssh -n cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src gives #+RESULTS: : bar (without the '-n', #+RESULTS: is empty). Same conclusion if I copy/yank the 2 bash instructions in an emacs terminal (M-x shell). -- EOST (École et Observatoire des Sciences de la Terre) ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr 5 rue René Descartes [bureau 110] | Phone: +33 (0)3 68 85 50 44 F-67084 Strasbourg Cedex, France | [ slot available for rent ] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-10-25 11:17 bash source code block: problem after ssh commands Alain.Cochard 2023-10-25 15:12 ` Leo Butler 2023-10-26 14:44 ` Russell Adams @ 2023-11-18 8:09 ` Max Nikulin 2023-11-18 8:36 ` Bruno Barbier 2 siblings, 1 reply; 54+ messages in thread From: Max Nikulin @ 2023-11-18 8:09 UTC (permalink / raw) To: emacs-orgmode On 25/10/2023 18:17, Alain.Cochard@unistra.fr wrote: > By contrast, it works with this one: > > #+begin_src bash :results output > sshcochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar" > #+end_src What about #+begin_src bash :results output ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar" echo more #+end_src ? ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: bash source code block: problem after ssh commands 2023-11-18 8:09 ` Max Nikulin @ 2023-11-18 8:36 ` Bruno Barbier 0 siblings, 0 replies; 54+ messages in thread From: Bruno Barbier @ 2023-11-18 8:36 UTC (permalink / raw) To: Max Nikulin, emacs-orgmode Max Nikulin <manikulin@gmail.com> writes: > On 25/10/2023 18:17, Alain.Cochard@unistra.fr wrote: >> By contrast, it works with this one: >> >> #+begin_src bash :results output >> sshcochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar" >> #+end_src > > What about > > #+begin_src bash :results output > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar" > echo more > #+end_src > > ? For me: #+begin_src bash :results output ssh phone "echo foo>foo_file" ; echo "bar" echo more #+end_src #+RESULTS: : bar And, telling SSH to not swallow the remaining commands (option '-n'): #+begin_src bash :results output ssh -n phone "echo foo>foo_file" ; echo "bar" echo more #+end_src #+RESULTS: : bar : more Bruno ^ permalink raw reply [flat|nested] 54+ messages in thread
end of thread, other threads:[~2023-11-22 17:07 UTC | newest] Thread overview: 54+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-10-25 11:17 bash source code block: problem after ssh commands Alain.Cochard 2023-10-25 15:12 ` Leo Butler 2023-10-25 16:14 ` Alain.Cochard 2023-10-25 16:47 ` Leo Butler 2023-10-25 16:59 ` yaxp 2023-10-26 8:44 ` Ihor Radchenko 2023-10-26 13:23 ` Alain.Cochard 2023-10-26 13:44 ` Ihor Radchenko 2023-10-27 18:26 ` Alain.Cochard 2023-10-28 5:22 ` Max Nikulin 2023-10-30 10:50 ` Bruno Barbier 2023-11-06 13:32 ` Ihor Radchenko 2023-11-06 18:25 ` Matt 2023-11-07 8:55 ` Ihor Radchenko 2023-11-08 19:41 ` Matt 2023-11-09 12:14 ` Ihor Radchenko 2023-11-09 17:48 ` Matt 2023-11-15 16:32 ` Matt 2023-11-15 18:04 ` Matt 2023-11-16 9:32 ` Ihor Radchenko 2023-11-16 19:03 ` Matt 2023-11-16 19:46 ` Alain.Cochard 2023-11-16 20:54 ` Matt 2023-11-17 9:22 ` Ihor Radchenko 2023-11-17 9:55 ` Alain.Cochard 2023-11-17 10:17 ` Ihor Radchenko 2023-11-17 15:32 ` Leo Butler 2023-11-17 15:47 ` Bruno Barbier 2023-11-18 10:37 ` Ihor Radchenko 2023-11-21 19:01 ` Bruno Barbier 2023-11-22 17:06 ` Max Nikulin 2023-11-19 4:17 ` Non-emacs shell (Re: bash source code block: problem after ssh commands) Max Nikulin 2023-11-21 15:33 ` Bruno Barbier 2023-11-18 8:04 ` bash source code block: problem after ssh commands Max Nikulin 2023-11-18 10:43 ` Ihor Radchenko 2023-11-18 16:18 ` Max Nikulin 2023-11-17 22:07 ` Matt 2023-11-18 3:11 ` Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands) Max Nikulin 2023-11-18 8:11 ` Matt 2023-11-18 8:29 ` Bruno Barbier 2023-11-18 8:43 ` Matt 2023-11-18 8:54 ` Bruno Barbier 2023-11-18 9:09 ` Matt 2023-11-18 9:11 ` Bruno Barbier 2023-11-18 10:47 ` Ihor Radchenko 2023-11-18 8:19 ` bash source code block: problem after ssh commands Bruno Barbier 2023-11-18 9:02 ` Matt 2023-11-18 15:51 ` Matt 2023-10-26 14:44 ` Russell Adams 2023-10-27 11:47 ` Alain.Cochard 2023-11-06 18:01 ` Matt 2023-11-07 0:51 ` Alain.Cochard 2023-11-18 8:09 ` Max Nikulin 2023-11-18 8:36 ` Bruno Barbier
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).