Redirecting stderr to stdout with babel/shell

Redirecting stderr to stdout with babel/shell

[Karl Voit]

Hi!

I enjoyed [1] where John is describing the issue of vanishing
stderr-messages through babel.

This could be fixed by "(setq org-babel-python-command "python -i -c
\"import sys; sys.stderr = sys.stdout\"")".

Is there a similar method to re-direct the stderr of a shell script
to stdout as well?

The issue is demonstrated below:

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
echo "testing stderr with manual redirect" 2>&1 >&2
#+END_SRC

#+RESULTS:
: testing stdout
: testing stderr with manual redirect

I want to skip the manual redirect after each command.

Thanks for any idea!


[1] http://kitchingroup.cheme.cmu.edu/blog/2014/12/21/Capturing-stderr-from-Python-in-org-mode-take-2/

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Redirecting stderr to stdout with babel/shell

[Michael Brand]

Hi Karl

On Thu, Jan 1, 2015 at 5:27 PM, Karl Voit <devnull@karl-voit.at> wrote:
> Is there a similar method to re-direct the stderr of a shell script
> to stdout as well?

Not that I know of an other way than what you already demonstrated.

Once I wrote a lengthy post with my opinion about stdout, stderr and
the exit status in babel/shell. It contains a complete set of the
different cases as I see them:
http://thread.gmane.org/gmane.emacs.orgmode/45828/focus=46415

Michael

Re: Redirecting stderr to stdout with babel/shell

[Samuel Wales]

hi karl,

i always wrap as follows and it works for me.

{
  your code
} 2>&1
:

the : eliminates fancy error handling.  i find that much less confusing.


samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Redirecting stderr to stdout with babel/shell

[John Kitchin]

Check out this solution:

http://kitchingroup.cheme.cmu.edu/blog/2015/01/04/Redirecting-stderr-in-org-mode-shell-blocks/

I think it does what you want, and uses the idea below.

Samuel Wales <samologist@gmail.com> writes:

> hi karl,
>
> i always wrap as follows and it works for me.
>
> {
>   your code
> } 2>&1
> :
>
> the : eliminates fancy error handling.  i find that much less confusing.
>
>
> samuel

-- 
-----------------------------------
John Kitchin
Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu

Re: Redirecting stderr to stdout with babel/shell

[Karl Voit]

* John Kitchin <johnrkitchin@gmail.com> wrote:
> Check out this solution:
>
> http://kitchingroup.cheme.cmu.edu/blog/2015/01/04/Redirecting-stderr-in-org-mode-shell-blocks/

Hm. This does not work on my machine: Debian Wheezy GNU/Linux

Testing the current satus (again):

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
#+END_SRC

#+RESULTS:

... no stderr on stdout and date error message in second buffer


Creating a wrapper-script similar as described in web-page above:

#+BEGIN_SRC sh
echo '#!/usr/bin/zsh
{
/usr/bin/zsh $1
} 2>&1' > ~/src/misc/zsh_stderr_redirected_to_stdout.sh
chmod +x ~/src/misc/zsh_stderr_redirected_to_stdout.sh
cd ~/bin
ln -s ../src/misc/zsh_stderr_redirected_to_stdout.sh .
#+END_SRC

#+RESULTS:

Setting the sh-command to this wrapper-script:

#+BEGIN_SRC elisp
(setq org-babel-sh-command "~/bin/zsh_stderr_redirected_to_stdout.sh")
#+END_SRC

#+RESULTS:
: ~/bin/zsh_stderr_redirected_to_stdout.sh

Re-testing status:

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
#+END_SRC

#+RESULTS:

... no change except the second buffer for the date error message is
empty. So it's actually worse than before.


Trying with bash (as web-page did):

#+BEGIN_SRC sh
echo '#!/bin/bash
{
/bin/bash $1
} 2>&1' > ~/src/misc/bash_stderr_redirected_to_stdout.sh
chmod +x ~/src/misc/bash_stderr_redirected_to_stdout.sh
cd ~/bin
ln -s ../src/misc/bash_stderr_redirected_to_stdout.sh .
#+END_SRC

#+RESULTS:

Setting the sh-command to this wrapper-script:

#+BEGIN_SRC elisp
(setq org-babel-sh-command "~/bin/bash_stderr_redirected_to_stdout.sh")
#+END_SRC

#+RESULTS:
: ~/bin/bash_stderr_redirected_to_stdout.sh

Same result as with zsh :-(


Re-setting to standard settings to revoke tests from above:

#+BEGIN_SRC elisp
(setq org-babel-sh-command "sh")
#+END_SRC

#+RESULTS:
: sh

Re-testing status:

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
#+END_SRC

#+RESULTS:


-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Redirecting stderr to stdout with babel/shell

[John Kitchin]

Karl Voit <devnull@Karl-Voit.at> writes:

Weird. It works for me on MacOSX with bash. Out of curiosity, did you
try the source: http://kitchingroup.cheme.cmu.edu/org/2015/01/04/Redirecting-stderr-in-org-mode-shell-blocks.org?

I updated the web page with Achim's  simpler solution.

> * John Kitchin <johnrkitchin@gmail.com> wrote:
>> Check out this solution:
>>
>> http://kitchingroup.cheme.cmu.edu/blog/2015/01/04/Redirecting-stderr-in-org-mode-shell-blocks/
>
> Hm. This does not work on my machine: Debian Wheezy GNU/Linux
>
> Testing the current satus (again):
>
> #+BEGIN_SRC sh :results output
> echo "testing stdout" >&1
> echo "testing stderr" >&2
> date -g
> #+END_SRC
>
> #+RESULTS:
>
> ... no stderr on stdout and date error message in second buffer
>
>
> Creating a wrapper-script similar as described in web-page above:
>
> #+BEGIN_SRC sh
> echo '#!/usr/bin/zsh
> {
> /usr/bin/zsh $1
> } 2>&1' > ~/src/misc/zsh_stderr_redirected_to_stdout.sh
> chmod +x ~/src/misc/zsh_stderr_redirected_to_stdout.sh
> cd ~/bin
> ln -s ../src/misc/zsh_stderr_redirected_to_stdout.sh .
> #+END_SRC
>
> #+RESULTS:
>
> Setting the sh-command to this wrapper-script:
>
> #+BEGIN_SRC elisp
> (setq org-babel-sh-command "~/bin/zsh_stderr_redirected_to_stdout.sh")
> #+END_SRC
>
> #+RESULTS:
> : ~/bin/zsh_stderr_redirected_to_stdout.sh
>
> Re-testing status:
>
> #+BEGIN_SRC sh :results output
> echo "testing stdout" >&1
> echo "testing stderr" >&2
> date -g
> #+END_SRC
>
> #+RESULTS:
>
> ... no change except the second buffer for the date error message is
> empty. So it's actually worse than before.
>
>
> Trying with bash (as web-page did):
>
> #+BEGIN_SRC sh
> echo '#!/bin/bash
> {
> /bin/bash $1
> } 2>&1' > ~/src/misc/bash_stderr_redirected_to_stdout.sh
> chmod +x ~/src/misc/bash_stderr_redirected_to_stdout.sh
> cd ~/bin
> ln -s ../src/misc/bash_stderr_redirected_to_stdout.sh .
> #+END_SRC
>
> #+RESULTS:
>
> Setting the sh-command to this wrapper-script:
>
> #+BEGIN_SRC elisp
> (setq org-babel-sh-command "~/bin/bash_stderr_redirected_to_stdout.sh")
> #+END_SRC
>
> #+RESULTS:
> : ~/bin/bash_stderr_redirected_to_stdout.sh
>
> Same result as with zsh :-(
>
>
> Re-setting to standard settings to revoke tests from above:
>
> #+BEGIN_SRC elisp
> (setq org-babel-sh-command "sh")
> #+END_SRC
>
> #+RESULTS:
> : sh
>
> Re-testing status:
>
> #+BEGIN_SRC sh :results output
> echo "testing stdout" >&1
> echo "testing stderr" >&2
> date -g
> #+END_SRC
>
> #+RESULTS:

--
-----------------------------------
John Kitchin
Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu

Re: Redirecting stderr to stdout with babel/shell

[Karl Voit]

* John Kitchin <johnrkitchin@gmail.com> wrote:
> Karl Voit <devnull@Karl-Voit.at> writes:
>
> Weird. It works for me on MacOSX with bash. Out of curiosity, did you
> try the source: http://kitchingroup.cheme.cmu.edu/org/2015/01/04/Redirecting-stderr-in-org-mode-shell-blocks.org?

I did it now.

What I found out:

#+BEGIN_SRC sh
echo '#!/usr/bin/zsh
{
/usr/bin/zsh $1
} 2>&1
#end' > ~/src/misc/zsh_stderr_redirected_to_stdout.sh
chmod +x ~/src/misc/zsh_stderr_redirected_to_stdout.sh
cd ~/bin
ln -s ../src/misc/zsh_stderr_redirected_to_stdout.sh .
#+END_SRC

#+BEGIN_SRC elisp
(setq org-babel-sh-command "~/bin/zsh_stderr_redirected_to_stdout.sh")
#+END_SRC

... my old script from my previous posting:

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
#+END_SRC

#+RESULTS:

... with empty additional buffer window. So not happy.

However with an additional "echo" at the end:

#+BEGIN_SRC sh :results output
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
echo
#+END_SRC

#+RESULTS:
: testing stdout
: testing stderr
: date: invalid option -- 'g'
: Try `date --help' for more information.
: 

... it works. :-O

Is there an issue with flushing stdout or something?

What is the explanation and the general rule?


Btw, the trick with tangling for writing the script file did not work on my
machine.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Redirecting stderr to stdout with babel/shell

[Achim Gratz]

Karl Voit writes:
> However with an additional "echo" at the end:

You need to understand what you're doing or at least copy the code
exactly.  The last line in my example is a colon ":" so that the shell
exit code is always zero.  If not, Babel will ignore the output since it
interprets any non-zero exit code as failure.  Since you've redirected
STDERR to STDOUT the error buffer that would normally let you know what
happened stays empty.  The echo has the same effect, but produces a
blank line you don't want.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Redirecting stderr to stdout with babel/shell

[Karl Voit]

* Achim Gratz <Stromeko@nexgo.de> wrote:
> Karl Voit writes:
>> However with an additional "echo" at the end:
>
> You need to understand what you're doing or at least copy the code
> exactly.  The last line in my example is a colon ":" so that the shell
> exit code is always zero.  

You're right. The only thing I did not know was the meaning of the
colon.

http://stackoverflow.com/a/3224910 did fix my ignorance :-)

> If not, Babel will ignore the output since it interprets any
> non-zero exit code as failure.  Since you've redirected STDERR to
> STDOUT the error buffer that would normally let you know what
> happened stays empty.  The echo has the same effect, but produces
> a blank line you don't want.

Yes, now everything makes sense to me. Thanks for your pointer!

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Redirecting stderr to stdout with babel/shell

[Achim Gratz]

Karl Voit writes:
> echo "testing stderr with manual redirect" 2>&1 >&2

The last redirection ">&2" is nonsense, it only works because STDERR is
already reopened on STDOUT and redirection to the same file descriptor
is ignored.

And to solve your original problem:

#+BEGIN_SRC sh :results output
exec 2>&1
echo "testing stdout" >&1
echo "testing stderr" >&2
date -g
:
#+END_SRC


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Redirecting stderr to stdout with babel/shell

[Karl Voit]

* Achim Gratz <Stromeko@nexgo.de> wrote:
> Karl Voit writes:
>> echo "testing stderr with manual redirect" 2>&1 >&2
>
> The last redirection ">&2" is nonsense, it only works because STDERR is
> already reopened on STDOUT and redirection to the same file descriptor
> is ignored.

Absolutely right. Must have been a non-deleted artefact after I
tested something different in that line.

> And to solve your original problem:
>
> #+BEGIN_SRC sh :results output
> exec 2>&1
> echo "testing stdout" >&1
> echo "testing stderr" >&2
> date -g
> :
> #+END_SRC

Together with the '{ ... } 2>&1' trick, this is a valid workaround.
What I wanted to achieve is a re-direct of any (sh) babel script
without this exec-command I do have to add.

However, I could imagine an additional babel-parameter like ":stderr
redirect" (or something more meaningful) because in most cases I
have to add the ":results output" line as well.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github