[babel] how to pass data to gnuplot from another block

[babel] how to pass data to gnuplot from another block

[Eric S Fraga]

Hello,

everything I have seen on the list and on Worg seems to indicate that it
should be possible to pass data from a babel src block to a gnuplot src
block.  See, for instance, the excellent article on data collection and
analysis at

http://orgmode.org/worg/org-contrib/babel/examples/data-collection-analysis.html

I don't have R so I have not tried the examples on this page.  However,
I have tried the following using octave and gnuplot:

#+begin_src org
  ,#+TITLE:     examplebug.org
  ,#+AUTHOR:    Eric S Fraga
  ,* bubble and dew points
  ,#+name: bubble-point-temperatures
  ,#+begin_src octave :results raw output
    A = [   6.86880    6.84941]';
    B = [1154.646   1206.001]';
    C = [ 226.046    223.148]';
    pv = @(T) 10.^(A-B./(T+C));
    x = 0;
    res = [];
    while x<(1+1e-3)
      f = @(T) 760 - [x (1-x)]*pv(T);
      T = fsolve(f, 70);
      printf("| %5.2f | %7.2f | \n", x, T);
      res = [res; [x T] ];
      x = x + 0.05;
    endwhile
  ,#+end_src
  
  ,#+begin_src gnuplot :var data=bubble-point-temperatures :file "bubble.pdf"
    set terminal pdfcairo mono enhanced font ",12" size 10cm,6cm
    plot data
  ,#+end_src
  
  ,#+results:
  [[file:bubble.pdf]]
#+end_src
  
Trying to execute the gnuplot block leads to an error in gnuplot because
the output of the octave data block is being put directly into the data
variable instead of into a file whose name is passed to the data
variable.  The latter happens if the source of the data is a table
instead of a src block.

I have tried various combinations of output directives for the first src
block but to no avail.

I have tried this with "emacs -Q", by the way, and my org is up to date
as of a few minutes ago!

What am I doing wrong please?

Thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.3.50.1, Org release_8.2.3c-266-g7a726d

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 80 bytes --]

Hi Eric,

The attached works fine for me (using sh since I don't have octave).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: sh-to-gnuplot.org --]
[-- Type: text/x-org, Size: 372 bytes --]


#+name: uptime
#+begin_src sh
  paste <(echo -e "1\n5\n15") <(uptime|sed 's/^.*average: //;s/,//g'|tr ' ' '\n')
#+end_src

#+RESULTS: uptime
|  1 | 0.02 |
|  5 | 0.06 |
| 15 | 0.05 |

#+begin_src gnuplot :var data=uptime :results silent
  set xrange [0:]
  set yrange [0:]
  set title "uptime"
  set xlabel "minutes ago"
  set ylabel "load"
  plot data w lines
#+end_src

[-- Attachment #3: Type: text/plain, Size: 2268 bytes --]


Ensure that the data you're passing into gnuplot is a table and not a
string.  Gnuplot blocks handle tables by writing them to a file, and
then replacing the variable with the file name.  As I recall gnuplot
blocks assume string data already is a file name, so the variable is
replaced directly.

Best,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hello,
>
> everything I have seen on the list and on Worg seems to indicate that it
> should be possible to pass data from a babel src block to a gnuplot src
> block.  See, for instance, the excellent article on data collection and
> analysis at
>
> http://orgmode.org/worg/org-contrib/babel/examples/data-collection-analysis.html
>
> I don't have R so I have not tried the examples on this page.  However,
> I have tried the following using octave and gnuplot:
>
> #+begin_src org
>   ,#+TITLE:     examplebug.org
>   ,#+AUTHOR:    Eric S Fraga
>   ,* bubble and dew points
>   ,#+name: bubble-point-temperatures
>   ,#+begin_src octave :results raw output
>     A = [   6.86880    6.84941]';
>     B = [1154.646   1206.001]';
>     C = [ 226.046    223.148]';
>     pv = @(T) 10.^(A-B./(T+C));
>     x = 0;
>     res = [];
>     while x<(1+1e-3)
>       f = @(T) 760 - [x (1-x)]*pv(T);
>       T = fsolve(f, 70);
>       printf("| %5.2f | %7.2f | \n", x, T);
>       res = [res; [x T] ];
>       x = x + 0.05;
>     endwhile
>   ,#+end_src
>   
>   ,#+begin_src gnuplot :var data=bubble-point-temperatures :file "bubble.pdf"
>     set terminal pdfcairo mono enhanced font ",12" size 10cm,6cm
>     plot data
>   ,#+end_src
>   
>   ,#+results:
>   [[file:bubble.pdf]]
> #+end_src
>   
> Trying to execute the gnuplot block leads to an error in gnuplot because
> the output of the octave data block is being put directly into the data
> variable instead of into a file whose name is passed to the data
> variable.  The latter happens if the source of the data is a table
> instead of a src block.
>
> I have tried various combinations of output directives for the first src
> block but to no avail.
>
> I have tried this with "emacs -Q", by the way, and my org is up to date
> as of a few minutes ago!
>
> What am I doing wrong please?
>
> Thanks,
> eric

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Nick Dokos]

Eric Schulte <schulte.eric@gmail.com> writes:


> The attached works fine for me (using sh since I don't have octave).
>
> #+name: uptime
> #+begin_src sh
>   paste <(echo -e "1\n5\n15") <(uptime|sed 's/^.*average: //;s/,//g'|tr ' ' '\n')
> #+end_src
>

Just an fyi: I had to set org-babel-sh-command to "bash" for this to
work. Why is "sh" the default value of this variable?

> #+RESULTS: uptime
> |  1 | 0.02 |
> |  5 | 0.06 |
> | 15 | 0.05 |
>
> #+begin_src gnuplot :var data=uptime :results silent
>   set xrange [0:]
>   set yrange [0:]
>   set title "uptime"
>   set xlabel "minutes ago"
>   set ylabel "load"
>   plot data w lines
> #+end_src
>
> Ensure that the data you're passing into gnuplot is a table and not a
> string.  Gnuplot blocks handle tables by writing them to a file, and
> then replacing the variable with the file name.  As I recall gnuplot
> blocks assume string data already is a file name, so the variable is
> replaced directly.
>

Ah, that explains everything! I also didn't have octave on this machine
so I wrote a python block. Initially, I had

--8<---------------cut here---------------start------------->8---
#+name: foo
#+begin_src python
  x = ((1, 1), (2, 4), (3, 9))
  return "\n".join(["|%d | %d |" % (y[0], y[1]) for y in x])
#+end_src


#+RESULTS: foo
| 1 | 1 |
| 2 | 4 |
| 3 | 9 |
--8<---------------cut here---------------end--------------->8---

which looks like a table, but isn't: the gnuplot block was blowing
up just like Eric F's. I replaced it with

--8<---------------cut here---------------start------------->8---
#+name: foo
#+begin_src python
  x = ((1, 1), (2, 4), (3, 9))
  return x
#+end_src


#+RESULTS: foo
| 1 | 1 |
| 2 | 4 |
| 3 | 9 |
--8<---------------cut here---------------end--------------->8---

and everything is working. The only problem is that the results
*look* the same, so it's hard to see what the type is.

Nick

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

Nick Dokos <ndokos@gmail.com> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>
>> The attached works fine for me (using sh since I don't have octave).
>>
>> #+name: uptime
>> #+begin_src sh
>>   paste <(echo -e "1\n5\n15") <(uptime|sed 's/^.*average: //;s/,//g'|tr ' ' '\n')
>> #+end_src
>>
>
> Just an fyi: I had to set org-babel-sh-command to "bash" for this to
> work. Why is "sh" the default value of this variable?
>

I think sh is more portable, but I guess almost any system should have
bash as well, I've just changed this default to bash.

Cheers,

>
>> #+RESULTS: uptime
>> |  1 | 0.02 |
>> |  5 | 0.06 |
>> | 15 | 0.05 |
>>
>> #+begin_src gnuplot :var data=uptime :results silent
>>   set xrange [0:]
>>   set yrange [0:]
>>   set title "uptime"
>>   set xlabel "minutes ago"
>>   set ylabel "load"
>>   plot data w lines
>> #+end_src
>>
>> Ensure that the data you're passing into gnuplot is a table and not a
>> string.  Gnuplot blocks handle tables by writing them to a file, and
>> then replacing the variable with the file name.  As I recall gnuplot
>> blocks assume string data already is a file name, so the variable is
>> replaced directly.
>>
>
> Ah, that explains everything! I also didn't have octave on this machine
> so I wrote a python block. Initially, I had
>
> --8<---------------cut here---------------start------------->8---
> #+name: foo
> #+begin_src python
>   x = ((1, 1), (2, 4), (3, 9))
>   return "\n".join(["|%d | %d |" % (y[0], y[1]) for y in x])
> #+end_src
>
>
> #+RESULTS: foo
> | 1 | 1 |
> | 2 | 4 |
> | 3 | 9 |
> --8<---------------cut here---------------end--------------->8---
>
> which looks like a table, but isn't: the gnuplot block was blowing
> up just like Eric F's. I replaced it with
>
> --8<---------------cut here---------------start------------->8---
> #+name: foo
> #+begin_src python
>   x = ((1, 1), (2, 4), (3, 9))
>   return x
> #+end_src
>
>
> #+RESULTS: foo
> | 1 | 1 |
> | 2 | 4 |
> | 3 | 9 |
> --8<---------------cut here---------------end--------------->8---
>
> and everything is working. The only problem is that the results
> *look* the same, so it's hard to see what the type is.
>
> Nick
>
>
>
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


Eric Schulte <schulte.eric@gmail.com> writes:

>> Just an fyi: I had to set org-babel-sh-command to "bash" for this to
>> work. Why is "sh" the default value of this variable?
>>
>
> I think sh is more portable, but I guess almost any system should have
> bash as well, I've just changed this default to bash.

(Assuming you mean that you changed the default in the org sources, not
in your config files.)

Please don't, at least without discussion of the consequences of adding
a dependency that is beyond POSIX..  sh is specified by posix, and bash
is a) sometimes not present and b) behaves differently than as specified
by POSIX, leading people to write nonportable code.

If someone wants to run bash explicitly, it makes sense to have a bash
language block that they can use.  But the sh language block should be
sh.   The real point is that bash is a different language.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

Greg Troxel <gdt@ir.bbn.com> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>>> Just an fyi: I had to set org-babel-sh-command to "bash" for this to
>>> work. Why is "sh" the default value of this variable?
>>>
>>
>> I think sh is more portable, but I guess almost any system should have
>> bash as well, I've just changed this default to bash.
>
> (Assuming you mean that you changed the default in the org sources, not
> in your config files.)
>
> Please don't, at least without discussion of the consequences of adding
> a dependency that is beyond POSIX..  sh is specified by posix, and bash
> is a) sometimes not present and b) behaves differently than as specified
> by POSIX, leading people to write nonportable code.
>
> If someone wants to run bash explicitly, it makes sense to have a bash
> language block that they can use.  But the sh language block should be
> sh.   The real point is that bash is a different language.

I understand your point, but in reality I doubt there are many systems
on which people use Org-mode with code blocks and on which sh is
available but no bash is installed.

Bash is the new normal shell and I would argue is what most users expect
from a shell code block.  E.g., the default value of `shell-file-name'
used by M-x shell is "/bin/bash".

It is possible to explicitly set shell code blocks to use sh.

Best,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 3523 bytes --]


Eric Schulte <schulte.eric@gmail.com> writes:

> Greg Troxel <gdt@ir.bbn.com> writes:
>
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>>> Just an fyi: I had to set org-babel-sh-command to "bash" for this to
>>>> work. Why is "sh" the default value of this variable?
>>>
>>> I think sh is more portable, but I guess almost any system should have
>>> bash as well, I've just changed this default to bash.
>>
>> (Assuming you mean that you changed the default in the org sources, not
>> in your config files.)
>>
>> Please don't, at least without discussion of the consequences of adding
>> a dependency that is beyond POSIX..  sh is specified by posix, and bash
>> is a) sometimes not present and b) behaves differently than as specified
>> by POSIX, leading people to write nonportable code.
>>
>> If someone wants to run bash explicitly, it makes sense to have a bash
>> language block that they can use.  But the sh language block should be
>> sh.   The real point is that bash is a different language.
>
> I understand your point, but in reality I doubt there are many systems
> on which people use Org-mode with code blocks and on which sh is
> available but no bash is installed.

That may be true on some flavors of Linux, but on BSDs:

  bash is not the normal shell (and is not part of the base system, at
  least on NetBSD, and I think that's still true on the others).  When
  it does exist it's not in /bin.
  
  It's not so odd to have a system without bash.

I am also under the impression that Debian does not use bash as the
/bin/sh.

org, like anything else, should be OS-agnostic, and follow open
standards whenever that's at all reasonable.

> Bash is the new normal shell and I would argue is what most users expect
> from a shell code block.

I find that pretty astounding.  In a block labeled sh it is obvious that
a shell conforming to the POSIX sh standard is expected, and it's not so
different from a file with "#!/bin/sh".  Users who expect bash in a
block labeled sh are wrong, although I agree that many people have been
misled this way by the culture of using "test ==" in a file that starts
#!/bin/sh.

The real issue is that org files that actually expect bash (test ==,
etc.)  become nonportable to other environments.  If someone is writing
a script and not intending to use beyond-posix features, it's harmful to
let them work (in cases where they are published).

>  E.g., the default value of `shell-file-name' used by M-x shell is
> "/bin/bash".

I just checked on my system (NetBSD 6 i386, emacs 23.4.1), and
shell-file-name is documented to inherit from SHELL if present, which it
does.  It's /bin/sh if SHELL is unset, which complies with the
documentation:

  *File name to load inferior shells from.
  Initialized from the SHELL environment variable, or to a system-dependent
  default if SHELL is not set.

which doesn't promise bash (or even a Bourne-style shell!).  (The emacs
package doesn't depend on the bash package.)  But shell-file-name is
about giving the user of emacs their shell, not using a particular
programming language, so this fuzz is fine.

> It is possible to explicitly set shell code blocks to use sh.

Sure, but that wasn't my point; it's the encouragement of nonportability
that is problematic.

I should point out that I'm not a bash hater --- I actually use it as my
interactive shell, and have done so since around 1990.  But I don't
write scripts in it.

Greg

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

>>
>> I understand your point, but in reality I doubt there are many systems
>> on which people use Org-mode with code blocks and on which sh is
>> available but no bash is installed.
>
> That may be true on some flavors of Linux, but on BSDs:
>
>   bash is not the normal shell (and is not part of the base system, at
>   least on NetBSD, and I think that's still true on the others).  When
>   it does exist it's not in /bin.
>   
>   It's not so odd to have a system without bash.
>
> I am also under the impression that Debian does not use bash as the
> /bin/sh.
>
> org, like anything else, should be OS-agnostic, and follow open
> standards whenever that's at all reasonable.
>
>> Bash is the new normal shell and I would argue is what most users expect
>> from a shell code block.
>
> I find that pretty astounding.  In a block labeled sh it is obvious that
> a shell conforming to the POSIX sh standard is expected, and it's not so
> different from a file with "#!/bin/sh".  Users who expect bash in a
> block labeled sh are wrong, although I agree that many people have been
> misled this way by the culture of using "test ==" in a file that starts
> #!/bin/sh.
>
> The real issue is that org files that actually expect bash (test ==,
> etc.)  become nonportable to other environments.  If someone is writing
> a script and not intending to use beyond-posix features, it's harmful to
> let them work (in cases where they are published).
>

Points well made, I was not aware of the BSD default.

Although purely semantically, in my opinion the "sh" in "#+begin_src sh"
indicates generic "shell-script", not the POSIX sh.  E.g., there is no
ob-bash.el or ob-csh.el.

See the first line in ob-sh.el,

,----
| ;;; ob-sh.el --- org-babel functions for shell evaluation
`----

>
>>  E.g., the default value of `shell-file-name' used by M-x shell is
>> "/bin/bash".
>
> I just checked on my system (NetBSD 6 i386, emacs 23.4.1), and
> shell-file-name is documented to inherit from SHELL if present, which it
> does.  It's /bin/sh if SHELL is unset, which complies with the
> documentation:
>
>   *File name to load inferior shells from.
>   Initialized from the SHELL environment variable, or to a system-dependent
>   default if SHELL is not set.
>
> which doesn't promise bash (or even a Bourne-style shell!).  (The emacs
> package doesn't depend on the bash package.)  But shell-file-name is
> about giving the user of emacs their shell, not using a particular
> programming language, so this fuzz is fine.
>

And this is where we disagree.  Sh code blocks don't currently promise
POSIX sh, they promise a shell.  This is certainly a much more dangerous
generalization than say Perl code blocks possibly using Perl 5 or 6.

How about the following resolution?  We rename ob-sh.el to ob-shell.el.
New "shell" code blocks could use the value of the
`org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
ash, dash (am I missing any other common ones) use the specific shell
specified.

What do you think?

In the mean time, I don't believe the change in default value for
`org-babel-sh-command' has been included in any maintenance releases, so
I've just reverted this to minimize any further confusion.

>
>> It is possible to explicitly set shell code blocks to use sh.
>
> Sure, but that wasn't my point; it's the encouragement of nonportability
> that is problematic.
>
> I should point out that I'm not a bash hater --- I actually use it as my
> interactive shell, and have done so since around 1990.  But I don't
> write scripts in it.
>

And I should say that I've argued the same point your making myself in
the past (on a project making the much more serious error of using bash
notation ">&" in a shell script starting with "#!/bin/sh").  I think we
only disagree on the current meaning of "sh" in code blocks, which
hopefully the suggestion above will rectify.

Best,

>
> Greg

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

>
> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
> New "shell" code blocks could use the value of the
> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
> ash, dash (am I missing any other common ones) use the specific shell
> specified.
>

The attached patches make this change and continue to pass the entire
test suite.  The problem being that with ob-sh, no longer present many
users may have to change their configuration and possible the value of
their local.mk file.  One solution there is to add a dummy ob-sh.el with
a deprecation message for some transition time.  Thoughts?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-multiple-code-block-types-for-multiple-shells.patch --]
[-- Type: text/x-diff, Size: 2054 bytes --]

From 7a6c0e35415c4a173d101336029262f3a09abb91 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 13 Dec 2013 09:52:05 -0700
Subject: [PATCH 1/4] multiple code block types for multiple shells

* lisp/ob-sh.el (org-babel-sh-command): Now set from
  `shell-file-name'.
  (org-babel-shell-names): List of specific shells known to Org mode
  shell code blocks.
  (org-babel-execute:shell): New generic shell execution function.
---
 lisp/ob-sh.el | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/lisp/ob-sh.el b/lisp/ob-sh.el
index 4984ff9..a115f38 100644
--- a/lisp/ob-sh.el
+++ b/lisp/ob-sh.el
@@ -38,9 +38,12 @@
 
 (defvar org-babel-default-header-args:sh '())
 
-(defvar org-babel-sh-command "sh"
+(defcustom org-babel-sh-command shell-file-name
   "Command used to invoke a shell.
-This will be passed to  `shell-command-on-region'")
+Set by default to the value of `shell-file-name'.  This will be
+passed to `shell-command-on-region'"
+  :group 'org-babel
+  :type 'string)
 
 (defcustom org-babel-sh-var-quote-fmt
   "$(cat <<'BABEL_TABLE'\n%s\nBABEL_TABLE\n)"
@@ -48,7 +51,22 @@ This will be passed to  `shell-command-on-region'")
   :group 'org-babel
   :type 'string)
 
-(defun org-babel-execute:sh (body params)
+(defcustom org-babel-shell-names '("sh" "bash" "csh" "ash" "dash")
+  "List of names of shell supported by babel shell code blocks."
+  :group 'org-babel
+  :type 'string
+  :initialize
+  (lambda (symbol value)
+    (set-default symbol (second value))
+    (mapc
+     (lambda (name)
+       (eval `(defun ,(intern (concat "org-babel-execute:" name)) (body params)
+		,(format "Execute a block of %s commands with Babel." name)
+		(let ((org-babel-sh-command ,name))
+		  (org-babel-execute:shell body params)))))
+     (second value))))
+
+(defun org-babel-execute:shell (body params)
   "Execute a block of Shell commands with Babel.
 This function is called by `org-babel-execute-src-block'."
   (let* ((session (org-babel-sh-initiate-session
-- 
1.8.5.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-rename-ob-sh-to-ob-shell.patch --]
[-- Type: text/x-diff, Size: 20447 bytes --]

From 583e7ab1757f54a656acb52ef60c6069d060cbe1 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 13 Dec 2013 09:54:28 -0700
Subject: [PATCH 2/4] rename ob-sh to ob-shell

---
 lisp/ob-sh.el              | 230 ---------------------------------------------
 lisp/ob-shell.el           | 230 +++++++++++++++++++++++++++++++++++++++++++++
 testing/lisp/test-ob-sh.el |   2 +-
 3 files changed, 231 insertions(+), 231 deletions(-)
 delete mode 100644 lisp/ob-sh.el
 create mode 100644 lisp/ob-shell.el

diff --git a/lisp/ob-sh.el b/lisp/ob-sh.el
deleted file mode 100644
index a115f38..0000000
--- a/lisp/ob-sh.el
+++ /dev/null
@@ -1,230 +0,0 @@
-;;; ob-sh.el --- org-babel functions for shell evaluation
-
-;; Copyright (C) 2009-2013 Free Software Foundation, Inc.
-
-;; Author: Eric Schulte
-;; Keywords: literate programming, reproducible research
-;; Homepage: http://orgmode.org
-
-;; This file is part of GNU Emacs.
-
-;; GNU Emacs is free software: you can redistribute it and/or modify
-;; it under the terms of the GNU General Public License as published by
-;; the Free Software Foundation, either version 3 of the License, or
-;; (at your option) any later version.
-
-;; GNU Emacs is distributed in the hope that it will be useful,
-;; but WITHOUT ANY WARRANTY; without even the implied warranty of
-;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-;; GNU General Public License for more details.
-
-;; You should have received a copy of the GNU General Public License
-;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
-
-;;; Commentary:
-
-;; Org-Babel support for evaluating shell source code.
-
-;;; Code:
-(require 'ob)
-(require 'shell)
-(eval-when-compile (require 'cl))
-
-(declare-function org-babel-comint-in-buffer "ob-comint" (buffer &rest body))
-(declare-function org-babel-comint-wait-for-output "ob-comint" (buffer))
-(declare-function org-babel-comint-buffer-livep "ob-comint" (buffer))
-(declare-function org-babel-comint-with-output "ob-comint" (meta &rest body))
-(declare-function orgtbl-to-generic "org-table" (table params))
-
-(defvar org-babel-default-header-args:sh '())
-
-(defcustom org-babel-sh-command shell-file-name
-  "Command used to invoke a shell.
-Set by default to the value of `shell-file-name'.  This will be
-passed to `shell-command-on-region'"
-  :group 'org-babel
-  :type 'string)
-
-(defcustom org-babel-sh-var-quote-fmt
-  "$(cat <<'BABEL_TABLE'\n%s\nBABEL_TABLE\n)"
-  "Format string used to escape variables when passed to shell scripts."
-  :group 'org-babel
-  :type 'string)
-
-(defcustom org-babel-shell-names '("sh" "bash" "csh" "ash" "dash")
-  "List of names of shell supported by babel shell code blocks."
-  :group 'org-babel
-  :type 'string
-  :initialize
-  (lambda (symbol value)
-    (set-default symbol (second value))
-    (mapc
-     (lambda (name)
-       (eval `(defun ,(intern (concat "org-babel-execute:" name)) (body params)
-		,(format "Execute a block of %s commands with Babel." name)
-		(let ((org-babel-sh-command ,name))
-		  (org-babel-execute:shell body params)))))
-     (second value))))
-
-(defun org-babel-execute:shell (body params)
-  "Execute a block of Shell commands with Babel.
-This function is called by `org-babel-execute-src-block'."
-  (let* ((session (org-babel-sh-initiate-session
-		   (cdr (assoc :session params))))
-	 (stdin (let ((stdin (cdr (assoc :stdin params))))
-                  (when stdin (org-babel-sh-var-to-string
-                               (org-babel-ref-resolve stdin)))))
-         (full-body (org-babel-expand-body:generic
-		     body params (org-babel-variable-assignments:sh params))))
-    (org-babel-reassemble-table
-     (org-babel-sh-evaluate session full-body params stdin)
-     (org-babel-pick-name
-      (cdr (assoc :colname-names params)) (cdr (assoc :colnames params)))
-     (org-babel-pick-name
-      (cdr (assoc :rowname-names params)) (cdr (assoc :rownames params))))))
-
-(defun org-babel-prep-session:sh (session params)
-  "Prepare SESSION according to the header arguments specified in PARAMS."
-  (let* ((session (org-babel-sh-initiate-session session))
-	 (var-lines (org-babel-variable-assignments:sh params)))
-    (org-babel-comint-in-buffer session
-      (mapc (lambda (var)
-              (insert var) (comint-send-input nil t)
-              (org-babel-comint-wait-for-output session)) var-lines))
-    session))
-
-(defun org-babel-load-session:sh (session body params)
-  "Load BODY into SESSION."
-  (save-window-excursion
-    (let ((buffer (org-babel-prep-session:sh session params)))
-      (with-current-buffer buffer
-        (goto-char (process-mark (get-buffer-process (current-buffer))))
-        (insert (org-babel-chomp body)))
-      buffer)))
-
-;; helper functions
-
-(defun org-babel-variable-assignments:sh (params)
-  "Return list of shell statements assigning the block's variables."
-  (let ((sep (cdr (assoc :separator params))))
-    (mapcar
-     (lambda (pair)
-       (format "%s=%s"
-	       (car pair)
-	       (org-babel-sh-var-to-sh (cdr pair) sep)))
-     (mapcar #'cdr (org-babel-get-header params :var)))))
-
-(defun org-babel-sh-var-to-sh (var &optional sep)
-  "Convert an elisp value to a shell variable.
-Convert an elisp var into a string of shell commands specifying a
-var of the same value."
-  (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep)))
-
-(defun org-babel-sh-var-to-string (var &optional sep)
-  "Convert an elisp value to a string."
-  (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
-    (cond
-     ((and (listp var) (or (listp (car var)) (equal (car var) 'hline)))
-      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var)))
-     ((listp var)
-      (mapconcat echo-var var "\n"))
-     (t (funcall echo-var var)))))
-
-(defun org-babel-sh-table-or-results (results)
-  "Convert RESULTS to an appropriate elisp value.
-If the results look like a table, then convert them into an
-Emacs-lisp table, otherwise return the results as a string."
-  (org-babel-script-escape results))
-
-(defun org-babel-sh-initiate-session (&optional session params)
-  "Initiate a session named SESSION according to PARAMS."
-  (when (and session (not (string= session "none")))
-    (save-window-excursion
-      (or (org-babel-comint-buffer-livep session)
-          (progn (shell session) (get-buffer (current-buffer)))))))
-
-(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
-  "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
-  "String to indicate that evaluation has completed.")
-
-(defun org-babel-sh-evaluate (session body &optional params stdin)
-  "Pass BODY to the Shell process in BUFFER.
-If RESULT-TYPE equals 'output then return a list of the outputs
-of the statements in BODY, if RESULT-TYPE equals 'value then
-return the value of the last statement in BODY."
-  (let ((results
-         (cond
-          (stdin                        ; external shell script w/STDIN
-           (let ((script-file (org-babel-temp-file "sh-script-"))
-                 (stdin-file (org-babel-temp-file "sh-stdin-"))
-                 (shebang (cdr (assoc :shebang params)))
-                 (padline (not (string= "no" (cdr (assoc :padline params))))))
-             (with-temp-file script-file
-               (when shebang (insert (concat shebang "\n")))
-               (when padline (insert "\n"))
-               (insert body))
-             (set-file-modes script-file #o755)
-             (with-temp-file stdin-file (insert stdin))
-             (with-temp-buffer
-               (call-process-shell-command
-                (if shebang
-                    script-file
-                  (format "%s %s" org-babel-sh-command script-file))
-                stdin-file
-                (current-buffer))
-               (buffer-string))))
-          (session                      ; session evaluation
-           (mapconcat
-            #'org-babel-sh-strip-weird-long-prompt
-            (mapcar
-             #'org-babel-trim
-             (butlast
-              (org-babel-comint-with-output
-                  (session org-babel-sh-eoe-output t body)
-                (mapc
-                 (lambda (line)
-                   (insert line)
-                   (comint-send-input nil t)
-                   (while (save-excursion
-                            (goto-char comint-last-input-end)
-                            (not (re-search-forward
-                                  comint-prompt-regexp nil t)))
-                     (accept-process-output
-                      (get-buffer-process (current-buffer)))))
-                 (append
-                  (split-string (org-babel-trim body) "\n")
-                  (list org-babel-sh-eoe-indicator))))
-              2)) "\n"))
-          ('otherwise                   ; external shell script
-           (if (and (cdr (assoc :shebang params))
-                    (> (length (cdr (assoc :shebang params))) 0))
-               (let ((script-file (org-babel-temp-file "sh-script-"))
-                     (shebang (cdr (assoc :shebang params)))
-                     (padline (not (equal "no" (cdr (assoc :padline params))))))
-                 (with-temp-file script-file
-                   (when shebang (insert (concat shebang "\n")))
-                   (when padline (insert "\n"))
-                   (insert body))
-                 (set-file-modes script-file #o755)
-                 (org-babel-eval script-file ""))
-             (org-babel-eval org-babel-sh-command (org-babel-trim body)))))))
-    (when results
-      (let ((result-params (cdr (assoc :result-params params))))
-        (org-babel-result-cond result-params
-          results
-          (let ((tmp-file (org-babel-temp-file "sh-")))
-            (with-temp-file tmp-file (insert results))
-            (org-babel-import-elisp-from-file tmp-file)))))))
-
-(defun org-babel-sh-strip-weird-long-prompt (string)
-  "Remove prompt cruft from a string of shell output."
-  (while (string-match "^% +[\r\n$]+ *" string)
-    (setq string (substring string (match-end 0))))
-  string)
-
-(provide 'ob-sh)
-
-
-
-;;; ob-sh.el ends here
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
new file mode 100644
index 0000000..a115f38
--- /dev/null
+++ b/lisp/ob-shell.el
@@ -0,0 +1,230 @@
+;;; ob-sh.el --- org-babel functions for shell evaluation
+
+;; Copyright (C) 2009-2013 Free Software Foundation, Inc.
+
+;; Author: Eric Schulte
+;; Keywords: literate programming, reproducible research
+;; Homepage: http://orgmode.org
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Org-Babel support for evaluating shell source code.
+
+;;; Code:
+(require 'ob)
+(require 'shell)
+(eval-when-compile (require 'cl))
+
+(declare-function org-babel-comint-in-buffer "ob-comint" (buffer &rest body))
+(declare-function org-babel-comint-wait-for-output "ob-comint" (buffer))
+(declare-function org-babel-comint-buffer-livep "ob-comint" (buffer))
+(declare-function org-babel-comint-with-output "ob-comint" (meta &rest body))
+(declare-function orgtbl-to-generic "org-table" (table params))
+
+(defvar org-babel-default-header-args:sh '())
+
+(defcustom org-babel-sh-command shell-file-name
+  "Command used to invoke a shell.
+Set by default to the value of `shell-file-name'.  This will be
+passed to `shell-command-on-region'"
+  :group 'org-babel
+  :type 'string)
+
+(defcustom org-babel-sh-var-quote-fmt
+  "$(cat <<'BABEL_TABLE'\n%s\nBABEL_TABLE\n)"
+  "Format string used to escape variables when passed to shell scripts."
+  :group 'org-babel
+  :type 'string)
+
+(defcustom org-babel-shell-names '("sh" "bash" "csh" "ash" "dash")
+  "List of names of shell supported by babel shell code blocks."
+  :group 'org-babel
+  :type 'string
+  :initialize
+  (lambda (symbol value)
+    (set-default symbol (second value))
+    (mapc
+     (lambda (name)
+       (eval `(defun ,(intern (concat "org-babel-execute:" name)) (body params)
+		,(format "Execute a block of %s commands with Babel." name)
+		(let ((org-babel-sh-command ,name))
+		  (org-babel-execute:shell body params)))))
+     (second value))))
+
+(defun org-babel-execute:shell (body params)
+  "Execute a block of Shell commands with Babel.
+This function is called by `org-babel-execute-src-block'."
+  (let* ((session (org-babel-sh-initiate-session
+		   (cdr (assoc :session params))))
+	 (stdin (let ((stdin (cdr (assoc :stdin params))))
+                  (when stdin (org-babel-sh-var-to-string
+                               (org-babel-ref-resolve stdin)))))
+         (full-body (org-babel-expand-body:generic
+		     body params (org-babel-variable-assignments:sh params))))
+    (org-babel-reassemble-table
+     (org-babel-sh-evaluate session full-body params stdin)
+     (org-babel-pick-name
+      (cdr (assoc :colname-names params)) (cdr (assoc :colnames params)))
+     (org-babel-pick-name
+      (cdr (assoc :rowname-names params)) (cdr (assoc :rownames params))))))
+
+(defun org-babel-prep-session:sh (session params)
+  "Prepare SESSION according to the header arguments specified in PARAMS."
+  (let* ((session (org-babel-sh-initiate-session session))
+	 (var-lines (org-babel-variable-assignments:sh params)))
+    (org-babel-comint-in-buffer session
+      (mapc (lambda (var)
+              (insert var) (comint-send-input nil t)
+              (org-babel-comint-wait-for-output session)) var-lines))
+    session))
+
+(defun org-babel-load-session:sh (session body params)
+  "Load BODY into SESSION."
+  (save-window-excursion
+    (let ((buffer (org-babel-prep-session:sh session params)))
+      (with-current-buffer buffer
+        (goto-char (process-mark (get-buffer-process (current-buffer))))
+        (insert (org-babel-chomp body)))
+      buffer)))
+
+;; helper functions
+
+(defun org-babel-variable-assignments:sh (params)
+  "Return list of shell statements assigning the block's variables."
+  (let ((sep (cdr (assoc :separator params))))
+    (mapcar
+     (lambda (pair)
+       (format "%s=%s"
+	       (car pair)
+	       (org-babel-sh-var-to-sh (cdr pair) sep)))
+     (mapcar #'cdr (org-babel-get-header params :var)))))
+
+(defun org-babel-sh-var-to-sh (var &optional sep)
+  "Convert an elisp value to a shell variable.
+Convert an elisp var into a string of shell commands specifying a
+var of the same value."
+  (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep)))
+
+(defun org-babel-sh-var-to-string (var &optional sep)
+  "Convert an elisp value to a string."
+  (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
+    (cond
+     ((and (listp var) (or (listp (car var)) (equal (car var) 'hline)))
+      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var)))
+     ((listp var)
+      (mapconcat echo-var var "\n"))
+     (t (funcall echo-var var)))))
+
+(defun org-babel-sh-table-or-results (results)
+  "Convert RESULTS to an appropriate elisp value.
+If the results look like a table, then convert them into an
+Emacs-lisp table, otherwise return the results as a string."
+  (org-babel-script-escape results))
+
+(defun org-babel-sh-initiate-session (&optional session params)
+  "Initiate a session named SESSION according to PARAMS."
+  (when (and session (not (string= session "none")))
+    (save-window-excursion
+      (or (org-babel-comint-buffer-livep session)
+          (progn (shell session) (get-buffer (current-buffer)))))))
+
+(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
+  "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
+  "String to indicate that evaluation has completed.")
+
+(defun org-babel-sh-evaluate (session body &optional params stdin)
+  "Pass BODY to the Shell process in BUFFER.
+If RESULT-TYPE equals 'output then return a list of the outputs
+of the statements in BODY, if RESULT-TYPE equals 'value then
+return the value of the last statement in BODY."
+  (let ((results
+         (cond
+          (stdin                        ; external shell script w/STDIN
+           (let ((script-file (org-babel-temp-file "sh-script-"))
+                 (stdin-file (org-babel-temp-file "sh-stdin-"))
+                 (shebang (cdr (assoc :shebang params)))
+                 (padline (not (string= "no" (cdr (assoc :padline params))))))
+             (with-temp-file script-file
+               (when shebang (insert (concat shebang "\n")))
+               (when padline (insert "\n"))
+               (insert body))
+             (set-file-modes script-file #o755)
+             (with-temp-file stdin-file (insert stdin))
+             (with-temp-buffer
+               (call-process-shell-command
+                (if shebang
+                    script-file
+                  (format "%s %s" org-babel-sh-command script-file))
+                stdin-file
+                (current-buffer))
+               (buffer-string))))
+          (session                      ; session evaluation
+           (mapconcat
+            #'org-babel-sh-strip-weird-long-prompt
+            (mapcar
+             #'org-babel-trim
+             (butlast
+              (org-babel-comint-with-output
+                  (session org-babel-sh-eoe-output t body)
+                (mapc
+                 (lambda (line)
+                   (insert line)
+                   (comint-send-input nil t)
+                   (while (save-excursion
+                            (goto-char comint-last-input-end)
+                            (not (re-search-forward
+                                  comint-prompt-regexp nil t)))
+                     (accept-process-output
+                      (get-buffer-process (current-buffer)))))
+                 (append
+                  (split-string (org-babel-trim body) "\n")
+                  (list org-babel-sh-eoe-indicator))))
+              2)) "\n"))
+          ('otherwise                   ; external shell script
+           (if (and (cdr (assoc :shebang params))
+                    (> (length (cdr (assoc :shebang params))) 0))
+               (let ((script-file (org-babel-temp-file "sh-script-"))
+                     (shebang (cdr (assoc :shebang params)))
+                     (padline (not (equal "no" (cdr (assoc :padline params))))))
+                 (with-temp-file script-file
+                   (when shebang (insert (concat shebang "\n")))
+                   (when padline (insert "\n"))
+                   (insert body))
+                 (set-file-modes script-file #o755)
+                 (org-babel-eval script-file ""))
+             (org-babel-eval org-babel-sh-command (org-babel-trim body)))))))
+    (when results
+      (let ((result-params (cdr (assoc :result-params params))))
+        (org-babel-result-cond result-params
+          results
+          (let ((tmp-file (org-babel-temp-file "sh-")))
+            (with-temp-file tmp-file (insert results))
+            (org-babel-import-elisp-from-file tmp-file)))))))
+
+(defun org-babel-sh-strip-weird-long-prompt (string)
+  "Remove prompt cruft from a string of shell output."
+  (while (string-match "^% +[\r\n$]+ *" string)
+    (setq string (substring string (match-end 0))))
+  string)
+
+(provide 'ob-sh)
+
+
+
+;;; ob-sh.el ends here
diff --git a/testing/lisp/test-ob-sh.el b/testing/lisp/test-ob-sh.el
index 1025ecf..ca94c9e 100644
--- a/testing/lisp/test-ob-sh.el
+++ b/testing/lisp/test-ob-sh.el
@@ -24,7 +24,7 @@
 
 ;;; Code:
 (org-test-for-executable "sh")
-(unless (featurep 'ob-sh)
+(unless (featurep 'ob-shell)
   (signal 'missing-test-dependency "Support for Sh code blocks"))
 
 (ert-deftest test-ob-sh/dont-insert-spaces-on-expanded-bodies ()
-- 
1.8.5.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-finish-rename-split-up-for-git.patch --]
[-- Type: text/x-diff, Size: 860 bytes --]

From 3b3dbca13c668299a24c93eae27059703361d789 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 13 Dec 2013 09:54:48 -0700
Subject: [PATCH 3/4] finish rename (split up for git)

---
 lisp/ob-shell.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index a115f38..d1c2130 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -1,4 +1,4 @@
-;;; ob-sh.el --- org-babel functions for shell evaluation
+;;; ob-shell.el --- org-babel functions for shell evaluation
 
 ;; Copyright (C) 2009-2013 Free Software Foundation, Inc.
 
@@ -223,8 +223,8 @@ return the value of the last statement in BODY."
     (setq string (substring string (match-end 0))))
   string)
 
-(provide 'ob-sh)
+(provide 'ob-shell)
 
 
 
-;;; ob-sh.el ends here
+;;; ob-shell.el ends here
-- 
1.8.5.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-fix-tests-with-the-renamed-ob-shell.el.patch --]
[-- Type: text/x-diff, Size: 925 bytes --]

From b319475a86451defbdbac064684dd59fd6b0b7d0 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 13 Dec 2013 10:03:05 -0700
Subject: [PATCH 4/4] fix tests with the renamed ob-shell.el

  Note, users may have to edit their local.mk files to change the
  value of BTEST_OB_LANGUAGES to remove sh and include shell.
---
 mk/default.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/default.mk b/mk/default.mk
index 886efc6..a1f42f8 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -63,7 +63,7 @@ BTEST	= $(BATCH) \
 	  --eval '(setq \
 		org-batch-test t \
 		org-babel-load-languages \
-	          (quote ($(foreach ob-lang,$(BTEST_OB_LANGUAGES) emacs-lisp sh org,$(lst-ob-lang)))) \
+	          (quote ($(foreach ob-lang,$(BTEST_OB_LANGUAGES) emacs-lisp shell org,$(lst-ob-lang)))) \
 	    org-test-select-re "$(BTEST_RE)" \
 	  )' \
 	  -l org-loaddefs.el \
-- 
1.8.5.1


[-- Attachment #6: Type: text/plain, Size: 63 bytes --]


-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Nick Dokos]

Eric Schulte <schulte.eric@gmail.com> writes:

>>
>> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
>> New "shell" code blocks could use the value of the
>> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
>> ash, dash (am I missing any other common ones) use the specific shell
>> specified.
>>
>
> The attached patches make this change and continue to pass the entire
> test suite.  The problem being that with ob-sh, no longer present many
> users may have to change their configuration and possible the value of
> their local.mk file.  One solution there is to add a dummy ob-sh.el with
> a deprecation message for some transition time.  Thoughts?
>
>

Since I'm the de facto instigator of the original change in the default,
let me add my 2 cents. I'm fine with this change (or without it). I'd be
fine with changing the org-babel-sh-command setting in my config and
leaving the default alone.

And I sympathize with Greg's wish for portability in general, although
imo it's not particularly important in this case (ducking and donning
my asbestos underwear here).

I write short scripts in org files to document some process: I can't
remember anything any longer, so putting the details in a file is the
only way for me to figure out what I did six months (or even two days)
ago (finding the file again two days hence is another matter...) In most
cases, what I put in there is some sequence of commands, which will be
interpreted correctly no matter which shell is used.  If I have anything
more complicated (non-trivial control flow, non-trivial i/o redirection,
etc etc), I put it in a script in ~/bin and invoke that in the source
code block. I may not be typical here of course, but that's why I think
that portability is not particularly important in this case - so leave
the default at sh and be done with it.

But when I tried and failed to run Eric's script in the original email,
I had to do a little digging to figure out what went wrong and how to
fix it (I don't remember running across org-babel-sh-command before
this). So I asked the question about sh and the rest is history. I
probably should have made the observation that org-babel-sh-command had
to be modified to run the code block (which was plainly true) and left
the question (which could be interpreted in various ways) out. OTOH,
there is now this discussion and presumably the end result will be
better than what we started with.

Nick

PS. I haven't tried out the patch but I plan to do so over the weekend.

Re: [babel] how to pass data to gnuplot from another block

[Achim Gratz]

Eric Schulte writes:
>> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
>> New "shell" code blocks could use the value of the
>> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
>> ash, dash (am I missing any other common ones) use the specific shell
>> specified.

I've also seen ksh, mksh, posh (the latter specifically for POSIX
compatibility checks).  But trying to enumerate all possible shell names
is futile, especially when the same shell dialect can have different
names on different systems and you'll only find a handful of those on
each particular system installed.  Then there are those systems where at
least two different shells exist with the same name in different paths
and you'll get one or the other depending on which way your path is set
up.

> The attached patches make this change and continue to pass the entire
> test suite.  The problem being that with ob-sh, no longer present many
> users may have to change their configuration and possible the value of
> their local.mk file.  One solution there is to add a dummy ob-sh.el with
> a deprecation message for some transition time.  Thoughts?

I'm not sure this does the right thing (if that is even possible in this
case).  It looks overcomplicated to me, anyway.

There are two sides to a shell: the programming language / scripting
part and the interactive part.  Of those shells that are somewhat POSIX
compatible, the programming language part isn't all that much different
(at least no more than, say, different C dialects).  Even csh does the
right thing with a lot of POSIX stuff and you shouldn't really use it
for serious scripting anyway.  The interactive part shouldn't really
figure into Babel, even though the particular choice will introduce one
or the other quirk in certain areas of scripting if you're not careful.

Emacs' shell-mode recognizes that ambiguity: it looks up the bang line
to decide which dialect to chose and waits for a user decision if it
can't find one.  I'd have no problem if ob-sh did the same and simply
ran with whatever it can get away with (assuming close-enough-to-POSIX)
and only chose a specific shell when asked (via bang line or otherwise).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

It sounds as though most people don't particularly care which shell is
used.  However, I believe Greg is correct and the *right* thing to do is
to have specific names (bash, sh, etc...) denote specific shells.  I'd
also like "#+begin_src shell" to specify the "don't care" option.  That
is what these patches do, with relatively little churn in the code (the
one huge commit just renames a file).

I just applied these patches.  The worst case is that users may have to
change "ob-sh" to "ob-shell" in their config (although some initial
testing seems to indicate that even this change won't be required), and
possibly replace "sh" with "shell" in their local.mk file (if they have
one) to run tests at the command line.  In my mind this short-term
hassle is worth the long-term correctness.

I've just applied these patches.  At this point I'll borrow Nick's
asbestos underwear and place it on my head.

Best Regards,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Achim Gratz]

Eric Schulte writes:
> I just applied these patches.  The worst case is that users may have to
> change "ob-sh" to "ob-shell" in their config (although some initial
> testing seems to indicate that even this change won't be required), and
> possibly replace "sh" with "shell" in their local.mk file (if they have
> one) to run tests at the command line.  In my mind this short-term
> hassle is worth the long-term correctness.

Since "sh" is redundant there and has been for some time (like
emacs-lisp and org it is always loaded since some non-Babel tests need
it), it would be easier to just delete it from BTEST_OB_LANGUAGES if
present.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [babel] how to pass data to gnuplot from another block

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]


Eric Schulte <schulte.eric@gmail.com> writes:

> Although purely semantically, in my opinion the "sh" in "#+begin_src sh"
> indicates generic "shell-script", not the POSIX sh.  E.g., there is no
> ob-bash.el or ob-csh.el.

I see your point.  But stepping back, I have always felt that
"#+begin_src foo" referred to a language, sometimes where that language
and a particular program are inseparable (e.g. gnuplot).  But sh is a
first-class language.

> See the first line in ob-sh.el,
> ,----
> | ;;; ob-sh.el --- org-babel functions for shell evaluation
> `----

Sure, but that's just repeated the ambiguity :-)

> And this is where we disagree.  Sh code blocks don't currently promise
> POSIX sh, they promise a shell.  This is certainly a much more dangerous
> generalization than say Perl code blocks possibly using Perl 5 or 6.

For shell, I see that there are two concepts to detangle:

  a shell is a particular command interpreter, particularly useful for
  humans

  there are multiple shell languages, but far fewer than the number of
  interpreters.


For languages, I see

   POSIX sh
   the bash flavor of POSIX sh
   csh

While bash and POSIX sh are close, csh isn't at all close, and is only
similar in that people also use it for a shell.


In an org document, I think it's better if the result depends less on
variables not set in the document.   So a code block in a document is
really written in some language.  And it therefore makes sense to
specify that in the begin_src wrapper.

I don't see that there is any call for csh, as the received wisdom is
that one shouldn't write scripts in it (at least in modern times).  (It
was originally a BSD thing, and BSD culture is very much POSIX sh now.)

So separately from how the lisp works, I would favor

  #+begin_src sh     # posix sh
  #+begin_src bash   # bash (leaving version ambiguous??)
  #+begin_src csh     # csh, but not sure there's a need


> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
> New "shell" code blocks could use the value of the
> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
> ash, dash (am I missing any other common ones) use the specific shell
> specified.

Are you aware of any significant use of "zsh scripts"?  I see that as
POSIX sh, with spiffy user-facing features.
> In the mean time, I don't believe the change in default value for
> `org-babel-sh-command' has been included in any maintenance releases, so
> I've just reverted this to minimize any further confusion.

Thanks.  It's good to be having the larger discussion first.

> And I should say that I've argued the same point your making myself in
> the past (on a project making the much more serious error of using bash
> notation ">&" in a shell script starting with "#!/bin/sh").  I think we
> only disagree on the current meaning of "sh" in code blocks, which
> hopefully the suggestion above will rectify.

I forgot about "&>", even though I type it all the time interactively
but I'm pretty careful in scripts :-)

Thanks,
Greg

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [babel] how to pass data to gnuplot from another block

[Sebastien Vauban]

Greg Troxel wrote:
> Eric Schulte <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> Although purely semantically, in my opinion the "sh" in "#+begin_src sh"
>> indicates generic "shell-script", not the POSIX sh.  E.g., there is no
>> ob-bash.el or ob-csh.el.
>
> I see your point.  But stepping back, I have always felt that
> "#+begin_src foo" referred to a language

I'd say, personally, that `foo' would refer to a mode (`foo-mode') [1]
which supports one (or multiple) language(s).

And I guess that, in Emacs, `sh' is the mode for editing Shell scripts
(in sh, bash, zsh, etc.). Though, I'm not 100% sure of what I'm saying
here...

> sometimes where that language and a particular program are inseparable
> (e.g. gnuplot). But sh is a first-class language.
>
>> See the first line in ob-sh.el,
>> ,----
>> | ;;; ob-sh.el --- org-babel functions for shell evaluation
>> `----

Best regards,
  Seb

[1] Do C-c ' and see that Org (tries to) call(s) the mode foo-mode.

-- 
Sebastien Vauban

Re: [babel] how to pass data to gnuplot from another block

[Achim Gratz]

Eric Schulte writes:
> I understand your point, but in reality I doubt there are many systems
> on which people use Org-mode with code blocks and on which sh is
> available but no bash is installed.

You might want to widen your horizon on the "many systems" front a bit.
The typical BSD system has no Bash at all and Debian specifically avoids
the use of any bashisms in any system related scripts in order to be
able to use something else but Bash as /bin/sh… and this is just talking
about the UN*Xoid part of the world.

> Bash is the new normal shell and I would argue is what most users expect
> from a shell code block.

Or mksh or fish or whatever is the shiny new shell thing of today.
While we are inventing statistics, I would argue that most users have no
idea what a shell is and that there are different kinds of those.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [babel] how to pass data to gnuplot from another block

[Eric S Fraga]

Eric Schulte <schulte.eric@gmail.com> writes:

> Hi Eric,
>
> The attached works fine for me (using sh since I don't have octave).

Dear Eric,

thanks for your quick reply and sorry for taking so long to get back to
you.  I ended up going to Chile for a week the day after sending my
original email and had very little Internet access (and, actually, no
time even if I had had proper connectivity).  Back now!

[...]

> Ensure that the data you're passing into gnuplot is a table and not a
> string.  Gnuplot blocks handle tables by writing them to a file, and
> then replacing the variable with the file name.  As I recall gnuplot
> blocks assume string data already is a file name, so the variable is
> replaced directly.

Ah ha, this is a subtle one!  The output looks the same in either case,
as Nick D. also notes.  By changing the results generated to value and
not output and adding an expression at the end of my octave code to
generate the value required got things working.

The subtlety makes this a rather frustrating experience, however.  Is
there any way to convince babel to treat a string output as an org table
so that subsequent chaining works in this case as well?

Thanks again,
eric
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.3.50.1, Org release_8.2.2-181-gf31eb4.dirty

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Hi Eric,
>>
>> The attached works fine for me (using sh since I don't have octave).
>
> Dear Eric,
>
> thanks for your quick reply and sorry for taking so long to get back to
> you.  I ended up going to Chile for a week the day after sending my
> original email and had very little Internet access (and, actually, no
> time even if I had had proper connectivity).  Back now!
>
> [...]
>
>> Ensure that the data you're passing into gnuplot is a table and not a
>> string.  Gnuplot blocks handle tables by writing them to a file, and
>> then replacing the variable with the file name.  As I recall gnuplot
>> blocks assume string data already is a file name, so the variable is
>> replaced directly.
>
> Ah ha, this is a subtle one!  The output looks the same in either case,
> as Nick D. also notes.  By changing the results generated to value and
> not output and adding an expression at the end of my octave code to
> generate the value required got things working.
>
> The subtlety makes this a rather frustrating experience, however.  Is
> there any way to convince babel to treat a string output as an org table
> so that subsequent chaining works in this case as well?
>

I'm not familiar with ob-octave, but I'd imagine ":results vector"
should do the trick.  If not then it might be worth adding something
like the following (borrowed from ob-sh.el) to ob-octave.

    (org-babel-result-cond (cdr (assoc :result-params params))
      results
      (let ((tmp-file (org-babel-temp-file "sh-")))
        (with-temp-file tmp-file (insert results))
        (org-babel-import-elisp-from-file tmp-file)))

Best,

>
> Thanks again,
> eric

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Eric S Fraga]

Eric Schulte <schulte.eric@gmail.com> writes:

[...]

> I'm not familiar with ob-octave, but I'd imagine ":results vector"
> should do the trick.  

Nope.  Unfortunately, this doesn't work.

If the result I want is what is actually "output" by octave, then this
ignores that output.  If I put ":results output vector", this doesn't
work either for chaining although at least the results come out looking
right.

Thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.3.50.1, Org release_8.2.3c-266-g7a726d

Re: [babel] how to pass data to gnuplot from another block

[Eric Schulte]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
> [...]
>
>> I'm not familiar with ob-octave, but I'd imagine ":results vector"
>> should do the trick.  
>
> Nope.  Unfortunately, this doesn't work.
>
> If the result I want is what is actually "output" by octave, then this
> ignores that output.  If I put ":results output vector", this doesn't
> work either for chaining although at least the results come out looking
> right.
>
> Thanks,
> eric

Sounds like someone with an interest in octave support should take a
look at ob-octave.el, this shouldn't be a difficult fix and there are
many examples of other languages (e.g., sh) handling the combination of
output and vectors.

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [babel] how to pass data to gnuplot from another block

[Eric S Fraga]

Eric Schulte <schulte.eric@gmail.com> writes:

[...]

> Sounds like someone with an interest in octave support should take a
> look at ob-octave.el, this shouldn't be a difficult fix and there are
> many examples of other languages (e.g., sh) handling the combination of
> output and vectors.

Okay.  I'll add it to my todo list... but with low priority given that
using a value output works just fine.

thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.3.50.1, Org release_8.2.3c-333-g487c74