From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pascal Fleury Subject: Re: Export arrays for 'sh' code blocks when using bash Date: Thu, 27 Mar 2014 23:52:39 +0100 Message-ID: References: <871txn1qni.fsf@bagel.gateway.pace.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=001a1136a95a04001e04f59e73f8 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:39748) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WTJAf-0000Tp-7d for emacs-orgmode@gnu.org; Thu, 27 Mar 2014 18:53:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WTJAd-0003f8-4q for emacs-orgmode@gnu.org; Thu, 27 Mar 2014 18:53:13 -0400 Received: from mail-ve0-x22d.google.com ([2607:f8b0:400c:c01::22d]:58812) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WTJAc-0003f4-UN for emacs-orgmode@gnu.org; Thu, 27 Mar 2014 18:53:11 -0400 Received: by mail-ve0-f173.google.com with SMTP id oy12so4868422veb.4 for ; Thu, 27 Mar 2014 15:53:10 -0700 (PDT) In-Reply-To: <871txn1qni.fsf@bagel.gateway.pace.com> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Eric Schulte Cc: emacs-orgmode@gnu.org --001a1136a95a04001e04f59e73f8 Content-Type: text/plain; charset=UTF-8 Hi Eric, see comments inline. On Thu, Mar 27, 2014 at 6:43 PM, Eric Schulte wrote: > Hi Pascal, > > This looks like a good patch. If I could make three changes/requests. > > Thanks, I am happy to see it makes sense. > 1. This patch should also work for "begin_src bash" code blocks. After > a very quick glance it doesn't appear to. > > I have actually tried this with the latest org-mode release (8.2.5h) and it did plainly not accept 'begin_src bash'. Maybe that's new with the ob-shell.el (I noticed it has been renamed :-) > 2. Could you include one or two tests ensuring that this patch does what > it intends with bash code blocks, and does not affect code blocks > executed with bin/sh? > Will do, seeing it has some traction. > > 3. Large contributions like this require some FSF paperwork. Please see > the following page for information on requirements to contribute. > > I checked internally, and apparently, Google has already such a signed document. It would run on the same document, as I am a google employee. > http://orgmode.org/worg/org-contribute.html > > Thanks, and I look forward to seeing this merged into Org-mode! > Eric > > Pascal Fleury writes: > > > Hello, > > > > I'dl like to propose a patch for inclusion into org-mode (ob-shell.el in > > particular). > > > > *TL;DR:* use arrays and associative arrays when exporting variables to > bash > > in 'sh' code blocks. > > > > *Details:* > > When variables are defined in a 'sh' code block, they are exported as > > strings. when the variable itself is an array or a table, then we simply > > get a shell variable that contains the list of all values in a > > non-structured form. E.g. > > > > #+NAME: my_list > > | one | > > | two | > > | three | > > > > #+NAME: experiment > > | name | first_attempt | > > | date | [2014-03-27 Thu] | > > | user | fleury | > > > > #+BEGIN_SRC sh :var scalar="some value" :var array=my_list :var > table=config > > echo ${scalar} # -> prints 'some value' > > echo ${array} # -> prints 'one two three' > > echo ${table} # -> prints 'first attempt [2014-03-27 Thu] fleury' > > #+END_SRC > > > > This will print simple strings. Also, there is no easy way to access the > > date of the experiment, for example. Now bash has things like arrays and > > associative arrays, but the current ob-shell.el does not use these. > > Probably because their syntax is bash-specific, and ob-shell.el is > > shell-agnostic. > > > > My patch (attached) changes this in the case you have > > (setq org-babel-sh-command "bash") > > in your emacs config somewhere. If any other value is used, it continues > to > > export them as we do today (I don't know enough about other shells). > > > > In that case, it will export the list as an array, the table as an > > associative array, and the scalar as it does already. So the 'sh' code > > block could then use > > > > #+BEGIN_SRC sh :var scalar="some value" :var array=my_list :var > table=config > > echo ${scalar} > > echo ${array[1]} # -> prints "two" > > echo ${table[user]} # -> prints "fleury" > > #+END_SRC > > > > In the case we have a bigger table, then the first row is used as key, > the > > others are represented as a string with all the values (as it is for > array > > currently). bash does not have multi-dimensional arrays, so it needs to > be > > a string. > > > > This makes writing shell snippets much easier in my experience, and there > > I'd like to propose this fix for the org-mode community at large. > > > > --paf > > > > diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el > > index 3ede701..0a691e2 100644 > > --- a/lisp/ob-shell.el > > +++ b/lisp/ob-shell.el > > @@ -106,6 +106,30 @@ This function is called by > `org-babel-execute-src-block'." > > > > ;; helper functions > > > > +(defun org-babel-variable-assignments:bash_array (varname values > &optional sep hline) > > + "Returns a list of statements declaring the values as a bash array." > > + (format "declare -a %s=( \"%s\" )" > > + varname > > + (mapconcat 'identity > > + (mapcar > > + (lambda (value) (org-babel-sh-var-to-sh value sep hline)) > > + values) > > + "\" \""))) > > + > > +(defun org-babel-variable-assignments:bash_associative (varname values > &optional sep hline) > > + "Returns a list of statements declaring the values as bash > associative array." > > + (format "declare -A %s\n%s" > > + varname > > + (mapconcat 'identity > > + (mapcar > > + (lambda (items) > > + (format "%s[\"%s\"]=%s" > > + varname > > + (org-babel-sh-var-to-sh (car items) sep hline) > > + (org-babel-sh-var-to-sh (cdr items) sep hline))) > > + values) > > + "\n"))) > > + > > (defun org-babel-variable-assignments:sh (params) > > "Return list of shell statements assigning the block's variables." > > (let ((sep (cdr (assoc :separator params))) > > @@ -114,9 +138,17 @@ This function is called by > `org-babel-execute-src-block'." > > "hline")))) > > (mapcar > > (lambda (pair) > > - (format "%s=%s" > > - (car pair) > > - (org-babel-sh-var-to-sh (cdr pair) sep hline))) > > + (if (and (string= org-babel-sh-command "bash") (listp (cdr > pair))) > > + (if (listp (car (cdr pair))) > > + (org-babel-variable-assignments:bash_associative > > + (car pair) (cdr pair) sep hline) > > + (org-babel-variable-assignments:bash_array > > + (car pair) (cdr pair) sep) hline) > > + (format "%s=%s" > > + (car pair) > > + (org-babel-sh-var-to-sh (cdr pair) sep hline)) > > + ) > > + ) > > (mapcar #'cdr (org-babel-get-header params :var))))) > > > > (defun org-babel-sh-var-to-sh (var &optional sep hline) > > -- > Eric Schulte > https://cs.unm.edu/~eschulte > PGP: 0x614CA05D > -- --paf --001a1136a95a04001e04f59e73f8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Eric, see comments inline.


On Thu, Mar 27, 2014 at 6:43 PM, Eric = Schulte <schulte.eric@gmail.com> wrote:
Hi Pascal,

This looks like a good patch. =C2=A0If I could make three changes/requests.=

=C2=A0
Thanks, I am happy to see it makes s= ense.
=C2=A0
1. This patch should also work for "begin_src bash" code blocks. = =C2=A0After
=C2=A0 =C2=A0a very quick glance it doesn't appear to.

I have actually tried this with the latest org-mode r= elease (8.2.5h) and it did plainly not accept 'begin_src bash'. May= be that's new with the ob-shell.el (I noticed it has been renamed :-)
=C2=A0
2. Could you include one or two tests ensuring that this patch does what =C2=A0 =C2=A0it intends with bash code blocks, and does not affect code blo= cks
=C2=A0 =C2=A0executed with bin/sh?

Will= do, seeing it has some traction.
=C2=A0

3. Large contributions like this require some FSF paperwork. =C2=A0Please s= ee
=C2=A0 =C2=A0the following page for information on requirements to contribu= te.


I checked internally, and apparently, = Google has already such a signed document. It would run on the same documen= t, as I am a google employee.
=C2=A0
=C2=A0 =C2=A0http://orgmode.org/worg/org-contribute.html

Thanks, and I look forward to seeing this merged into Org-mode!
Eric

Pascal Fleury <fleury@google.com> writes:

> Hello,
>
> I'dl like to propose a patch for inclusion into org-mode (ob-shell= .el in
> particular).
>
> *TL;DR:* use arrays and associative arrays when exporting variab= les to bash
> in 'sh' code blocks.
>
> *Details:*
> When variables are defined in a 'sh' co= de block, they are exported as
> strings. when the variable itself is an array or a table, then we simp= ly
> get a shell variable that contains the list of all values in a
> non-structured form. E.g.
>
> #+NAME: my_list
> | one =C2=A0 |
> | two =C2=A0 |
> | three |
>
> #+NAME: experiment
> | name | first_attempt =C2=A0 =C2=A0|
> | date | [2014-03-27 Thu] |
> | user | fleury =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
>
> #+BEGIN_SRC sh :var scalar=3D"some value" :var array=3Dmy_li= st :var table=3Dconfig
> echo ${scalar} =C2=A0# -> prints 'some value'
> echo ${array} =C2=A0 # -> prints 'one two three'
> echo ${table} =C2=A0 # -> prints 'first attempt [2014-03-27 Thu= ] fleury'
> #+END_SRC
>
> This will print simple strings. Also, there is no easy way to access t= he
> date of the experiment, for example. Now bash has things like arrays a= nd
> associative arrays, but the current ob-shell.el does not use these. > Probably because their syntax is bash-specific, and ob-shell.el is
> shell-agnostic.
>
> My patch (attached) changes this in the case you have
> (setq org-babel-sh-command "bash")
> in your emacs config somewhere. If any other value is used, it continu= es to
> export them as we do today (I don't know enough about other shells= ).
>
> In that case, it will export the list as an array, the table as an
> associative array, and the scalar as it does already. So the 'sh&#= 39; code
> block could then use
>
> #+BEGIN_SRC sh :var scalar=3D"some value" :var array=3Dmy_li= st :var table=3Dconfig
> echo ${scalar}
> echo ${array[1]} # -> prints "two"
> echo ${table[user]} # -> prints "fleury"
> #+END_SRC
>
> In the case we have a bigger table, then the first row is used as key,= the
> others are represented as a string with all the values (as it is for a= rray
> currently). bash does not have multi-dimensional arrays, so it needs t= o be
> a string.
>
> This makes writing shell snippets much easier in my experience, and th= ere
> I'd like to propose this fix for the org-mode community at large.<= br> >
> --paf
>
> diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
> index 3ede701..0a691e2 100644
> --- a/lisp/ob-shell.el
> +++ b/lisp/ob-shell.el
> @@ -106,6 +106,30 @@ This function is called by `org-babel-execute-src= -block'."
>
> =C2=A0;; helper functions
>
> +(defun org-babel-variable-assignments:bash_array (varname values &= ;optional sep hline)
> + =C2=A0"Returns a list of statements declaring the values as a b= ash array."
> + =C2=A0(format "declare -a %s=3D( \"%s\" )"
> + =C2=A0 =C2=A0 varname
> + =C2=A0 =C2=A0 (mapconcat 'identity
> + =C2=A0 =C2=A0 =C2=A0 (mapcar
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 (lambda (value) (org-babel-sh-var-to-sh = value sep hline))
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 values)
> + =C2=A0 =C2=A0 =C2=A0 "\" \"")))
> +
> +(defun org-babel-variable-assignments:bash_associative (varname value= s &optional sep hline)
> + =C2=A0"Returns a list of statements declaring the values as bas= h associative array."
> + =C2=A0(format "declare -A %s\n%s"
> + =C2=A0 =C2=A0varname
> + =C2=A0 =C2=A0(mapconcat 'identity
> + =C2=A0 =C2=A0 =C2=A0(mapcar
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda (items)
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(format "%s[\"%s\"]= =3D%s"
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0varname
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(org-babel-sh-var-to-sh (ca= r items) sep hline)
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(org-babel-sh-var-to-sh (cd= r items) sep hline)))
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0values)
> + =C2=A0 =C2=A0 =C2=A0"\n")))
> +
> =C2=A0(defun org-babel-variable-assignments:sh (params)
> =C2=A0 =C2=A0"Return list of shell statements assigning the block= 's variables."
> =C2=A0 =C2=A0(let ((sep (cdr (assoc :separator params)))
> @@ -114,9 +138,17 @@ This function is called by `org-babel-execute-src= -block'."
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0&= quot;hline"))))
> =C2=A0 =C2=A0 =C2=A0(mapcar
> =C2=A0 =C2=A0 =C2=A0 (lambda (pair)
> - =C2=A0 =C2=A0 =C2=A0 (format "%s=3D%s"
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(car pair)
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(org-babel-sh-var-to-sh (cd= r pair) sep hline)))
> + =C2=A0 =C2=A0 =C2=A0 (if (and (string=3D org-babel-sh-command "= bash") (listp (cdr pair)))
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 (if (listp (car (cdr pair)))
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-variable-assignments:b= ash_associative
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (car pair) (cdr pair) sep hline)<= br> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-variable-assignments:b= ash_array
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (car pair) (cdr pair) sep) hline)=
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 (format "%s=3D%s"
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 (car pair)
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-sh-var-to-sh (cdr pair) sep h= line))
> + =C2=A0 =C2=A0 =C2=A0 )
> + =C2=A0 =C2=A0 )
> =C2=A0 =C2=A0 =C2=A0 (mapcar #'cdr (org-babel-get-header params :v= ar)))))
>
> =C2=A0(defun org-babel-sh-var-to-sh (var &optional sep hline)

--
Eric Schulte
https://cs.unm.e= du/~eschulte
PGP: 0x614CA05D



-- --paf
--001a1136a95a04001e04f59e73f8--