Re: Export arrays for 'sh' code blocks when using bash

[Pascal Fleury <fleury@google.com>]

[-- Attachment #1.1: Type: text/plain, Size: 818 bytes --]

Hello,

Great, thanks for the guidance. I hope I managed it all correctly.

On Fri, Apr 11, 2014 at 11:33 AM, Bastien <bzg@gnu.org> wrote:

> Hi Eric and Pascal,
>
> Eric Schulte <schulte.eric@gmail.com> writes:
>
> > Also, I think the google-wide copyright stuff is sorted out.
>
> Yes it is: we can accept patch from employees of Google, Inc.
>
>
Good :-)


> Pascal, I guess it's safe to assume anyone with a @google.com
> email address is a Google employee -- let me know if it's not
> the case.
>
>
Yes, I checked internally, and this is a safe assumption.


> Also, if you can sign your patches (git format-patch -s) that'd
> be even better, but not mandatory.
>
>
I did, also wrote the description of the patch according to the rules I
found on orgmode.org


> Thanks!
>
> --
>  Bastien
>


Best regards,
--paf

[-- Attachment #1.2: Type: text/html, Size: 1922 bytes --]

[-- Attachment #2: 0001-ob-shell.el-export-vars-as-arrays-for-sh-code-blocks.patch --]
[-- Type: text/x-patch, Size: 8463 bytes --]

From c61c28f0b97544a12c3f89180b309cb25ed9f3a9 Mon Sep 17 00:00:00 2001
From: Pascal Fleury <fleury@google.com>
Date: Fri, 11 Apr 2014 23:27:02 +0200
Subject: [PATCH] ob-shell.el: export vars as arrays for 'sh' code blocks

* lisp/ob-shell.el: added support to serialize vars as arrays or associative arrays as appropriate if it is using bash.
* testing/examples/ob-shell-test.org: a file containing a few code blocks both illustrating the use of arrays as well as serving as test for the new export functionality.
* testing/lisp/test-ob-shell.el: added a few unit tests that verify that this new logic only triggers for bash and no other shell at this time.

When variables are defined in a 'sh' code block, they are exported as strings. when the variable itself is an array or a table, then we simply get a shell variable that contains the list of all values in a non-structured form.
When calling the code block with bash, however, it will now export the list as an array, the table as an associative array. A scalar is exported the same way as before.

Signed-off-by: Pascal Fleury <fleury@google.com>
---
 lisp/ob-shell.el                   | 51 ++++++++++++++++++++--
 testing/examples/ob-shell-test.org | 88 ++++++++++++++++++++++++++++++++++++++
 testing/lisp/test-ob-shell.el      | 39 +++++++++++++++++
 3 files changed, 174 insertions(+), 4 deletions(-)
 create mode 100644 testing/examples/ob-shell-test.org

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 3ede701..d7f1802 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -105,6 +105,44 @@ This function is called by `org-babel-execute-src-block'."
       buffer)))
 
 ;; helper functions
+(defun org-babel-variable-assignments:generic (varname values &optional sep hline)
+  "Returns a list of statements declaring the values as a generic variable."
+  (format "%s=%s" varname (org-babel-sh-var-to-sh values sep hline)))
+
+(defun org-babel-variable-assignments:bash_array (varname values &optional sep hline)
+  "Returns a list of statements declaring the values as a bash array."
+  (format "unset %s\ndeclare -a %s=( \"%s\" )"
+     varname varname
+     (mapconcat 'identity
+       (mapcar
+         (lambda (value) (org-babel-sh-var-to-sh value sep hline))
+         values)
+       "\" \"")))
+
+(defun org-babel-variable-assignments:bash_assoc (varname values &optional sep hline)
+  "Returns a list of statements declaring the values as bash associative array."
+  (format "unset %s\ndeclare -A %s\n%s"
+    varname varname
+    (mapconcat 'identity
+      (mapcar
+        (lambda (items)
+          (format "%s[\"%s\"]=%s"
+            varname
+            (org-babel-sh-var-to-sh (car items) sep hline)
+            (org-babel-sh-var-to-sh (cdr items) sep hline)))
+        values)
+      "\n")))
+
+(defun org-babel-variable-assignments:bash (varname values &optional sep hline)
+  "Represents the parameters as useful Bash shell variables."
+  (if (listp values)
+    (if (and (listp (car values)) (= 1 (length (car values))))
+      (org-babel-variable-assignments:bash_array varname values sep hline)
+      (org-babel-variable-assignments:bash_assoc varname values sep hline)
+    )
+    (org-babel-variable-assignments:generic varname values sep hline)
+  )
+)
 
 (defun org-babel-variable-assignments:sh (params)
   "Return list of shell statements assigning the block's variables."
@@ -114,10 +152,15 @@ This function is called by `org-babel-execute-src-block'."
 		     "hline"))))
     (mapcar
      (lambda (pair)
-       (format "%s=%s"
-	       (car pair)
-	       (org-babel-sh-var-to-sh (cdr pair) sep hline)))
-     (mapcar #'cdr (org-babel-get-header params :var)))))
+       (if (string= org-babel-sh-command "bash")
+         (org-babel-variable-assignments:bash 
+            (car pair) (cdr pair) sep hline)
+         (org-babel-variable-assignments:generic 
+	    (car pair) (cdr pair) sep hline)
+       )
+     )
+     (mapcar #'cdr (org-babel-get-header params :var))))
+)
 
 (defun org-babel-sh-var-to-sh (var &optional sep hline)
   "Convert an elisp value to a shell variable.
diff --git a/testing/examples/ob-shell-test.org b/testing/examples/ob-shell-test.org
new file mode 100644
index 0000000..a54e5c0
--- /dev/null
+++ b/testing/examples/ob-shell-test.org
@@ -0,0 +1,88 @@
+#+Title: a collection of examples for ob-shell tests
+#+OPTIONS: ^:nil
+
+* Sample data structures
+#+NAME: sample_array
+| one   |
+| two   |
+| three |
+
+#+NAME: sample_mapping_table
+| first  | one   |
+| second | two   |
+| third  | three |
+
+#+NAME: sample_big_table
+| bread     |  2 | kg |
+| spaghetti | 20 | cm |
+| milk      | 50 | dl |
+
+* Array tests
+  :PROPERTIES:
+  :ID:       0ba56632-8dc1-405c-a083-c204bae477cf
+  :END:
+** Generic shell: no arrays
+#+begin_src sh :exports results :var array=sample_array
+echo ${array}
+#+end_src
+
+#+RESULTS:
+: one two three
+
+** Bash shell: support for arrays
+Bash will see a simple indexed array. In this test, we check that the
+returned value is indeed only the first item of the array, as opposed to
+the generic serialiation that will return all elements of the array as 
+a single string.
+#+begin_src bash :exports results :var array=sample_array
+echo ${array}
+#+end_src
+
+#+RESULTS:
+: one
+
+* Associative array tests (simple map)
+  :PROPERTIES:
+  :ID:       bec1a5b0-4619-4450-a8c0-2a746b44bf8d
+  :END:
+** Generic shell: no special handing
+The shell will see all values as a single string.
+#+begin_src sh :exports results :var table=sample_mapping_table
+echo ${table}
+#+end_src
+
+#+RESULTS:
+: first one second two third three
+
+** Bash shell: support for associative arrays
+Bash will see a table that contains the first column as the 'index'
+of the associative array, and the second column as the value.
+#+begin_src bash :exports results :var table=sample_mapping_table
+echo ${table[second]}
+#+end_src
+
+#+RESULTS:
+: two
+
+* Associative array tests (more than 2 columns)
+  :PROPERTIES:
+  :ID:       82320a48-3409-49d7-85c9-5de1c6d3ff87
+  :END:
+** Generic shell: no special handing
+#+begin_src sh :exports results :var table=sample_big_table
+echo ${table}
+#+end_src
+
+#+RESULTS:
+: bread 2 kg spaghetti 20 cm milk 50 dl
+   
+** Bash shell: support for associative arrays with lists
+Bash will see an associative array that contains each row as a single
+string. Bash cannot handle lists in associative arrays.
+#+begin_src bash :exports results :var table=sample_big_table
+echo ${table[spaghetti]}
+#+end_src
+
+#+RESULTS:
+: 20 cm
+
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 2b3e48f..58a7859 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -47,6 +47,45 @@ ob-comint.el, which was not previously tested."
     (should res)
     (should (listp res))))
 
+; A list of tests using the samples in ob-shell-test.org
+(ert-deftest ob-shell/generic-uses-no-arrays ()
+  "No arrays for generic"
+  (org-test-at-id "0ba56632-8dc1-405c-a083-c204bae477cf"
+    (org-babel-next-src-block)
+    (should (equal "one two three" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-shell/bash-uses-arrays ()
+  "Bash arrays"
+  (org-test-at-id "0ba56632-8dc1-405c-a083-c204bae477cf"
+    (org-babel-next-src-block 2)
+    (should (equal "one" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-shell/generic-uses-no-assoc-arrays ()
+  "No associative arrays for generic"
+  (org-test-at-id "bec1a5b0-4619-4450-a8c0-2a746b44bf8d"
+    (org-babel-next-src-block)
+    (should (equal "first one second two third three"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-shell/bash-uses-assoc-arrays ()
+  "Bash associative arrays"
+  (org-test-at-id "bec1a5b0-4619-4450-a8c0-2a746b44bf8d"
+    (org-babel-next-src-block 2)
+    (should (equal "two" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-shell/generic-uses-no-assoc-arrays ()
+  "No associative arrays for generic"
+  (org-test-at-id "82320a48-3409-49d7-85c9-5de1c6d3ff87"
+    (org-babel-next-src-block)
+    (should (equal "bread 2 kg spaghetti 20 cm milk 50 dl"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-shell/bash-uses-assoc-arrays ()
+  "Bash associative arrays as strings for the row"
+  (org-test-at-id "82320a48-3409-49d7-85c9-5de1c6d3ff87"
+    (org-babel-next-src-block 2)
+    (should (equal "20 cm" (org-babel-execute-src-block)))))
+
 
 (provide 'test-ob-shell)
 
-- 
1.8.3.2