Is reading nested simple lists into org-babel code blocks currently supported?

Is reading nested simple lists into org-babel code blocks currently supported?

[James Boyle]

Hi all,

Since this is a bit of a long post, I've put my questions and summary
comment at the top, with the detailed context below.

1. What is the level of support for reading nested simple lists into
   org-babel code blocks?
2. When the manual (see link below) says that they are not supported,
   is it known if it meant "not at all supported," or, that support is
   currently language-dependent?
3. Directionally, does the org-mode project care to support nested
   simple lists?

I'm happy to submit patches for both code and manual in any event, but
wanted to align on if there are goals for supporting reading nested
lists, and if so, at what level.

At a minimum, I think updating the manual to be more precise would be
great.

Thanks for your any assistance you can provide.

Details:

I was going through the manual on this page:

https://orgmode.org/manual/Environment-of-a-Code-Block.html#Environment-of-a-Code-Block

and testing out the example, "A simple named list."

Contrary to this line in the manual:

> Note that only the top level list items are passed along. Nested list items are ignored.

It looks like the example org-babel block *does* work for nested
lists, depending upon the language. I am using org-mode 9.4.

Here is the org block verbatim from the manual, with the
results that are supposed to occur:

#+NAME: example-list
- simple
  - not
  - nested
- list

#+BEGIN_SRC emacs-lisp :var x=example-list
  (print x)
#+END_SRC

#+RESULTS:
| simple | list |


And here is the same block, when I evaluate it locally, with the
actual results, showing that unordered, nested lists are not ignored.

#+NAME: example-list
- simple
  - not
  - nested
- list

#+BEGIN_SRC emacs-lisp :var x=example-list
  (print x)
#+END_SRC

#+RESULTS:
| simple | (unordered (not) (nested)) |
| list   |                            |


I thought this was a nice surprise, until I noticed that whether or
not it works is language-dependent.

In ruby, I get an error:

#+begin_src ruby :var x=example-list
  # This is the error I get:

  # `main': undefined local variable or method `unordered' for
  # main:Object (NameError)
  puts x
#+end_src

#+RESULTS:

And here is the equivalent error in js. Presumably other languages
have variable support for this feature.

#+begin_src js :var x=example-list
  // Below is a partial error trace:

  // /private/var/folders/mc/jylgk04j5r1f96hcsd3ywm8h0000gn/T/babel-shSmEq/js-script-xzngmd:2
  // var x=[["simple", [unordered, ["not"], ["nested"]]], ["list"]]; ^
  // ReferenceError: unordered is not defined
  return x
#+end_src

#+RESULTS:

The error can be trivially "fixed" by some code like this, or making
the equivalent change in the actual source line to output a string
rather than a symbol. Changing L#2365 linked below to use "unordered"
(string) rather than 'unordered (symbol) seemed to fix it, and
produced no new test failures for me.

#+begin_src emacs-lisp
  ; See this line for where 'unordered is being spat out:
  ; https://code.orgmode.org/bzg/org-mode/src/master/lisp/ob-core.el#L2365
  (defun patch-get-vars-output (vars)
    "Substitute special Lisp symbols with a Lisp keyword.
  This allows Ruby to not barf on an undefined variable error
  for certain input structures like nested lists.
  Argument VARS is a tree structure.  See above example format."
    (subst "unordered" 'unordered vars))

  (advice-add 'org-babel--get-vars :filter-return #'patch-get-vars-output)
#+end_src

Thanks again for any help or direction you can provide.

James Boyle

Re: Is reading nested simple lists into org-babel code blocks currently supported?

[Tom Gillespie]

Hi James,
    I would file this along with other ob-lang dependent features such
as TRAMP remote execution support. For example, python works as in
your other examples, but only if you define :var unordered="unordered"
as another variable before the variable you pass the plain list to
because python tries to dereference the symbol. A global solution like
the one you propose breaks the semantics in cases where a language
does correctly deal with quoted lists (e.g. common lisp). As a result
each ob-lang will probably have to deal with this itself, or at least
they will have to define how to consume quoted elisp lists without
raising errors. That said, I'm not sure that org babel requires that
ob-lang implementations handle this. Maybe it should? I've added this
to my growing list of issues related to org babel regularization.
Best!
Tom

On Thu, Oct 29, 2020 at 7:25 AM James Boyle <jboyle@fastmail.com> wrote:
>
> Hi all,
>
> Since this is a bit of a long post, I've put my questions and summary
> comment at the top, with the detailed context below.
>
> 1. What is the level of support for reading nested simple lists into
>    org-babel code blocks?
> 2. When the manual (see link below) says that they are not supported,
>    is it known if it meant "not at all supported," or, that support is
>    currently language-dependent?
> 3. Directionally, does the org-mode project care to support nested
>    simple lists?
>
> I'm happy to submit patches for both code and manual in any event, but
> wanted to align on if there are goals for supporting reading nested
> lists, and if so, at what level.
>
> At a minimum, I think updating the manual to be more precise would be
> great.
>
> Thanks for your any assistance you can provide.
>
> Details:
>
> I was going through the manual on this page:
>
> https://orgmode.org/manual/Environment-of-a-Code-Block.html#Environment-of-a-Code-Block
>
> and testing out the example, "A simple named list."
>
> Contrary to this line in the manual:
>
> > Note that only the top level list items are passed along. Nested list items are ignored.
>
> It looks like the example org-babel block *does* work for nested
> lists, depending upon the language. I am using org-mode 9.4.
>
> Here is the org block verbatim from the manual, with the
> results that are supposed to occur:
>
> #+NAME: example-list
> - simple
>   - not
>   - nested
> - list
>
> #+BEGIN_SRC emacs-lisp :var x=example-list
>   (print x)
> #+END_SRC
>
> #+RESULTS:
> | simple | list |
>
>
> And here is the same block, when I evaluate it locally, with the
> actual results, showing that unordered, nested lists are not ignored.
>
> #+NAME: example-list
> - simple
>   - not
>   - nested
> - list
>
> #+BEGIN_SRC emacs-lisp :var x=example-list
>   (print x)
> #+END_SRC
>
> #+RESULTS:
> | simple | (unordered (not) (nested)) |
> | list   |                            |
>
>
> I thought this was a nice surprise, until I noticed that whether or
> not it works is language-dependent.
>
> In ruby, I get an error:
>
> #+begin_src ruby :var x=example-list
>   # This is the error I get:
>
>   # `main': undefined local variable or method `unordered' for
>   # main:Object (NameError)
>   puts x
> #+end_src
>
> #+RESULTS:
>
> And here is the equivalent error in js. Presumably other languages
> have variable support for this feature.
>
> #+begin_src js :var x=example-list
>   // Below is a partial error trace:
>
>   // /private/var/folders/mc/jylgk04j5r1f96hcsd3ywm8h0000gn/T/babel-shSmEq/js-script-xzngmd:2
>   // var x=[["simple", [unordered, ["not"], ["nested"]]], ["list"]]; ^
>   // ReferenceError: unordered is not defined
>   return x
> #+end_src
>
> #+RESULTS:
>
> The error can be trivially "fixed" by some code like this, or making
> the equivalent change in the actual source line to output a string
> rather than a symbol. Changing L#2365 linked below to use "unordered"
> (string) rather than 'unordered (symbol) seemed to fix it, and
> produced no new test failures for me.
>
> #+begin_src emacs-lisp
>   ; See this line for where 'unordered is being spat out:
>   ; https://code.orgmode.org/bzg/org-mode/src/master/lisp/ob-core.el#L2365
>   (defun patch-get-vars-output (vars)
>     "Substitute special Lisp symbols with a Lisp keyword.
>   This allows Ruby to not barf on an undefined variable error
>   for certain input structures like nested lists.
>   Argument VARS is a tree structure.  See above example format."
>     (subst "unordered" 'unordered vars))
>
>   (advice-add 'org-babel--get-vars :filter-return #'patch-get-vars-output)
> #+end_src
>
> Thanks again for any help or direction you can provide.
>
> James Boyle
>

Re: Is reading nested simple lists into org-babel code blocks currently supported? [PATCH] - manual.org

[James Boyle]

[-- Attachment #1.1: Type: text/plain, Size: 5602 bytes --]

Hi Tom (orgmode list cc'ed),

> As a result
> each ob-lang will probably have to deal with this itself, or at least
> they will have to define how to consume quoted elisp lists without
> raising errors. 

Thanks -- this is exactly the judgment check I was looking for -- if this type of support should fall on individual languages
or not. 

To this end, I've attached a TINYCHANGE patch for manual.org that would have helped me in my confusion. The change updates the example to reflect the current state of the output, and clarifies the level of support.

Thanks for any help you can provide.

James

On Thu, Oct 29, 2020, at 12:52 PM, Tom Gillespie wrote:
> Hi James,
>     I would file this along with other ob-lang dependent features such
> as TRAMP remote execution support. For example, python works as in
> your other examples, but only if you define :var unordered="unordered"
> as another variable before the variable you pass the plain list to
> because python tries to dereference the symbol. A global solution like
> the one you propose breaks the semantics in cases where a language
> does correctly deal with quoted lists (e.g. common lisp). As a result
> each ob-lang will probably have to deal with this itself, or at least
> they will have to define how to consume quoted elisp lists without
> raising errors. That said, I'm not sure that org babel requires that
> ob-lang implementations handle this. Maybe it should? I've added this
> to my growing list of issues related to org babel regularization.
> Best!
> Tom
> 
> On Thu, Oct 29, 2020 at 7:25 AM James Boyle <jboyle@fastmail.com> wrote:
> >
> > Hi all,
> >
> > Since this is a bit of a long post, I've put my questions and summary
> > comment at the top, with the detailed context below.
> >
> > 1. What is the level of support for reading nested simple lists into
> >    org-babel code blocks?
> > 2. When the manual (see link below) says that they are not supported,
> >    is it known if it meant "not at all supported," or, that support is
> >    currently language-dependent?
> > 3. Directionally, does the org-mode project care to support nested
> >    simple lists?
> >
> > I'm happy to submit patches for both code and manual in any event, but
> > wanted to align on if there are goals for supporting reading nested
> > lists, and if so, at what level.
> >
> > At a minimum, I think updating the manual to be more precise would be
> > great.
> >
> > Thanks for your any assistance you can provide.
> >
> > Details:
> >
> > I was going through the manual on this page:
> >
> > https://orgmode.org/manual/Environment-of-a-Code-Block.html#Environment-of-a-Code-Block
> >
> > and testing out the example, "A simple named list."
> >
> > Contrary to this line in the manual:
> >
> > > Note that only the top level list items are passed along. Nested list items are ignored.
> >
> > It looks like the example org-babel block *does* work for nested
> > lists, depending upon the language. I am using org-mode 9.4.
> >
> > Here is the org block verbatim from the manual, with the
> > results that are supposed to occur:
> >
> > #+NAME: example-list
> > - simple
> >   - not
> >   - nested
> > - list
> >
> > #+BEGIN_SRC emacs-lisp :var x=example-list
> >   (print x)
> > #+END_SRC
> >
> > #+RESULTS:
> > | simple | list |
> >
> >
> > And here is the same block, when I evaluate it locally, with the
> > actual results, showing that unordered, nested lists are not ignored.
> >
> > #+NAME: example-list
> > - simple
> >   - not
> >   - nested
> > - list
> >
> > #+BEGIN_SRC emacs-lisp :var x=example-list
> >   (print x)
> > #+END_SRC
> >
> > #+RESULTS:
> > | simple | (unordered (not) (nested)) |
> > | list   |                            |
> >
> >
> > I thought this was a nice surprise, until I noticed that whether or
> > not it works is language-dependent.
> >
> > In ruby, I get an error:
> >
> > #+begin_src ruby :var x=example-list
> >   # This is the error I get:
> >
> >   # `main': undefined local variable or method `unordered' for
> >   # main:Object (NameError)
> >   puts x
> > #+end_src
> >
> > #+RESULTS:
> >
> > And here is the equivalent error in js. Presumably other languages
> > have variable support for this feature.
> >
> > #+begin_src js :var x=example-list
> >   // Below is a partial error trace:
> >
> >   // /private/var/folders/mc/jylgk04j5r1f96hcsd3ywm8h0000gn/T/babel-shSmEq/js-script-xzngmd:2
> >   // var x=[["simple", [unordered, ["not"], ["nested"]]], ["list"]]; ^
> >   // ReferenceError: unordered is not defined
> >   return x
> > #+end_src
> >
> > #+RESULTS:
> >
> > The error can be trivially "fixed" by some code like this, or making
> > the equivalent change in the actual source line to output a string
> > rather than a symbol. Changing L#2365 linked below to use "unordered"
> > (string) rather than 'unordered (symbol) seemed to fix it, and
> > produced no new test failures for me.
> >
> > #+begin_src emacs-lisp
> >   ; See this line for where 'unordered is being spat out:
> >   ; https://code.orgmode.org/bzg/org-mode/src/master/lisp/ob-core.el#L2365
> >   (defun patch-get-vars-output (vars)
> >     "Substitute special Lisp symbols with a Lisp keyword.
> >   This allows Ruby to not barf on an undefined variable error
> >   for certain input structures like nested lists.
> >   Argument VARS is a tree structure.  See above example format."
> >     (subst "unordered" 'unordered vars))
> >
> >   (advice-add 'org-babel--get-vars :filter-return #'patch-get-vars-output)
> > #+end_src
> >
> > Thanks again for any help or direction you can provide.
> >
> > James Boyle
> >
> 

[-- Attachment #1.2: Type: text/html, Size: 8957 bytes --]

[-- Attachment #2: 0001-org-manual.org-Clarify-org-babel-nested-list-support.patch --]
[-- Type: application/octet-stream, Size: 1118 bytes --]

From 745de2927a9619effbd35699a74b88a399e7151a Mon Sep 17 00:00:00 2001
From: James Boyle <jhhboyle@gmail.com>
Date: Fri, 30 Oct 2020 15:03:12 -0400
Subject: [PATCH] org-manual.org: Clarify org-babel nested list support

* doc/org-manual.org (Environment of a code block): Clarify
the behavior for nested lists in org-babel code blocks
---
 doc/org-manual.org | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index ef2dad9ef..d6bfeef6a 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16824,11 +16824,13 @@ Here are examples of passing values by reference:
   ,#+END_SRC
 
   ,#+RESULTS:
-  | simple | list |
+  | simple | (unordered (not) (nested)) |
+  | list   |                            |
   #+end_example
 
-  Note that only the top level list items are passed along.  Nested
-  list items are ignored.
+  Note that the nested list items are passed along, along with the
+  list type of the first element of the nested list. Support for
+  nested lists can vary by source language.
 
 - code block without arguments ::
 
-- 
2.28.0