[RFC] [PATCH] [babel] read description lists as lists of lists

[RFC] [PATCH] [babel] read description lists as lists of lists

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]


Hello all,

The attached patch makes babel read description lists as lists of the
following format: (("term" "description") ...).  The present default is
to simply read in the text of each list item, yielding:
("term :: description" ...).

Of course, it’s possible to interconvert between the two formats, but I
think the greater structure of this proposal makes things easier for
babel authors.  (Another way of thinking of the proposal is that it
treats description lists like two-column tables.)

What do people think?

Thanks,

-- 
Aaron Ecay

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core.el-Read-description-lists-as-lisp-lists.patch --]
[-- Type: text/x-diff, Size: 1577 bytes --]

From a7e01675f2c89fb648e528c3efe535ed0b2389f1 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Fri, 19 Sep 2014 14:39:26 -0400
Subject: [PATCH] ob-core.el: Read description lists as lisp lists.

* lisp/ob-core.el (org-babel-read-list): Read description lists as
lisp lists.

This allows description lists to be used as structured input to a
babel block.
---
 lisp/ob-core.el | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index e01c4d2..f1661cb 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2035,9 +2035,21 @@ following the source block."
           (org-table-to-lisp)))
 
 (defun org-babel-read-list ()
-  "Read the list at `point' into emacs-lisp."
-  (mapcar (lambda (el) (org-babel-read el 'inhibit-lisp-eval))
-	  (mapcar #'cadr (cdr (org-list-parse-list)))))
+  "Read the list at `point' into emacs-lisp.
+
+The result is a list of strings \(the list items), unless the
+input list is a description list.  In that case, the result will
+be a list of lists; each of the latter lists will have two
+elements: the term and the description."
+  (let* ((parsed (org-list-parse-list))
+	 (elements (mapcar #'cadr (cdr parsed))))
+    (if (eq (car parsed) 'descriptive)
+	(mapcar (lambda (el)
+		  (let ((s (split-string el " :: ")))
+		    (list (nth 0 s) (mapconcat #'identity (cdr s) " :: "))))
+		elements)
+      (mapcar (lambda (el) (org-babel-read el 'inhibit-lisp-eval))
+	      elements))))
 
 (defvar org-link-types-re)
 (defun org-babel-read-link ()
-- 
2.1.0

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Charles Berry]

Aaron Ecay <aaronecay <at> gmail.com> writes:

> 
> 
> Hello all,
> 
> The attached patch makes babel read description lists as lists of the
> following format: (("term" "description") ...).  The present default is
> to simply read in the text of each list item, yielding:
> ("term :: description" ...).
> 
> Of course, it’s possible to interconvert between the two formats, but I
> think the greater structure of this proposal makes things easier for
> babel authors.  (Another way of thinking of the proposal is that it
> treats description lists like two-column tables.)
> 
> What do people think?

With that change 

#+BEGIN_SRC R :var a=my-desc-list
a
#+END_SRC

returns a data.frame with one column of terms and one of descriptions!

Which, no doubt, is what you were thinking.

This will be handy for creating DESCRIPTION files for R packages.

I love it. 

FWIW, rgrep-ing org-babel-\(ref-resolve\)\|\(read-result\) and clicking
on the links, I didn't catch any obvious problem cases.

HTH,

Chuck

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> The attached patch makes babel read description lists as lists of the
> following format: (("term" "description") ...).  The present default is
> to simply read in the text of each list item, yielding:
> ("term :: description" ...).

Thank you.

> Of course, it’s possible to interconvert between the two formats, but I
> think the greater structure of this proposal makes things easier for
> babel authors.  (Another way of thinking of the proposal is that it
> treats description lists like two-column tables.)
>
> What do people think?

The problem I see here is that you're introducing yet another internal
representation for lists (along with element's and
org-list-parse-list's). Worse, it can only be discovered when reading
the docstring of a Babel internal function and will only benefit to
Babel.

If this new internal representation is better than current one, by all
means, improve `org-list-parse-list', and document it in

  (info "(org) Radio lists")

This is more work, but, IMO, it is also the only sane way to proceed.

> +THE result is a list of strings \(the list items), unless the

You only need to escape parenthesis at the beginning of a line.

> +	(mapcar (lambda (el)
> +		  (let ((s (split-string el " :: ")))
> +		    (list (nth 0 s) (mapconcat #'identity (cdr s) " :: "))))

This is really awkward. You can use a regexp to extract the tag.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Aaron Ecay]

Hi Nicolas,

Thanks for your feedback.

2014ko irailak 20an, Nicolas Goaziou-ek idatzi zuen:
> 
> The problem I see here is that you're introducing yet another internal
> representation for lists (along with element's and
> org-list-parse-list's). Worse, it can only be discovered when reading
> the docstring of a Babel internal function and will only benefit to
> Babel.
> 
> If this new internal representation is better than current one, by all
> means, improve `org-list-parse-list', and document it in
> 
>   (info "(org) Radio lists")
> 
> This is more work, but, IMO, it is also the only sane way to proceed.

Indeed.  The internals of org-list are not pretty.  org-list-parse-list
has few callers:
- org-list-make-subtree
- org-babel-read-list
- org-toggle-heading

org-list-to-subtree has only two callers:
- org-list-make-subtree
- org-toggle-heading

org-list-to-generic also has only two callers:
- org-babel-insert-result
- org-list-to-subtree

I think I can remove these three functions (-parse-list, -to-subtree,
and -to-generic), and rewrite their callers to use org-element.  Thus,
the org-list-parse-list format would be eradicated from the code base
incl. contrib (AFAICT).  Can I do that, or do I need to care about
preserving backwards compatibility with external callers of these
functions?  If backwards compatibility must be preserved, may I mark
these functions as deprecated and what is the minimum period (measured
in calendar time and/or org versions) that should pass before their
removal?

The babel feature is compelling to me (and I guess Chuck) on its
own.  It’s familiar (e.g. in the case of tables) that babel gets to
have its own data format for org elements.  I’m happy to undertake
the above-described demolition job on org-list-parse-list in order
to offset the added complexity from the babel change (we can call it
a cap-and-trade system).  But given that org-list-parse-list is a
marginal part of the code base – and perhaps moribund in the era of
org-element – I don’t really think it’s worth it (to me) to try and
engineer an improvement to it in order to enable the babel feature.

WDYT?

-- 
Aaron Ecay

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> I think I can remove these three functions (-parse-list, -to-subtree,
> and -to-generic), and rewrite their callers to use org-element.  Thus,
> the org-list-parse-list format would be eradicated from the code base
> incl. contrib (AFAICT).  Can I do that, or do I need to care about
> preserving backwards compatibility with external callers of these
> functions?  If backwards compatibility must be preserved, may I mark
> these functions as deprecated and what is the minimum period (measured
> in calendar time and/or org versions) that should pass before their
> removal?

You cannot do that. This is not about backwards compatibility.

`org-list-parse-list' generates an easy to produce and work on internal
representation for lists (similar to what `org-table-to-lisp' does for
tables). `org-list-to-generic' is used for radio lists (similar to
`org-table-to-generic'): it is expected to consume
a `org-list-parse-list'-like return value.

IOW both functions are important and are not meant to be replaced by
Elements (however, at some point `org-list-to-generic' should use
"ox.el", but that's for another day).

Note that since `org-list-parse-list' is meant for extraneous buffer, it
cannot rely on Elements. It shouldn't even use `org-list-struct' because
I plan to make this function use Elements, too.

> The babel feature is compelling to me (and I guess Chuck) on its
> own.  It’s familiar (e.g. in the case of tables) that babel gets to
> have its own data format for org elements.

It's the same for lists. Internal representation for lists should come
from "org-list.el", not from Babel. Internal representation for tables
comes from "org-table.el", too.

> I’m happy to undertake the above-described demolition job on
> org-list-parse-list in order to offset the added complexity from the
> babel change (we can call it a cap-and-trade system). But given that
> org-list-parse-list is a marginal part of the code base and perhaps
> moribund in the era of org-element

I don't consider radio lists moribund. Are they?

> I don’t really think it’s worth it (to me) to try and engineer an
> improvement to it in order to enable the babel feature.

It is not (or should not be) a Babel feature.

Anyway, it's not about rewriting `org-list-parse-list', but if Babel
understands a new representation for plain lists, this function should
be able to generate it and `org-list-to-generic' should be able to
interpret it.

Could you detail the exact specifications of the suggested internal
plain list representation?


Regards,

-- 
Nicolas Goaziou

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Aaron Ecay]

Hi Nicolas,

Thanks for the discussion.

2014ko irailak 24an, Nicolas Goaziou-ek idatzi zuen:
> 
> You cannot do that. This is not about backwards compatibility.
> 
> `org-list-parse-list' generates an easy to produce and work on internal
> representation for lists (similar to what `org-table-to-lisp' does for
> tables). 

Isn’t the org-element format also easy to work on?  It requires a bit
more than just car and cdr, but it’s well documented and used in many
places across the code base (= cognitive burden to use is lower).  It’s
also easy to produce in the sense that org-element.el already exists for
independent reasons; we just have to use it.

> `org-list-to-generic' is used for radio lists (similar to
> `org-table-to-generic'): it is expected to consume a
> `org-list-parse-list'-like return value.

Radio lists is a feature, org-list-to-generic is an implementation.  We
can change the implementation without changing the user-visible aspects
of the feature.  IOW, nothing about the user-facing functionality of
org-list-to-generic requires it to accept a particular type of argument
(as long as that arg is some representation or other of a list).

> 
> IOW both functions are important and are not meant to be replaced by
> Elements (however, at some point `org-list-to-generic' should use
> "ox.el", but that's for another day).

...doesn’t using ox.el entail using elements?

> 
> Note that since `org-list-parse-list' is meant for extraneous buffer, it
> cannot rely on Elements. It shouldn't even use `org-list-struct' because
> I plan to make this function use Elements, too.

One approach would be to detect when it’s called from a non-org-mode
buffer, and copy the text into a temporary org-mode buffer for parsing.
Then org-element would be available.

> 
>> The babel feature is compelling to me (and I guess Chuck) on its
>> own.  It’s familiar (e.g. in the case of tables) that babel gets to
>> have its own data format for org elements.
> 
> It's the same for lists. Internal representation for lists should come
> from "org-list.el", not from Babel. Internal representation for tables
> comes from "org-table.el", too.

Hmm.  I had missed that, you are correct.

> 
>> I’m happy to undertake the above-described demolition job on
>> org-list-parse-list in order to offset the added complexity from the
>> babel change (we can call it a cap-and-trade system). But given that
>> org-list-parse-list is a marginal part of the code base and perhaps
>> moribund in the era of org-element
> 
> I don't consider radio lists moribund. Are they?

IDK.  You’re probably in a better position to know that than I am.  There’s
only one message even mentioning them (very tangentially) in my 2-ish years
of messages from the list: <http://mid.gmane.org/87obc6scty.fsf@pank.eu>.
I’m not advocating their removal or deprecation, but they certainly seem
like the tail and not the dog when considering what parts of org ought to
wag what others.

(I’d hope their usefulness would eventually naturally wane as org
becomes compelling enough that people commit to it wholesale, rather
than relying just on the list editing features while living in another
document composition regime.  Whether it’s worth keeping them around
as a sort of training wheels I don’t really have an opinion on.)

> 
>> I don’t really think it’s worth it (to me) to try and engineer an
>> improvement to it in order to enable the babel feature.
> 
> It is not (or should not be) a Babel feature.
> 
> Anyway, it's not about rewriting `org-list-parse-list', but if Babel
> understands a new representation for plain lists, this function should
> be able to generate it and `org-list-to-generic' should be able to
> interpret it.

Why?  Babel’s representation is for babel.
org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
said this connection seems accidental rather than essential).  Babel
calls org-list-parse-list, but I don’t see why it should be forbidden
from doing more processing on the result before passing it along
(indeed, it already does some processing to remove the list type
indicators, remove nested structure, etc.).

> 
> Could you detail the exact specifications of the suggested internal
> plain list representation?

I dunno if I’d call my proposal an “internal plain list representation,”
but rather “babel’s interpretation of plain lists.”

Ordered and unordered lists are lists of strings (exactly as now).
Description lists are lists of 2-element lists, each of the form
(“TERM” “DESCRIPTION”) (unlike now, when they are lists of strings of
the form “TERM :: DESCRIPTION”).

It might be nice to handle nested lists somehow, if a sensible design
can be created, but it looks like babel just discards them currently.
So I propose to leave this unchanged, for the present at least:

#+name: data
- foo
- bar
  - baz
  - quux
- abc

#+begin_src emacs-lisp :var data=data
(pp-to-string data)
#+end_src

#+RESULTS:
: ("foo" "bar" "abc")

-- 
Aaron Ecay

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> Isn’t the org-element format also easy to work on?  It requires a bit
> more than just car and cdr, but it’s well documented and used in many
> places across the code base (= cognitive burden to use is lower).  It’s
> also easy to produce in the sense that org-element.el already exists for
> independent reasons; we just have to use it.

It is not as easy to produce ex nihilo, i.e., without any Org syntax
under point. But, really, I do not mind if both radio lists and Babel
move to this internal syntax. It will require much more work, though.

Also, it doesn't mean we can remove or replace `org-list-parse-list' and
`org-list-to-generic'.

> Radio lists is a feature, org-list-to-generic is an implementation.  We
> can change the implementation without changing the user-visible aspects
> of the feature.  IOW, nothing about the user-facing functionality of
> org-list-to-generic requires it to accept a particular type of argument
> (as long as that arg is some representation or other of a list).

I agree.

> One approach would be to detect when it’s called from a non-org-mode
> buffer, and copy the text into a temporary org-mode buffer for parsing.
> Then org-element would be available.

Of course, if the internal representation is changed to Elements', that
is probably the way to go.

> IDK.  You’re probably in a better position to know that than I am.  There’s
> only one message even mentioning them (very tangentially) in my 2-ish years
> of messages from the list: <http://mid.gmane.org/87obc6scty.fsf@pank.eu>.
> I’m not advocating their removal or deprecation, but they certainly seem
> like the tail and not the dog when considering what parts of org ought to
> wag what others.

I think you are missing my point.

Again, I'm fine with any improvement needed for Babel, but other, even
remotely, related parts should be moved along. This is about
consistency. I certainly don't want to see various parts of Org drift
away. Or, to put it differently: mind the tail, do not act as if the dog
had none.

> Why?  Babel’s representation is for babel.

Which I strongly frown upon.

> org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
> said this connection seems accidental rather than essential).  Babel
> calls org-list-parse-list, but I don’t see why it should be forbidden
> from doing more processing on the result before passing it along
> (indeed, it already does some processing to remove the list type
> indicators, remove nested structure, etc.).

It is best to use as much common ground as possible. We should strive to
decrease need for such processing, not the other way.

As I already stated in my first answer, in the long run, it is the only
sane way to proceed. I agree it is less work to simply tweak Babel right
now and ignore the whole Org ecosystem, but it does no good to Org as
a whole.

> I dunno if I’d call my proposal an “internal plain list representation,”
> but rather “babel’s interpretation of plain lists.”

See above.

> Ordered and unordered lists are lists of strings (exactly as now).
> Description lists are lists of 2-element lists, each of the form
> (“TERM” “DESCRIPTION”) (unlike now, when they are lists of strings of
> the form “TERM :: DESCRIPTION”).
>
> It might be nice to handle nested lists somehow, if a sensible design
> can be created, but it looks like babel just discards them currently.
> So I propose to leave this unchanged, for the present at least:

`org-list-parse-list' handles nested lists just fine. Another advantage
of not re-inventing the wheel in every part of Org.


Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Aaron Ecay]

Hi Nicolas,

2014ko irailak 26an, Nicolas Goaziou-ek idatzi zuen:
> 
>> Why?  Babel’s representation is for babel.
> 
> Which I strongly frown upon.

Let’s back up a step.  The representation I am targeting with my change
is what babel uses to ship a list off as input to code in a babel block.
This code could be emacs lisp, but it could also be R, python, etc.  So
the question is, how to provide a consistent language-agnostic view of
org structure to other languages.  The resultant structure doesn’t hang
around inside babel, it just gets handed off to a code block.

> 
>> org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
>> said this connection seems accidental rather than essential).  Babel
>> calls org-list-parse-list, but I don’t see why it should be forbidden
>> from doing more processing on the result before passing it along
>> (indeed, it already does some processing to remove the list type
>> indicators, remove nested structure, etc.).
> 
> It is best to use as much common ground as possible. We should strive to
> decrease need for such processing, not the other way.
> 
> As I already stated in my first answer, in the long run, it is the only
> sane way to proceed. I agree it is less work to simply tweak Babel right
> now and ignore the whole Org ecosystem, but it does no good to Org as
> a whole.

It’s not work that I’m afraid of: I offered to rewrite both babel and
radio lists in terms of org-elements.  Maybe I am insane, as you imply.

What I’m afraid of is old and disused sort-of-APIs like
org-list-parse-list calcifying and preventing good things from happening
to parts of org that people actually use.

What if I rewrote org-babel-read-list in terms of org-elements?  That
would satisfy me wrt. babel, and wouldn’t necessitate disturbing
org-list-parse-list, radio lists, or indeed anything outside of babel.

> 
> `org-list-parse-list' handles nested lists just fine. Another advantage
> of not re-inventing the wheel in every part of Org.

I know.  But babel’s processing of parse-list’s output strips the nested
structure:

#+name: a-list
- foo
- bar
  - abc
  - def
- baz

#+begin_src emacs-lisp :var lst=a-list
  (pp-to-string lst)
#+end_src

#+RESULTS:
: ("foo" "bar" "baz")

That’s because it’s hard to come up with a good representation of a nested
list in a language-agnostic way.  “List of strings” is a straightforward
datatype in every language babel supports, but not all of them have a
convenient “labeled n-ary tree with string leaves” (which you’d need for
arbitrarily nested lists).

Thanks,

-- 
Aaron Ecay

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Thorsten Jolitz]

Aaron Ecay <aaronecay@gmail.com> writes:

Hi Aaron,

> So the question is, how to provide a consistent language-agnostic view
> of org structure to other languages. 

well, that would be the parse-tree normally, its a nested list
containing all info about org structure. 

I tried to make two Lisps talk to each other (Emacs Lisp and PicoLisp),
and initially thought it would be easy - just send lists back and
forth. But when these lists are parse-trees produced by the Org parser
framework, its not that straight-forward anymore, because these lists
are send as strings and then read again by the other side, and there are
lots of peculiarities in the string representation of Org parse-trees
that make other languages choke, even if they otherwise understand lists
very well.

As an example, this nested list representation of this "created"
headline is simple enough, PicoLisp can read it "as-is":

#+NAME: hl1
#+BEGIN_SRC emacs-lisp :results verbatim
  (org-dp-create 'headline "Hallo World" 'data '(:name "foo")
                 :level 2
                 :title "2nd level"
                 :todo-keyword "NEXT"
                 :tags '("office")
                 :priority ?B)
#+END_SRC

#+results: hl1
: (headline (:level 2 :title "2nd level" :todo-keyword "NEXT" :tags
("office") :priority 66 :name "foo") (section nil "Hallo World"))

#+BEGIN_SRC emacs-lisp :var lst=hl1
 (length lst)
#+END_SRC

#+results:
: 131

#+BEGIN_SRC picolisp :var lst=hl1 :results pp
 (in NIL (length lst))
#+END_SRC

#+results:
: 131

#+BEGIN_SRC picolisp :var lst=hl1 :results pp
 (in NIL (last (car (str lst))))
#+END_SRC

#+results:
: (section nil "Hallo World")

So this can be read by PicoLisp (and probably many other list
processing languages), except that nil is not NIL in PicoLisp.

But when parsing this Org buffer its easy to see that syntax and
semantics of Emacs Lisp circular lists as well as of strings with
text-properties cannot easily be consumed by other languages (expecially
when # is their comment-start character). Some post-processing is
necessary, e.g. suppressing text properties, nil -> NIL, \n -> ^J, ^ ->
\^, # -> \# etc., and I doubt this can be done in a language agnostic
way. And then the circular relationships needs to be resolved to
something the target language can understand...

#+BEGIN_SRC emacs-lisp :results pp
 (org-element-parse-buffer)
#+END_SRC

#+results: (org-data nil (headline (:raw-value "--text follows this
line--" :begin 1 :end 17295 :pre-blank 0 :contents-begin 30
:contents-end 17295 :level 1 :priority nil :tags nil :todo-keyword nil
:todo-type nil :post-blank 0 :footnote-section-p nil :archivedp nil
:commentedp nil :post-affiliated 1 :title (#("--text follows this
line--" 0 26 (:parent #1))) :parent #0) [...cut...] (paragraph (:begin
17274 :end 17295 :contents-begin 17274 :contents-end 17295 :post-blank 0
:post-affiliated 17274 :parent #3) #("-- \ncheers,\nThorsten\n" 0 21
(:parent #4)))))

-- 
cheers,
Thorsten

Re: [RFC] [PATCH] [babel] read description lists as lists of lists

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> Let’s back up a step.  The representation I am targeting with my change
> is what babel uses to ship a list off as input to code in a babel block.
> This code could be emacs lisp, but it could also be R, python, etc.  So
> the question is, how to provide a consistent language-agnostic view of
> org structure to other languages.  The resultant structure doesn’t hang
> around inside babel, it just gets handed off to a code block.

I know. But another internal representation is an additional maintenance
burden.

> It’s not work that I’m afraid of: I offered to rewrite both babel and
> radio lists in terms of org-elements.  Maybe I am insane, as you imply.

I never wrote, implied or even thought you were insane.

> What if I rewrote org-babel-read-list in terms of org-elements?  That
> would satisfy me wrt. babel, and wouldn’t necessitate disturbing
> org-list-parse-list, radio lists, or indeed anything outside of babel.

That is even better.


Regards,

-- 
Nicolas Goaziou