[BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Johan Tolö]

After the changes to ob-core.el introduced in commit: 
b4e437f968771df9555f9306467846965857f632 plain lists passed to R 
source code blocks are generated with extra (empty) items.

Example:

#+name: alist
- first item
- second item
- third item

#+begin_src R :var list=alist
  list
#+end_src

#+RESULTS:
| first item | second item | third item |   |   |   |   |   |   | 
  |


Expected result would be:

#+RESULTS:
| first item | second item | third item |

as generated by:

#+begin_src elisp :var list=alist
  list
#+end_src

#+RESULTS:
| first item | second item | third item |


Emacs  : GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ 
Version 3.24.35, cairo version 1.17.6)
 of 2022-12-01
Package: Org mode version 9.6 (release_9.6-3-ga4d38e @ 
/usr/share/emacs/30.0.50/lisp/org/)
-- 
Johan Tolö

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

i see this same behavior.  to me, (org-babel-R-assign-elisp) seems to be
the problem, but it hasn't changed any time recently.  (nor, if my =git
blame= is done correctly, has anything else in ob-R.el.)

i don't understand how (org-babel-R-assign-elisp) thinks.  it seems to
assign =max= and =min= assuming that =value= is a list of (at least one)
list(s).

this *might* be a fix.  but, i don't have a lot of confidence it will
not have bad side effects for other cases.  (though i tested it with a
simple list, and a simple table with more than one row, and a list of
two lists, the first of length 3, the second of length 2.):
----
modified   lisp/ob-R.el
@@ -241,7 +241,7 @@ This function is called by `org-babel-execute-src-block'."
 (defun org-babel-R-assign-elisp (name value colnames-p rownames-p)
   "Construct R code assigning the elisp VALUE to a variable named NAME."
   (if (listp value)
-      (let* ((lengths (mapcar 'length (cl-remove-if-not 'sequencep value)))
+      (let* ((lengths (mapcar 'length (cl-remove-if-not 'listp value)))
 	     (max (if lengths (apply 'max lengths) 0))
 	     (min (if lengths (apply 'min lengths) 0)))
         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
----

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Johan Tolö]

Greg Minshall <minshall@umich.edu> writes:

> i see this same behavior.  to me, (org-babel-R-assign-elisp) 
> seems to be
> the problem, but it hasn't changed any time recently.  (nor, if 
> my =git
> blame= is done correctly, has anything else in ob-R.el.)
[snipped]

Yeah, the problem is not a change to ob-R.el but the recent change 
to ob-core.el (see the message of the commit I referenced). But 
the change to ob-core.el is final if I understand it correctly 
which means that ob-R.el has to be updated. Sorry if I was unclear 
in my description.

Named plain lists were previously turned into data.frames with 
each first level item in one row. Second level items were in the 
second column I believe. After the change to ob-core.el they are 
still data.frames but first level items form columns instead of 
rows. Second (or deeper) level items are dropped as they should be 
after the change to ob-core.el (according to the commit message).

-- 
Johan Tolö

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Jérémie Juste]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

Hello Johan,

Many thanks for reporting the issue.
I'll investigate further and submit a patch next weekend at the latest.

Best regards,
Jeremie

On Mon, Dec 5, 2022 at 11:42 AM Johan Tolö <johan@toloe.se> wrote:

> Greg Minshall <minshall@umich.edu> writes:
>
> > i see this same behavior.  to me, (org-babel-R-assign-elisp)
> > seems to be
> > the problem, but it hasn't changed any time recently.  (nor, if
> > my =git
> > blame= is done correctly, has anything else in ob-R.el.)
> [snipped]
>
> Yeah, the problem is not a change to ob-R.el but the recent change
> to ob-core.el (see the message of the commit I referenced). But
> the change to ob-core.el is final if I understand it correctly
> which means that ob-R.el has to be updated. Sorry if I was unclear
> in my description.
>
> Named plain lists were previously turned into data.frames with
> each first level item in one row. Second level items were in the
> second column I believe. After the change to ob-core.el they are
> still data.frames but first level items form columns instead of
> rows. Second (or deeper) level items are dropped as they should be
> after the change to ob-core.el (according to the commit message).
>
> --
> Johan Tolö
>
>

-- 
Jérémie Juste

[-- Attachment #2: Type: text/html, Size: 1835 bytes --]

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Berry, Charles]

> On Dec 4, 2022, at 6:43 PM, Greg Minshall <minshall@umich.edu> wrote:
> 
> i see this same behavior.  to me, (org-babel-R-assign-elisp) seems to be
> the problem, but it hasn't changed any time recently.  (nor, if my =git
> blame= is done correctly, has anything else in ob-R.el.)
> 
> i don't understand how (org-babel-R-assign-elisp) thinks.  it seems to
> assign =max= and =min= assuming that =value= is a list of (at least one)
> list(s).
> 
> this *might* be a fix.  but, i don't have a lot of confidence it will
> not have bad side effects for other cases.  (though i tested it with a
> simple list, and a simple table with more than one row, and a list of
> two lists, the first of length 3, the second of length 2.):
> ----
> modified   lisp/ob-R.el
> @@ -241,7 +241,7 @@ This function is called by `org-babel-execute-src-block'."
> (defun org-babel-R-assign-elisp (name value colnames-p rownames-p)
>   "Construct R code assigning the elisp VALUE to a variable named NAME."
>   (if (listp value)
> -      (let* ((lengths (mapcar 'length (cl-remove-if-not 'sequencep value)))
> +      (let* ((lengths (mapcar 'length (cl-remove-if-not 'listp value)))
> 	     (max (if lengths (apply 'max lengths) 0))
> 	     (min (if lengths (apply 'min lengths) 0)))
>         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).

-         (unless (listp (car value)) (setq value (list value)))
+         (unless (listp (car value)) (setq value (mapcar 'list value)))


> ----

This makes a list like '("a" "b" "c") into a data.frame with one column.

HTH,
Chuck

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

hi, Charles,

> This makes a list like '("a" "b" "c") into a data.frame with one column.

1. which version of Org are you running?
: Org mode version 9.6 (9.6-g60de19 @ /home/minshall/.emacs.d/straight/build/org/):

2. is my below example yours?

i think i used to see the behavior you describe in 9.5.

cheers, Greg

----

#+begin_src R :var list='("a" "b" "c")
  list
#+end_src

#+RESULTS:
| a | b | c |

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Berry, Charles]

Hi Greg,


> On Dec 5, 2022, at 4:41 PM, Greg Minshall <minshall@umich.edu> wrote:
> 
> hi, Charles,
> 
>> This makes a list like '("a" "b" "c") into a data.frame with one column.
> 
> 1. which version of Org are you running?
> : Org mode version 9.6 (9.6-g60de19 @ /home/minshall/.emacs.d/straight/build/org/):
> 

Org mode version 9.6 (release_9.6-14-g53814a (dated 2022-12-02)

> 2. is my below example yours?


Not with the change I indicated (which includes your changes of sequencep to listp)

I have


#+begin_src R :var list='("a" "b" "c")
 list
#+end_src

#+RESULTS:
| a |
| b |
| c |



> 
> i think i used to see the behavior you describe in 9.5.
> 

So, isn't that what is wanted?


HTH,

Chuck


> cheers, Greg
> 
> ----
> 
> #+begin_src R :var list='("a" "b" "c")
>  list
> #+end_src
> 
> #+RESULTS:
> | a | b | c |

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

Chuck,

sorry, i didn't see the change you had added.

> So, isn't that what is wanted?

that's a policy question that i can't answer.  :)

cheers, Greg

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

for the record, these are the tests i ran with my off-the-cuff "fix"
(s/sequencep/listp/):
----
#+name: alist
- first item
- second item
- third item

#+begin_src R :var list=alist :session ss
 list
#+end_src

#+RESULTS:
| first item | second item | third item |

#+begin_src R :var variable='(a test)
  variable
#+end_src

#+RESULTS:
| a | test |

#+name: atable
| this | is | a | header | line |
|------+----+---+--------+------|
| 1    | 2  | 3 | 4      | 5    |
| a    | b  | c | d      | e    |

#+begin_src R :var list=atable :session ss
 list
#+end_src

#+RESULTS:
| 1 | 2 | 3 | 4 | 5 |
| a | b | c | d | e |

#+begin_src R :var list='((1 2 3) (1 2))
  list
#+end_src

#+RESULTS:
| 1 | 2 | 3 |
| 1 | 2 |   |


#+begin_src elisp :var list=alist
 list
#+end_src

#+RESULTS:
| first item | second item | third item |

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Jeremie Juste]

Hello,

Many thanks for the insights. I confess that I have never transferred
list from org to R before. I've always use tables and as far as I
understand they works fine in 9.6.

So assuming this list

#+name: alist 
- first item
- second item
- third item
  - 3.1 item
- fourth item


before c72d5ee84 we could do something like 

#+begin_src R :var list=alist
  list
#+end_src

#+RESULTS:
| first item  |                        |
| second item |                        |
| third item  | (unordered (3.1 item)) |
| fourth item |                        |

and after we end up with 


#+begin_src R :var list=alist
  list
#+end_src

#+RESULTS:
| first item | second item | third item | fourth item |   |   |   |   |   |   |

Here I'm on uncharted territory. We could go with 
1.
| first item | second item | third item | fourth item |

or be closer to the version 9.5 with
2. 
| first item  | 
| second item | 
| third item  | 
| fourth item | 

However I'm still tempted to choose the second option to break as little
workflow as possible.

If we go in this direction the solution of Chuck works fine. Many thanks
for the suggestions.

> @@ -241,7 +241,7 @@ This function is called by `org-babel-execute-src-block'."
> (defun org-babel-R-assign-elisp (name value colnames-p rownames-p)
>   "Construct R code assigning the elisp VALUE to a variable named NAME."
>   (if (listp value)
> -      (let* ((lengths (mapcar 'length (cl-remove-if-not 'sequencep value)))
> +      (let* ((lengths (mapcar 'length (cl-remove-if-not 'listp value)))
> 	     (max (if lengths (apply 'max lengths) 0))
> 	     (min (if lengths (apply 'min lengths) 0)))
>         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).

-         (unless (listp (car value)) (setq value (list value)))
+         (unless (listp (car value)) (setq value (mapcar 'list value)))



Do you have any case of using org-list to R? Do you think we could use
the list for a different purpose.  Suggestions are welcome.

Best regards,
Jeremie








On Sunday,  4 Dec 2022 at 18:49, Greg Minshall wrote:
> for the record, these are the tests i ran with my off-the-cuff "fix"
> (s/sequencep/listp/):
> ----
>
> #+name: alist
> - first item
>
> - second item
> - third item
>
> #+begin_src R :var list=alist :session ss
>  list
> #+end_src
>
>
> #+RESULTS:
> | first item | second item | third item |
>
> #+begin_src R :var variable='(a test)
>   variable
> #+end_src
>
>
> #+RESULTS:
> | a | test |
>
> #+name: atable
> | this | is | a | header | line |
>
> |------+----+---+--------+------|
> | 1    | 2  | 3 | 4      | 5    |
> | a    | b  | c | d      | e    |
>
> #+begin_src R :var list=atable :session ss
>  list
> #+end_src
>
>
> #+RESULTS:
> | 1 | 2 | 3 | 4 | 5 |
> | a | b | c | d | e |
>
> #+begin_src R :var list='((1 2 3) (1 2))
>   list
> #+end_src
>
>
> #+RESULTS:
> | 1 | 2 | 3 |
> | 1 | 2 |   |
>
> #+begin_src elisp :var list=alist
>  list
> #+end_src
>
> #+RESULTS:
> | first item | second item | third item |

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

Jeremie,

i am neutral w.r.t. what the output should be.  like you, i've always
sent in tables.

for me, i don't know that it makes much difference how lists are
presented to R code, as long as it is well-defined and stable over time.

given that, iiuc, 9.5 presented a list as (the equivalent of) multiple
rows of a table, i'd vote for staying with (going back to) that.

(i think that's what the two modifications in Chuck's e-mail give you.)

cheers, Greg

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[William Denton]

On 6 December 2022, Greg Minshall wrote:

> given that, iiuc, 9.5 presented a list as (the equivalent of) multiple
> rows of a table, i'd vote for staying with (going back to) that.
>
> (i think that's what the two modifications in Chuck's e-mail give you.)

I agree: I've never used this, but if it used to become rows of a table, let's 
keep it that way.  In R we might think of the list as a vector, I guess, but 
there's no way to represent that nicely in Org, though it is easy to step 
through the rows of a column or table.

Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

Jeremie Juste <jeremiejuste@gmail.com> writes:

> Many thanks for the insights. I confess that I have never transferred
> list from org to R before. I've always use tables and as far as I
> understand they works fine in 9.6.
>
> So assuming this list
>
> #+name: alist 
> - first item
> - second item
> - third item
>   - 3.1 item
> - fourth item
>
>
> before c72d5ee84 we could do something like 
>
> #+begin_src R :var list=alist
>   list
> #+end_src
>
> #+RESULTS:
> | first item  |                        |
> | second item |                        |
> | third item  | (unordered (3.1 item)) |
> | fourth item |                        |

And it was a bug that the ob-core patch fixed.
Unfortunately, a number of babel backends also developed workarounds
meanwhile.

For the context, the ground truth is Org manual:

16.4 Environment of a Code Block

    list
         A simple named list.
    
              #+NAME: example-list
              - simple
                - not
                - nested
              - list
    
              #+BEGIN_SRC emacs-lisp :var x=example-list
                (print x)
              #+END_SRC
    
              #+RESULTS:
              | simple | list |
    
         Note that only the top level list items are passed along.  Nested
         list items are ignored.

> and after we end up with 
>
>
> #+begin_src R :var list=alist
>   list
> #+end_src
>
> #+RESULTS:
> | first item | second item | third item | fourth item |   |   |   |   |   |   |
>
> Here I'm on uncharted territory. We could go with 
> 1.
> | first item | second item | third item | fourth item |
>
> or be closer to the version 9.5 with
> 2. 
> | first item  | 
> | second item | 
> | third item  | 
> | fourth item | 
>
> However I'm still tempted to choose the second option to break as little
> workflow as possible.
>
> If we go in this direction the solution of Chuck works fine. Many thanks
> for the suggestions.

I also like the second option as **printing** more, but it is _not_ what
we have in the manual.

Note that the origin of the issue is in how list=alist is assigned.

Before the problematic commit, we had awkward situation when
alist=(("first item") ("second item") ...)

I changed things to
alist=("first item" "second item" ...)
which makes a lot more sense when passed as variable value.

Now, printing:

ob-core prints lists as a special case of tables.
Since ("first item" "second item" ...) is a row vector, we get
| first item | second item | ... |

changing row vectors to be printed as column vectors is easy, but I am
afraid about non-trivial breakage.


Hope the above clarifies about what happened and why I changes things
like I did.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Jeremie Juste]

Hello,

Many thanks to you all for your feedback.
From 1ad16ffb9, I have restored the expected output in R. that is.


#+NAME: example-list
- simple
  - not
  - nested
- list

#+BEGIN_SRC R :var x=example-list
x
#+END_SRC

#+RESULTS:
| simple |
| list   |


On Wednesday,  7 Dec 2022 at 12:16, Ihor Radchenko wrote:

> I also like the second option as **printing** more, but it is _not_ what
> we have in the manual.

Thanks for the precision. We propably need to update the doc. I guess it also
has to be language specific.

Best regards,
Jeremie

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

Jeremie Juste <jeremiejuste@gmail.com> writes:

> Many thanks to you all for your feedback.
> From 1ad16ffb9, I have restored the expected output in R. that is.
>
>
> #+NAME: example-list
> - simple
>   - not
>   - nested
> - list
>
> #+BEGIN_SRC R :var x=example-list
> x
> #+END_SRC
>
> #+RESULTS:
> | simple |
> | list   |

I am not sure if I like the approach you used in the commit.

-        (unless (listp (car value)) (setq value (list value)))
+        (unless (listp (car value)) (setq value (mapcar 'list value)))

In the above, you are transforming (val1 val2 val3 ...) list into
((val1) (val2) (val3) ...).

Does it make sense from the point of view of R code?
AFAIU, the current ob-R implementation converts lists into R tables,
which is not accurate? Would it make sense to convert Elisp lists into R
lists directly?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

Ihor,

> Does it make sense from the point of view of R code?
> AFAIU, the current ob-R implementation converts lists into R tables,
> which is not accurate? Would it make sense to convert Elisp lists into R
> lists directly?

my "barely half a cent" would be that backwards compatibility here
trumps, hmm, "logic".  i suspect a note in the wiki page for ob-R
stating that lists are converted to single-column tables would be
sufficient.  (partly because i think people writing code need to refer
to such pages to understand these sorts of mappings.)

(one could make the case that "no one" passes lists to R code; that
could push *me* towards your suggestion.)

cheers, Greg

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

Greg Minshall <minshall@umich.edu> writes:

>> Does it make sense from the point of view of R code?
>> AFAIU, the current ob-R implementation converts lists into R tables,
>> which is not accurate? Would it make sense to convert Elisp lists into R
>> lists directly?
>
> my "barely half a cent" would be that backwards compatibility here
> trumps, hmm, "logic".  i suspect a note in the wiki page for ob-R
> stating that lists are converted to single-column tables would be
> sufficient.  (partly because i think people writing code need to refer
> to such pages to understand these sorts of mappings.)
>
> (one could make the case that "no one" passes lists to R code; that
> could push *me* towards your suggestion.)

Once upon a time, ob-core converted lists to actual lists. It is what is
reflected in the manual.

Later, the code was changed and lists were interpreted as lists of
lists.

Now, I reverted to the original behaviour.

The time span is years and I do not recall any related bug reports.
So, nobody seems to care.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

Ihor,

> The time span is years and I do not recall any related bug reports.
> So, nobody seems to care.

:)

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Johan Tolö]

Ihor Radchenko <yantar92@posteo.net> writes:

> Greg Minshall <minshall@umich.edu> writes:
[snipped]
> The time span is years and I do not recall any related bug 
> reports.
> So, nobody seems to care.

I do pass org plain lists to R source code blocks which is why I 
found the current issue. It is easy enough for me to adapt my R 
code whenever there is a change in the way it is handled. It is 
unfortunate that it will no longer be possible to access any 
nested elements of plain lists but it does not affect me because I 
was not using that feature.

I think that the output from R source code blocks should be as 
close to the example in the org manual as possible. I also think 
that the handling within ob-R.el should be as simple as possible. 
I do not mind if that means that I have to convert a data.frame 
from columns to rows.

For anyone wondering, you can simply unlist() a data.frame to get 
a vector of the columns.

Johan

-- 
Johan Tolö

[FR] Allow passing nested list structures to variables in src blocks (was: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)])

[Ihor Radchenko]

Johan Tolö <johan@toloe.se> writes:

> I do pass org plain lists to R source code blocks which is why I 
> found the current issue. It is easy enough for me to adapt my R 
> code whenever there is a change in the way it is handled. It is 
> unfortunate that it will no longer be possible to access any 
> nested elements of plain lists but it does not affect me because I 
> was not using that feature.

If necessary, we may adapt the recent idea with list-name[] and allow
passing ranges to the lists, similar to tables.

What I have in mind:

1. variable=list-name will assign top-level items as a list (item1 item2 ...)
2. variable=list-name[] will allow nesting: ((item1) (item2 (sub-list)) ...)
3. variable=list-name[from:to,...]

It will be a new feature, of course.

> I think that the output from R source code blocks should be as 
> close to the example in the org manual as possible. I also think 
> that the handling within ob-R.el should be as simple as possible. 
> I do not mind if that means that I have to convert a data.frame 
> from columns to rows.
>
> For anyone wondering, you can simply unlist() a data.frame to get 
> a vector of the columns.

I am also in favour of staying as close as possible to the manual. I find
converting lists to tables non-intuitive.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Jeremie Juste]

Hello Ihor

On Thursday,  8 Dec 2022 at 09:07, Ihor Radchenko wrote:

> I am not sure if I like the approach you used in the commit.
>
> -        (unless (listp (car value)) (setq value (list value)))
> +        (unless (listp (car value)) (setq value (mapcar 'list value)))
>
> In the above, you are transforming (val1 val2 val3 ...) list into
> ((val1) (val2) (val3) ...).
>
> Does it make sense from the point of view of R code?
> AFAIU, the current ob-R implementation converts lists into R tables,
> which is not accurate? Would it make sense to convert Elisp lists into R
> lists directly?

Many thanks for the feedback. At this point I don't know. On one hand you are
right on the other, this option is backward compatible, and the user can
always create an interface in R to suit his need.

If there are more complaints about that in the future, I'll reconsider. 

Best regards,
Jeremie

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

Jeremie Juste <jeremiejuste@gmail.com> writes:

> On Thursday,  8 Dec 2022 at 09:07, Ihor Radchenko wrote:
>
>> I am not sure if I like the approach you used in the commit.
>>
>> -        (unless (listp (car value)) (setq value (list value)))
>> +        (unless (listp (car value)) (setq value (mapcar 'list value)))
>>
>> In the above, you are transforming (val1 val2 val3 ...) list into
>> ((val1) (val2) (val3) ...).
>>
>> Does it make sense from the point of view of R code?
>> AFAIU, the current ob-R implementation converts lists into R tables,
>> which is not accurate? Would it make sense to convert Elisp lists into R
>> lists directly?
>
> Many thanks for the feedback. At this point I don't know. On one hand you are
> right on the other, this option is backward compatible, and the user can
> always create an interface in R to suit his need.
>
> If there are more complaints about that in the future, I'll reconsider. 

Note that my NEWS entry may not be accurate for ob-R then:

** List references in source block variable assignments are now proper lists

So, we may create confusion one way or another.

Or I may need to put a special clause regarding ob-R into the NEWS item.

Bastien, maybe you have something to say about this situation?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

hi, Ihor,

> ** List references in source block variable assignments are now proper lists

just for background, one oddity is that the meaning of a "list" in R is
a bit loose.
----
> list(1,2,3)
[[1]]
[1] 1

[[2]]
[1] 2

[[3]]
[1] 3

> c(1,2,3)
[1] 1 2 3
----
(where =c= is the "concatenation operator".)

so, to me, at least, "proper list" is not a well-defined term in R.

cheers, Greg

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

Greg Minshall <minshall@umich.edu> writes:

>> ** List references in source block variable assignments are now proper lists
>
> just for background, one oddity is that the meaning of a "list" in R is
> a bit loose.
> ...
> so, to me, at least, "proper list" is not a well-defined term in R.

Sounds reasonable.

I also tried

> list_data <- list("a", list("b"), "c")
> print(list_data[1])
[[1]]
[1] "a"

> print(list_data[2])
[[1]]
[[1]][[1]]
[1] "b"


> print(list_data[3])
[[1]]
[1] "c"

Although, I am not sure if there are no subtle issues with the actual
representation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Johan Tolö]

Greg Minshall <minshall@umich.edu> writes:
>> list(1,2,3)
> [[1]]
> [1] 1
>
> [[2]]
> [1] 2
>
> [[3]]
> [1] 3
>
>> c(1,2,3)
> [1] 1 2 3
> ----
> (where =c= is the "concatenation operator".)
>
> so, to me, at least, "proper list" is not a well-defined term in 
> R.
>
> cheers, Greg

=c= is "a generic function which combines its arguments" according 
to the R documentation. It could do different things depending on 
the input.

=list()= will create an actual R list which are "Generic Vectors" 
according to the R documentation. Each element of a list can be 
any other R data structure including other lists, expressions and 
symbols.

"Proper list" in the context of this discussion and pertaining to 
R would be a =list()=, not a vector which is what is usually 
returned by =c()=. A =data.frame()= is a special case of a 
=list()= where every column has to have the same length.


Johan

-- 
Johan Tolö

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

hi, Johan,

> "Proper list" in the context of this discussion and pertaining to R
> would be a =list()=, not a vector which is what is usually returned by
> =c()=. A =data.frame()= is a special case of a =list()= where every
> column has to have the same length.

well, it's a language mapping problem.  what one considers a "list" in
org-mode is
- well
- something like
- this
  - maybe with
  - this

whereas in e-lisp, '("well" "something like" '("this" '("maybe with"
"this"))).

then, the question arises of how to translate something like that to
whatever data structures a given programming language offers.  it
*might* be to something that programming language calls a "list".

if we are ignoring "sub lists", then for R, one could argue either
vectors or lists.  (someone -- possibly you? -- pointed out that going
from an R list to a vector is as simple as an unlist() call.)

if we ever want to provide support for sub lists, then passing lists as
R lists seems like the way to go.

cheers, Greg
----

> list("well", "something like", list("this", list("maybe with")))
[[1]]
[1] "well"

[[2]]
[1] "something like"

[[3]]
[[3]][[1]]
[1] "this"

[[3]][[2]]
[[3]][[2]][[1]]
[1] "maybe with"



> unlist(list("well", "something like", list("this", list("maybe with"))))
[1] "well"           "something like" "this"           "maybe with"

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Johan Tolö]

Greg Minshall <minshall@umich.edu> writes:

[snipped]
>> "Proper list" in the context of this discussion and pertaining 
>> to R
>> would be a =list()=, not a vector which is what is usually 
>> returned by
>> =c()=. A =data.frame()= is a special case of a =list()= where 
>> every
>> column has to have the same length.
>
> well, it's a language mapping problem.  what one considers a 
> "list" in
> org-mode is
> - well
> - something like
> - this
>   - maybe with
>   - this
>
> whereas in e-lisp, '("well" "something like" '("this" '("maybe 
> with"
> "this"))).

Absolutely, I just wanted to clarify that there is no confusion as 
to what an R list is in the context of R itself (as far as I can 
tell). Your post made it sound like there is.

[snipped]


-- 
Johan Tolö

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Greg Minshall]

Johan,

> Absolutely, I just wanted to clarify that there is no confusion as to
> what an R list is in the context of R itself (as far as I can
> tell). Your post made it sound like there is.

ah, indeed -- thanks!

cheers, Greg

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Bastien Guerry]

I think it would make sense to convert Elisp lists into R lists
directly.  Jeremie, would you be okay with this?

Ihor Radchenko <yantar92@posteo.net> writes:

> Or I may need to put a special clause regarding ob-R into the NEWS
> item.

... that way we don't need such a special clause regarding ob-R.

2 cts,

-- 
 Bastien

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Berry, Charles]

Bastien et al,

> On Dec 29, 2022, at 8:00 AM, Bastien Guerry <bzg@gnu.org> wrote:
> 
> I think it would make sense to convert Elisp lists into R lists
> directly.  Jeremie, would you be okay with this?
> 


Perhaps there are some hiccups. 

The R `data.frame' type is a list with some added attributes. In that list each top level element is a *column* in the data.frame.

In the elisp list produced by rendering a table as in `:var mydf=atab', each top level element is a sequence containing one *row* of the table `atab'.

So if elisp lists are to be converted to R lists, ob-R will need to know whether the list came from a table to decide whether to render top level elements as list elements or as data.frame rows.

---

There might be some useful applications for converting org lists to R lists and vice versa. But it looks like a significant amount of effort to get it right.

Best,

Chuck
 

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Bastien]

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

> The R `data.frame' type is a list with some added attributes. In
> that list each top level element is a *column* in the data.frame.

Thanks for the explanation.

> There might be some useful applications for converting org lists to R
> lists and vice versa. But it looks like a significant amount of effort
> to get it right.

Yes... up to Jeremie, depending on users' needs.

Best,

-- 
 Bastien

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Ihor Radchenko]

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

> So if elisp lists are to be converted to R lists, ob-R will need to know whether the list came from a table to decide whether to render top level elements as list elements or as data.frame rows.

This information should be available: lists are passed as Elisp lists to
ob-R; tables are passed as lists of lists.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-R.el: extra empty data.frame columns generated from plain lists after recent change [9.6 (release_9.6-3-ga4d38e @ /usr/share/emacs/30.0.50/lisp/org/)]

[Bastien Guerry]

So, based on Charles recent feedback:

>> If there are more complaints about that in the future, I'll
>> reconsider.

... let's do as Jeremie originally said: wait until someone complains.

> Note that my NEWS entry may not be accurate for ob-R then:
>
> ** List references in source block variable assignments are now proper lists
>
> So, we may create confusion one way or another.
>
> Or I may need to put a special clause regarding ob-R into the NEWS
> item.

And add this special clause regarding ob-R into the NEWS item.  

Ihor, could you do that?

-- 
 Bastien