Table width cookies become part of data table in R

Table width cookies become part of data table in R

[William Denton]

Up until recently, width cookies were ignored when reading a table into an R code block as data.  For example, take this test table:

#+name: test_table
|       Date | Weather              |
|       <10> | <50>                 |
|------------+----------------------|
| 2024-11-01 | Warm                 |
| 2024-11-02 | Warm                 |
| 2024-11-03 | Still strangely warm |

This code block turns it into a basic data table:

#+begin_src R :var t=test_table :colnames yes
t
#+end_src

#+RESULTS:
|       Date | Weather              |
|------------+----------------------|
|       <10> | <50>                 |
| 2024-11-01 | Warm                 |
| 2024-11-02 | Warm                 |
| 2024-11-03 | Still strangely warm |

That first row, with "<10>", shouldn't be there.  It wasn't until recently, but I can't see anything in ORG-NEWS or git commits about why it might have started.  (I'm running from the development tree, updated just now.)

Is this a bug or am I missing something?


Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada

Re: Table width cookies become part of data table in R

[Rens Oliemans]

William Denton <william@williamdenton.org> writes:

> Up until recently, width cookies were ignored when reading a table into an R
> code block as data.

According to git bisect, the first bad commit is:
[e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output

Rens

Re: Table width cookies become part of data table in R

[William Denton]

On Friday, November 15th, 2024 at 06:59, Rens Oliemans <hallo@rensoliemans.nl> wrote:

> According to git bisect, the first bad commit is:
> [e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output

Aha!  Thank you.  I wasn't looking at tables generally.  I see now that ORG-NEWS has a "~orgtbl-to-generic~ retains special rows when exporting to Org" entry.  It says, "To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argument."  But:

#+name: test_table
|       Date | Weather              |
|       <10> | <50>                 |
|------------+----------------------|
| 2024-11-01 | Warm                 |
| 2024-11-02 | Warm                 |
| 2024-11-03 | Still strangely warm |

#+begin_src R :var t=test_table :colnames yes :with-special-rows nil
t
#+end_src

#+RESULTS:
|       Date | Weather              |
|------------+----------------------|
|       <10> | <50>                 |
| 2024-11-01 | Warm                 |
| 2024-11-02 | Warm                 |
| 2024-11-03 | Still strangely warm |

If I (setq org-org-with-special-rows nil) then this stops, but shouldn't the parameter work?

Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada

Re: Table width cookies become part of data table in R

[William Denton]

Sorry to confuse the threading, but I wanted to bring this back up given Ihor and Suhail Singh talking about this change in R behaviour.

I agree it seems best to revert to the earlier was of handling data tables in R until, if possible, there is a nice way to clean up all the languages at once.  The whole situation may be confusing, since there's a mix of how languages handle them, but it doesn't break anything.  It seems like a small thing to break a lot of R code over.

I don't know if this bug will matter after that, but I wanted to make sure it got noted just in case.


Thanks,

Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada


On Friday, November 15th, 2024 at 09:23, William Denton <william@williamdenton.org> wrote:

> On Friday, November 15th, 2024 at 06:59, Rens Oliemans hallo@rensoliemans.nl wrote:
> 
> > According to git bisect, the first bad commit is:
> > [e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output
> 
> 
> Aha! Thank you. I wasn't looking at tables generally. I see now that ORG-NEWS has a "~orgtbl-to-generic~ retains special rows when exporting to Org" entry. It says, "To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argument." But:
> 
> #+name: test_table
> | Date | Weather |
> | <10> | <50> |
> 
> |------------+----------------------|
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
> 
> #+begin_src R :var t=test_table :colnames yes :with-special-rows nil
> t
> #+end_src
> 
> #+RESULTS:
> | Date | Weather |
> |------------+----------------------|
> | <10> | <50> |
> 
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
> 
> If I (setq org-org-with-special-rows nil) then this stops, but shouldn't the parameter work?

Re: Table width cookies become part of data table in R

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2571 bytes --]

William Denton <william@williamdenton.org> writes:

> Up until recently, width cookies were ignored when reading a table into an R code block as data.  For example, take this test table:
>
> #+name: test_table
> |       Date | Weather              |
> |       <10> | <50>                 |
> |------------+----------------------|
> | 2024-11-01 | Warm                 |
> | 2024-11-02 | Warm                 |
> | 2024-11-03 | Still strangely warm |
>
> This code block turns it into a basic data table:
>
> #+begin_src R :var t=test_table :colnames yes
> t
> #+end_src
>
> #+RESULTS:
> |       Date | Weather              |
> |------------+----------------------|
> |       <10> | <50>                 |
> | 2024-11-01 | Warm                 |
> | 2024-11-02 | Warm                 |
> | 2024-11-03 | Still strangely warm |
>
> That first row, with "<10>", shouldn't be there.  It wasn't until
> recently, but I can't see anything in ORG-NEWS or git commits about
> why it might have started.  (I'm running from the development tree,
> updated just now.)
>
> Is this a bug or am I missing something?

It is indeed a breaking change.

Among various ob-* backends, we do not have consistency in handling
table width specifiers and other metadata.
For example, ob-python takes the table literally when assigning
variables. ob-R, in the past, removed the width specifications. Not
anymore.

I believe that the end result is more consistency, but, clearly, we got
a regression. Not just in ob-R, but in a number of other babel backends.

I can force the previous behavior in ob-R and similar backends, but I am
not sure if it is the right thing to do.

Pros:
- We will retain historic behavior

Cons:
- We will retain historic behavior with different backends behaving
   differently

Here is a list of backends that filtered out special rows from tables in
variable assignments: ob-julia, ob-R, ob-sql, ob-shell, ob-awk, ob-gnuplot&org-plot
6 in total.

Here is a list of backends that did not filter out special rows from tables:
ob-ruby, ob-python, ob-processing, ob-perl, ob-octave, ob-ocaml, ob-lua,
ob-js, ob-java, ob-haskell, ob-eshell
11 in total.

I am not really sure about the best approach here.

Tentatively, I am leaning towards restoring the old behavior for the
time being, until we can come up with something better.

Patch attached.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-Restore-historic-removal-of-special-rows-in-table.patch --]
[-- Type: text/x-patch, Size: 6225 bytes --]

From 36ddf840db2b6eae3bb25efb228117a8df79a0b8 Mon Sep 17 00:00:00 2001
Message-ID: <36ddf840db2b6eae3bb25efb228117a8df79a0b8.1734196827.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 14 Dec 2024 18:17:51 +0100
Subject: [PATCH] ob-*: Restore historic removal of special rows in table
 assignments

* lisp/ob-R.el (org-babel-R-assign-elisp):
* lisp/ob-awk.el (org-babel-awk-var-to-awk):
* lisp/ob-gnuplot.el (org-babel-gnuplot-table-to-data):
* lisp/ob-julia.el (org-babel-julia-assign-elisp):
* lisp/ob-shell.el (org-babel-sh-var-to-string):
* lisp/ob-sql.el (org-babel-sql-expand-vars):
* lisp/org-plot.el (org-plot/gnuplot-to-data): Restore historic
behavior when special rows are removed before assigning table data to
variables.  Note that it is (historically) inconsistent with ob-ruby,
ob-python, ob-processing, ob-perl, ob-octave, ob-ocaml, ob-lua, ob-js,
ob-java, ob-haskell, and ob-eshell where special rows are retained.

Reported-by: William Denton <william@williamdenton.org>
Link: https://orgmode.org/list/_XqzGYdYTbgEql7bdh5hazhuc9DpC5k6pPjWT126KEL_CE5NY-SlkHua-bbPDUsJ_-e4_oopiG7qvGJzCH2iyTZDDkyvtbM-JPqCONyCc8k=@williamdenton.org
---
 lisp/ob-R.el       | 5 ++++-
 lisp/ob-awk.el     | 6 +++++-
 lisp/ob-gnuplot.el | 1 +
 lisp/ob-julia.el   | 5 ++++-
 lisp/ob-shell.el   | 7 +++++--
 lisp/ob-sql.el     | 8 +++++---
 lisp/org-plot.el   | 3 ++-
 7 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 1dbbd83f2c..c22f2aa2a3 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -251,7 +251,10 @@ (defun org-babel-R-assign-elisp (name value colnames-p rownames-p)
 	     (min (if lengths (apply 'min lengths) 0)))
         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
         (unless (listp (car value)) (setq value (mapcar 'list value)))
-	(let ((file (orgtbl-to-tsv value '(:fmt org-babel-R-quote-tsv-field)))
+	(let ((file (orgtbl-to-tsv
+                     value
+                     '( :fmt org-babel-R-quote-tsv-field
+                        :with-special-rows nil)))
 	      (header (if (or (eq (nth 1 value) 'hline) colnames-p)
 			  "TRUE" "FALSE"))
 	      (row-names (if rownames-p "1" "NULL")))
diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el
index 0c07aec829..16fbd63f5c 100644
--- a/lisp/ob-awk.el
+++ b/lisp/ob-awk.el
@@ -111,7 +111,11 @@ (defun org-babel-awk-var-to-awk (var &optional sep)
   (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
     (cond
      ((and (listp var) (listp (car var)))
-      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var)))
+      (orgtbl-to-generic
+       var
+       (list :sep (or sep "\t")
+             :fmt echo-var
+             :with-special-rows nil)))
      ((listp var)
       (mapconcat echo-var var "\n"))
      (t (funcall echo-var var)))))
diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index 2c53f30664..b7598adf0b 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -317,6 +317,7 @@ (defun org-babel-gnuplot-table-to-data (table data-file params)
 		`( :sep "\t" :fmt org-babel-gnuplot-quote-tsv-field
                    ;; Two setting below are needed to make :fmt work.
                    :raw t
+                   :with-special-rows nil
                    :backend ,ob-gnuplot-data)
 		params)))))
   data-file)
diff --git a/lisp/ob-julia.el b/lisp/ob-julia.el
index f9da734b37..9ed0ff80a6 100644
--- a/lisp/ob-julia.el
+++ b/lisp/ob-julia.el
@@ -171,7 +171,10 @@ (defun org-babel-julia-assign-elisp (name value)
              (min (if lengths (apply #'min lengths) 0)))
         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
         (unless (listp (car value)) (setq value (list value)))
-        (let ((file (orgtbl-to-csv value '(:fmt org-babel-julia-quote-csv-field))))
+        (let ((file (orgtbl-to-csv
+                     value
+                     '( :fmt org-babel-julia-quote-csv-field
+                        :with-special-rows nil))))
           (if (= max min)
               (format "%s = begin
     using CSV
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 7706234919..535729d2bb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,8 +263,11 @@ (defun org-babel-sh-var-to-string (var &optional sep hline)
   (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
     (cond
      ((and (listp var) (or (listp (car var)) (eq (car var) 'hline)))
-      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var
-				    :hline hline)))
+      (orgtbl-to-generic
+       var
+       (list :sep (or sep "\t") :fmt echo-var
+	     :hline hline
+             :with-special-rows nil)))
      ((listp var)
       (mapconcat echo-var var "\n"))
      (t (funcall echo-var var)))))
diff --git a/lisp/ob-sql.el b/lisp/ob-sql.el
index d7bcaa0973..8795934520 100644
--- a/lisp/ob-sql.el
+++ b/lisp/ob-sql.el
@@ -418,9 +418,11 @@ (defun org-babel-sql-expand-vars (body vars &optional sqlite)
                       (insert (orgtbl-to-csv
                                val (if sqlite
                                        nil
-                                     '(:fmt (lambda (el) (if (stringp el)
-                                                        el
-                                                      (format "%S" el))))))))
+                                     '( :fmt (lambda (el)
+                                               (if (stringp el)
+                                                   el
+                                                 (format "%S" el)))
+                                        :with-special-rows nil)))))
                     data-file)
                 (if (stringp val) val (format "%S" val))))
 	    body t t)))
diff --git a/lisp/org-plot.el b/lisp/org-plot.el
index 836cfaffca..7f21a4cd55 100644
--- a/lisp/org-plot.el
+++ b/lisp/org-plot.el
@@ -140,7 +140,8 @@ (defun org-plot/gnuplot-to-data (table data-file params)
       (insert (orgtbl-to-generic
 	       table
 	       (org-combine-plists
-	        '(:sep "\t" :fmt org-plot-quote-tsv-field)
+	        '( :sep "\t" :fmt org-plot-quote-tsv-field
+                   :with-special-rows nil)
 	        params)))))
   nil)
 
-- 
2.47.1

Re: Table width cookies become part of data table in R

[Suhail Singh]

Ihor Radchenko <yantar92@posteo.net> writes:

> Tentatively, I am leaning towards restoring the old behavior for the
> time being, until we can come up with something better.

I think doing so is prudent.

Further, I think it would help if the manner in which special rows are
handled were user-configurable.  Once we have that in place, then the
old behaviour can be retained by choosing appropriate default values for
the babel backends you mentioned.  Then, finally, as a subsequent step a
single default can perhaps be set to improve consistency across babel
backends (along with a mention in the NEWS, appropriate version bump
etc.)

If the above is done, I would argue that, eventually, the default
treatment where special rows are elided would be the more useful
setting.

-- 
Suhail

Re: Table width cookies become part of data table in R

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Tentatively, I am leaning towards restoring the old behavior for the
> time being, until we can come up with something better.
>
> Patch attached.

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=c93ddd23bf

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>