* Table width cookies become part of data table in R
@ 2024-11-15 4:29 William Denton
2024-11-15 11:59 ` Rens Oliemans
2024-12-14 17:21 ` Ihor Radchenko
0 siblings, 2 replies; 7+ messages in thread
From: William Denton @ 2024-11-15 4:29 UTC (permalink / raw)
To: Emacs Org mode mailing list
Up until recently, width cookies were ignored when reading a table into an R code block as data. For example, take this test table:
#+name: test_table
| Date | Weather |
| <10> | <50> |
|------------+----------------------|
| 2024-11-01 | Warm |
| 2024-11-02 | Warm |
| 2024-11-03 | Still strangely warm |
This code block turns it into a basic data table:
#+begin_src R :var t=test_table :colnames yes
t
#+end_src
#+RESULTS:
| Date | Weather |
|------------+----------------------|
| <10> | <50> |
| 2024-11-01 | Warm |
| 2024-11-02 | Warm |
| 2024-11-03 | Still strangely warm |
That first row, with "<10>", shouldn't be there. It wasn't until recently, but I can't see anything in ORG-NEWS or git commits about why it might have started. (I'm running from the development tree, updated just now.)
Is this a bug or am I missing something?
Bill
--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-11-15 4:29 Table width cookies become part of data table in R William Denton
@ 2024-11-15 11:59 ` Rens Oliemans
2024-11-15 14:23 ` William Denton
2024-12-14 17:21 ` Ihor Radchenko
1 sibling, 1 reply; 7+ messages in thread
From: Rens Oliemans @ 2024-11-15 11:59 UTC (permalink / raw)
To: William Denton, Emacs Org mode mailing list
William Denton <william@williamdenton.org> writes:
> Up until recently, width cookies were ignored when reading a table into an R
> code block as data.
According to git bisect, the first bad commit is:
[e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output
Rens
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-11-15 11:59 ` Rens Oliemans
@ 2024-11-15 14:23 ` William Denton
2024-12-17 2:40 ` William Denton
0 siblings, 1 reply; 7+ messages in thread
From: William Denton @ 2024-11-15 14:23 UTC (permalink / raw)
To: Emacs Org mode mailing list
On Friday, November 15th, 2024 at 06:59, Rens Oliemans <hallo@rensoliemans.nl> wrote:
> According to git bisect, the first bad commit is:
> [e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output
Aha! Thank you. I wasn't looking at tables generally. I see now that ORG-NEWS has a "~orgtbl-to-generic~ retains special rows when exporting to Org" entry. It says, "To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argument." But:
#+name: test_table
| Date | Weather |
| <10> | <50> |
|------------+----------------------|
| 2024-11-01 | Warm |
| 2024-11-02 | Warm |
| 2024-11-03 | Still strangely warm |
#+begin_src R :var t=test_table :colnames yes :with-special-rows nil
t
#+end_src
#+RESULTS:
| Date | Weather |
|------------+----------------------|
| <10> | <50> |
| 2024-11-01 | Warm |
| 2024-11-02 | Warm |
| 2024-11-03 | Still strangely warm |
If I (setq org-org-with-special-rows nil) then this stops, but shouldn't the parameter work?
Bill
--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-11-15 4:29 Table width cookies become part of data table in R William Denton
2024-11-15 11:59 ` Rens Oliemans
@ 2024-12-14 17:21 ` Ihor Radchenko
2024-12-15 5:04 ` Suhail Singh
2024-12-22 10:38 ` Ihor Radchenko
1 sibling, 2 replies; 7+ messages in thread
From: Ihor Radchenko @ 2024-12-14 17:21 UTC (permalink / raw)
To: William Denton; +Cc: Emacs Org mode mailing list
[-- Attachment #1: Type: text/plain, Size: 2571 bytes --]
William Denton <william@williamdenton.org> writes:
> Up until recently, width cookies were ignored when reading a table into an R code block as data. For example, take this test table:
>
> #+name: test_table
> | Date | Weather |
> | <10> | <50> |
> |------------+----------------------|
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
>
> This code block turns it into a basic data table:
>
> #+begin_src R :var t=test_table :colnames yes
> t
> #+end_src
>
> #+RESULTS:
> | Date | Weather |
> |------------+----------------------|
> | <10> | <50> |
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
>
> That first row, with "<10>", shouldn't be there. It wasn't until
> recently, but I can't see anything in ORG-NEWS or git commits about
> why it might have started. (I'm running from the development tree,
> updated just now.)
>
> Is this a bug or am I missing something?
It is indeed a breaking change.
Among various ob-* backends, we do not have consistency in handling
table width specifiers and other metadata.
For example, ob-python takes the table literally when assigning
variables. ob-R, in the past, removed the width specifications. Not
anymore.
I believe that the end result is more consistency, but, clearly, we got
a regression. Not just in ob-R, but in a number of other babel backends.
I can force the previous behavior in ob-R and similar backends, but I am
not sure if it is the right thing to do.
Pros:
- We will retain historic behavior
Cons:
- We will retain historic behavior with different backends behaving
differently
Here is a list of backends that filtered out special rows from tables in
variable assignments: ob-julia, ob-R, ob-sql, ob-shell, ob-awk, ob-gnuplot&org-plot
6 in total.
Here is a list of backends that did not filter out special rows from tables:
ob-ruby, ob-python, ob-processing, ob-perl, ob-octave, ob-ocaml, ob-lua,
ob-js, ob-java, ob-haskell, ob-eshell
11 in total.
I am not really sure about the best approach here.
Tentatively, I am leaning towards restoring the old behavior for the
time being, until we can come up with something better.
Patch attached.
--
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-Restore-historic-removal-of-special-rows-in-table.patch --]
[-- Type: text/x-patch, Size: 6225 bytes --]
From 36ddf840db2b6eae3bb25efb228117a8df79a0b8 Mon Sep 17 00:00:00 2001
Message-ID: <36ddf840db2b6eae3bb25efb228117a8df79a0b8.1734196827.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 14 Dec 2024 18:17:51 +0100
Subject: [PATCH] ob-*: Restore historic removal of special rows in table
assignments
* lisp/ob-R.el (org-babel-R-assign-elisp):
* lisp/ob-awk.el (org-babel-awk-var-to-awk):
* lisp/ob-gnuplot.el (org-babel-gnuplot-table-to-data):
* lisp/ob-julia.el (org-babel-julia-assign-elisp):
* lisp/ob-shell.el (org-babel-sh-var-to-string):
* lisp/ob-sql.el (org-babel-sql-expand-vars):
* lisp/org-plot.el (org-plot/gnuplot-to-data): Restore historic
behavior when special rows are removed before assigning table data to
variables. Note that it is (historically) inconsistent with ob-ruby,
ob-python, ob-processing, ob-perl, ob-octave, ob-ocaml, ob-lua, ob-js,
ob-java, ob-haskell, and ob-eshell where special rows are retained.
Reported-by: William Denton <william@williamdenton.org>
Link: https://orgmode.org/list/_XqzGYdYTbgEql7bdh5hazhuc9DpC5k6pPjWT126KEL_CE5NY-SlkHua-bbPDUsJ_-e4_oopiG7qvGJzCH2iyTZDDkyvtbM-JPqCONyCc8k=@williamdenton.org
---
lisp/ob-R.el | 5 ++++-
lisp/ob-awk.el | 6 +++++-
lisp/ob-gnuplot.el | 1 +
lisp/ob-julia.el | 5 ++++-
lisp/ob-shell.el | 7 +++++--
lisp/ob-sql.el | 8 +++++---
lisp/org-plot.el | 3 ++-
7 files changed, 26 insertions(+), 9 deletions(-)
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 1dbbd83f2c..c22f2aa2a3 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -251,7 +251,10 @@ (defun org-babel-R-assign-elisp (name value colnames-p rownames-p)
(min (if lengths (apply 'min lengths) 0)))
;; Ensure VALUE has an orgtbl structure (depth of at least 2).
(unless (listp (car value)) (setq value (mapcar 'list value)))
- (let ((file (orgtbl-to-tsv value '(:fmt org-babel-R-quote-tsv-field)))
+ (let ((file (orgtbl-to-tsv
+ value
+ '( :fmt org-babel-R-quote-tsv-field
+ :with-special-rows nil)))
(header (if (or (eq (nth 1 value) 'hline) colnames-p)
"TRUE" "FALSE"))
(row-names (if rownames-p "1" "NULL")))
diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el
index 0c07aec829..16fbd63f5c 100644
--- a/lisp/ob-awk.el
+++ b/lisp/ob-awk.el
@@ -111,7 +111,11 @@ (defun org-babel-awk-var-to-awk (var &optional sep)
(let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
(cond
((and (listp var) (listp (car var)))
- (orgtbl-to-generic var (list :sep (or sep "\t") :fmt echo-var)))
+ (orgtbl-to-generic
+ var
+ (list :sep (or sep "\t")
+ :fmt echo-var
+ :with-special-rows nil)))
((listp var)
(mapconcat echo-var var "\n"))
(t (funcall echo-var var)))))
diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index 2c53f30664..b7598adf0b 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -317,6 +317,7 @@ (defun org-babel-gnuplot-table-to-data (table data-file params)
`( :sep "\t" :fmt org-babel-gnuplot-quote-tsv-field
;; Two setting below are needed to make :fmt work.
:raw t
+ :with-special-rows nil
:backend ,ob-gnuplot-data)
params)))))
data-file)
diff --git a/lisp/ob-julia.el b/lisp/ob-julia.el
index f9da734b37..9ed0ff80a6 100644
--- a/lisp/ob-julia.el
+++ b/lisp/ob-julia.el
@@ -171,7 +171,10 @@ (defun org-babel-julia-assign-elisp (name value)
(min (if lengths (apply #'min lengths) 0)))
;; Ensure VALUE has an orgtbl structure (depth of at least 2).
(unless (listp (car value)) (setq value (list value)))
- (let ((file (orgtbl-to-csv value '(:fmt org-babel-julia-quote-csv-field))))
+ (let ((file (orgtbl-to-csv
+ value
+ '( :fmt org-babel-julia-quote-csv-field
+ :with-special-rows nil))))
(if (= max min)
(format "%s = begin
using CSV
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 7706234919..535729d2bb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,8 +263,11 @@ (defun org-babel-sh-var-to-string (var &optional sep hline)
(let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
(cond
((and (listp var) (or (listp (car var)) (eq (car var) 'hline)))
- (orgtbl-to-generic var (list :sep (or sep "\t") :fmt echo-var
- :hline hline)))
+ (orgtbl-to-generic
+ var
+ (list :sep (or sep "\t") :fmt echo-var
+ :hline hline
+ :with-special-rows nil)))
((listp var)
(mapconcat echo-var var "\n"))
(t (funcall echo-var var)))))
diff --git a/lisp/ob-sql.el b/lisp/ob-sql.el
index d7bcaa0973..8795934520 100644
--- a/lisp/ob-sql.el
+++ b/lisp/ob-sql.el
@@ -418,9 +418,11 @@ (defun org-babel-sql-expand-vars (body vars &optional sqlite)
(insert (orgtbl-to-csv
val (if sqlite
nil
- '(:fmt (lambda (el) (if (stringp el)
- el
- (format "%S" el))))))))
+ '( :fmt (lambda (el)
+ (if (stringp el)
+ el
+ (format "%S" el)))
+ :with-special-rows nil)))))
data-file)
(if (stringp val) val (format "%S" val))))
body t t)))
diff --git a/lisp/org-plot.el b/lisp/org-plot.el
index 836cfaffca..7f21a4cd55 100644
--- a/lisp/org-plot.el
+++ b/lisp/org-plot.el
@@ -140,7 +140,8 @@ (defun org-plot/gnuplot-to-data (table data-file params)
(insert (orgtbl-to-generic
table
(org-combine-plists
- '(:sep "\t" :fmt org-plot-quote-tsv-field)
+ '( :sep "\t" :fmt org-plot-quote-tsv-field
+ :with-special-rows nil)
params)))))
nil)
--
2.47.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-12-14 17:21 ` Ihor Radchenko
@ 2024-12-15 5:04 ` Suhail Singh
2024-12-22 10:38 ` Ihor Radchenko
1 sibling, 0 replies; 7+ messages in thread
From: Suhail Singh @ 2024-12-15 5:04 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: William Denton, Emacs Org mode mailing list
Ihor Radchenko <yantar92@posteo.net> writes:
> Tentatively, I am leaning towards restoring the old behavior for the
> time being, until we can come up with something better.
I think doing so is prudent.
Further, I think it would help if the manner in which special rows are
handled were user-configurable. Once we have that in place, then the
old behaviour can be retained by choosing appropriate default values for
the babel backends you mentioned. Then, finally, as a subsequent step a
single default can perhaps be set to improve consistency across babel
backends (along with a mention in the NEWS, appropriate version bump
etc.)
If the above is done, I would argue that, eventually, the default
treatment where special rows are elided would be the more useful
setting.
--
Suhail
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-11-15 14:23 ` William Denton
@ 2024-12-17 2:40 ` William Denton
0 siblings, 0 replies; 7+ messages in thread
From: William Denton @ 2024-12-17 2:40 UTC (permalink / raw)
Cc: Emacs Org mode mailing list
Sorry to confuse the threading, but I wanted to bring this back up given Ihor and Suhail Singh talking about this change in R behaviour.
I agree it seems best to revert to the earlier was of handling data tables in R until, if possible, there is a nice way to clean up all the languages at once. The whole situation may be confusing, since there's a mix of how languages handle them, but it doesn't break anything. It seems like a small thing to break a lot of R code over.
I don't know if this bug will matter after that, but I wanted to make sure it got noted just in case.
Thanks,
Bill
--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada
On Friday, November 15th, 2024 at 09:23, William Denton <william@williamdenton.org> wrote:
> On Friday, November 15th, 2024 at 06:59, Rens Oliemans hallo@rensoliemans.nl wrote:
>
> > According to git bisect, the first bad commit is:
> > [e0924db3c55a44c8aa8d126fbb9b42cfc54f104c] orgtbl-to-generic: Retain special rows in code block table output
>
>
> Aha! Thank you. I wasn't looking at tables generally. I see now that ORG-NEWS has a "~orgtbl-to-generic~ retains special rows when exporting to Org" entry. It says, "To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argument." But:
>
> #+name: test_table
> | Date | Weather |
> | <10> | <50> |
>
> |------------+----------------------|
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
>
> #+begin_src R :var t=test_table :colnames yes :with-special-rows nil
> t
> #+end_src
>
> #+RESULTS:
> | Date | Weather |
> |------------+----------------------|
> | <10> | <50> |
>
> | 2024-11-01 | Warm |
> | 2024-11-02 | Warm |
> | 2024-11-03 | Still strangely warm |
>
> If I (setq org-org-with-special-rows nil) then this stops, but shouldn't the parameter work?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Table width cookies become part of data table in R
2024-12-14 17:21 ` Ihor Radchenko
2024-12-15 5:04 ` Suhail Singh
@ 2024-12-22 10:38 ` Ihor Radchenko
1 sibling, 0 replies; 7+ messages in thread
From: Ihor Radchenko @ 2024-12-22 10:38 UTC (permalink / raw)
To: William Denton; +Cc: Emacs Org mode mailing list
Ihor Radchenko <yantar92@posteo.net> writes:
> Tentatively, I am leaning towards restoring the old behavior for the
> time being, until we can come up with something better.
>
> Patch attached.
Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=c93ddd23bf
--
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-12-22 10:38 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-11-15 4:29 Table width cookies become part of data table in R William Denton
2024-11-15 11:59 ` Rens Oliemans
2024-11-15 14:23 ` William Denton
2024-12-17 2:40 ` William Denton
2024-12-14 17:21 ` Ihor Radchenko
2024-12-15 5:04 ` Suhail Singh
2024-12-22 10:38 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).