[Bug] #+call does not respect :colnames argument

[Bug] #+call does not respect :colnames argument

[Rick Frankel]

Eric-

I have debugged why the :colnames argument is not respected as to
removing the header from a table var.

* Given
#+name: with-hline
| A | B | C |
|---+---+---|
| 1 | 2 | 3 |
| 4 | 5 | 6 |

#+name: with-hline2
| B | C | D |
|---+---+---|
| 2 | 3 | 4 |
| 5 | 6 | 7 |

#+name: emacs-echo
#+BEGIN_SRC emacs-lisp :var table=with-hline :colnames yes
(mapcar (lambda (x) (mapcar '1+ x)) table)
#+END_SRC

#+call: emacs-echo(table=with-hline2)[:colnames yes]

* Evaluation
When the `#+call' line is executed, `org-babel-process-params' is
called multiple times. When it is called the last time to process
the actual input (here, calling =emacs-echo= with the param
=with-hline2=), the `:colname-names' parameter has already been (to
=(table "A" "B" "C")=, taken from the variable in the source blocks
header) as a side-effect of the call to
`org-babel-get-src-block-info'. So, when the block is executed later
in the function, then next call to `org-babel-process-params' does
not call disassemble table (taking into account the value of
`:colnames') but uses =processed-vars= directly instead, causing the 
full
table to be used without calling `org-babel-disassemble-tables'.

Here's the relevant section of code from `org-babel-process-params':

#+BEGIN_SRC emacs-lisp
(if (and (assoc :colname-names params)
(assoc :rowname-names params))
(list processed-vars)
(org-babel-disassemble-tables
processed-vars
(cdr (assoc :hlines params))
(cdr (assoc :colnames params))
(cdr (assoc :rownames params))))
#+END_SRC

And here's a cleaned-up debug trace:

#+BEGIN_EXAMPLE
(list processed-vars)
;; called again later in execute-src-block, does not properly parse 
=with-hline2= var
org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
"no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
"results") (:results . "silent") (:var . "table=with-hline2") (:hlines . 
"yes") (:padnewline . "yes") (:session . "none") (:colnames . "yes") 
(:result-type . value) (:result-params "replace") (:rowname-names) 
(:colname-names (table "A" "B" "C"))))
;; calls disassemble-tables and returns with :colname-names set
org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
"no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
"results") (:results . "replace") (:var . "table=with-hline") (:colnames 
. "yes") (:session . "none") (:padnewline . "yes") (:hlines . "yes")))
org-babel-get-src-block-info()
org-babel-execute-src-block(nil nil ((:var . "table=with-hline2") 
(:results . "silent")))
org-babel-ref-resolve("emacs-echo(table=with-hline2)[:colnames yes]")
org-babel-ref-parse("results=emacs-echo(table=with-hline2)[:colnames 
yes]")
org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
"no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
"results") (:results . "replace") (:var . 
"results=emacs-echo(table=with-hline2)[:colnames yes]") (:colnames . 
"no") (:hlines . "yes") (:padnewline . "yes") (:session . "none")))
org-babel-lob-execute(("emacs-echo(table=with-hline2)[:colnames yes]" 
nil 0 nil))
#+END_EXAMPLE

rick

Re: [Bug] #+call does not respect :colnames argument

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]

Rick Frankel <rick@rickster.com> writes:

> Eric-
>
> I have debugged why the :colnames argument is not respected as to
> removing the header from a table var.
>
> * Given
> #+name: with-hline
> | A | B | C |
> |---+---+---|
> | 1 | 2 | 3 |
> | 4 | 5 | 6 |
>
> #+name: with-hline2
> | B | C | D |
> |---+---+---|
> | 2 | 3 | 4 |
> | 5 | 6 | 7 |
>
> #+name: emacs-echo
> #+BEGIN_SRC emacs-lisp :var table=with-hline :colnames yes
> (mapcar (lambda (x) (mapcar '1+ x)) table)
> #+END_SRC
>
> #+call: emacs-echo(table=with-hline2)[:colnames yes]
>
> * Evaluation
> When the `#+call' line is executed, `org-babel-process-params' is
> called multiple times. When it is called the last time to process
> the actual input (here, calling =emacs-echo= with the param
> =with-hline2=), the `:colname-names' parameter has already been (to
> =(table "A" "B" "C")=, taken from the variable in the source blocks
> header) as a side-effect of the call to
> `org-babel-get-src-block-info'. So, when the block is executed later
> in the function, then next call to `org-babel-process-params' does
> not call disassemble table (taking into account the value of
> `:colnames') but uses =processed-vars= directly instead, causing the 
> full
> table to be used without calling `org-babel-disassemble-tables'.
>
> Here's the relevant section of code from `org-babel-process-params':
>
> #+BEGIN_SRC emacs-lisp
> (if (and (assoc :colname-names params)
> (assoc :rowname-names params))
> (list processed-vars)
> (org-babel-disassemble-tables
> processed-vars
> (cdr (assoc :hlines params))
> (cdr (assoc :colnames params))
> (cdr (assoc :rownames params))))
> #+END_SRC
>
> And here's a cleaned-up debug trace:
>
> #+BEGIN_EXAMPLE
> (list processed-vars)
> ;; called again later in execute-src-block, does not properly parse 
> =with-hline2= var
> org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
> "no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
> "results") (:results . "silent") (:var . "table=with-hline2") (:hlines . 
> "yes") (:padnewline . "yes") (:session . "none") (:colnames . "yes") 
> (:result-type . value) (:result-params "replace") (:rowname-names) 
> (:colname-names (table "A" "B" "C"))))
> ;; calls disassemble-tables and returns with :colname-names set
> org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
> "no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
> "results") (:results . "replace") (:var . "table=with-hline") (:colnames 
> . "yes") (:session . "none") (:padnewline . "yes") (:hlines . "yes")))
> org-babel-get-src-block-info()
> org-babel-execute-src-block(nil nil ((:var . "table=with-hline2") 
> (:results . "silent")))
> org-babel-ref-resolve("emacs-echo(table=with-hline2)[:colnames yes]")
> org-babel-ref-parse("results=emacs-echo(table=with-hline2)[:colnames 
> yes]")
> org-babel-process-params(((:comments . "") (:shebang . "") (:cache . 
> "no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . 
> "results") (:results . "replace") (:var . 
> "results=emacs-echo(table=with-hline2)[:colnames yes]") (:colnames . 
> "no") (:hlines . "yes") (:padnewline . "yes") (:session . "none")))
> org-babel-lob-execute(("emacs-echo(table=with-hline2)[:colnames yes]" 
> nil 0 nil))
> #+END_EXAMPLE
>
> rick

Hi Rick,

Thanks for taking the time to find the root of this problem.  I believe
I've fixed this by change the `org-babel-merge-params' function so that
when the value of a variable is update, then colname-names and
rowname-names values saved for that variable are removed.

In my local tests the attached patch fixes this issue.  If it works for
you as well then I'll apply it.

Thanks,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-delete-colnames-rownames-for-replaced-variables.patch --]
[-- Type: text/x-patch, Size: 2940 bytes --]

From e789367e7ec4badd74e9aafb0249aa8798842f65 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 26 Jul 2013 11:48:51 -0600
Subject: [PATCH] delete colnames/rownames for replaced variables

  Thanks to Rick Frankel for help debugging this problem.

* lisp/ob-core.el (org-babel-merge-params): When merging parameters, if
  a variable is replaced with a new value, then delete colnames/rownames
  for the original value of that variable.
---
 lisp/ob-core.el | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index b213c2a..c2722db 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2269,7 +2269,8 @@ parameters when merging lists."
 				    new-params))
 			    result-params)
 		      output)))
-	 params results exports tangle noweb cache vars shebang comments padline)
+	 params results exports tangle noweb cache vars shebang comments padline
+	 clearnames)
 
     (mapc
      (lambda (plist)
@@ -2286,21 +2287,25 @@ parameters when merging lists."
 		   (setq vars
 			 (append
 			  (if (member name (mapcar #'car vars))
-			      (delq nil
-				    (mapcar
-				     (lambda (p)
-				       (unless (equal (car p) name) p))
-				     vars))
+			      (progn
+				(push name clearnames)
+				(delq nil
+				      (mapcar
+				       (lambda (p)
+					 (unless (equal (car p) name) p))
+				       vars)))
 			    vars)
 			  (list (cons name pair))))
 		 ;; if no name is given and we already have named variables
 		 ;; then assign to named variables in order
 		 (if (and vars (nth variable-index vars))
-		     (prog1 (setf (cddr (nth variable-index vars))
-				  (concat (symbol-name
-					   (car (nth variable-index vars)))
-					  "=" (cdr pair)))
-		       (incf variable-index))
+		     (let ((name (car (nth variable-index vars))))
+		       (push name clearnames) ; clear out colnames
+					      ; and rownames
+					      ; for replace vars
+		       (prog1 (setf (cddr (nth variable-index vars))
+				    (concat (symbol-name name) "=" (cdr pair)))
+			 (incf variable-index)))
 		   (error "Variable \"%s\" must be assigned a default value"
 			  (cdr pair))))))
 	    (:results
@@ -2347,6 +2352,19 @@ parameters when merging lists."
      plists)
     (setq vars (reverse vars))
     (while vars (setq params (cons (cons :var (cddr (pop vars))) params)))
+    ;; clear out col-names and row-names for replaced variables
+    (mapc
+     (lambda (name)
+       (mapc
+	(lambda (param)
+	  (setf (cdr (assoc param params))
+		(remove-if (lambda (pair) (equal (car pair) name))
+			   (cdr (assoc param params))))
+	  (setf params (remove-if (lambda (pair) (and (equal (car pair) param)
+						 (null (cdr pair))))
+				  params)))
+	(list :colname-names :rowname-names)))
+     clearnames)
     (mapc
      (lambda (hd)
        (let ((key (intern (concat ":" (symbol-name hd))))
-- 
1.8.3.3


[-- Attachment #3: Type: text/plain, Size: 46 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [Bug] #+call does not respect :colnames argument

[Rick Frankel]

On Fri, Jul 26, 2013 at 11:53:33AM -0600, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:
> 
> > I have debugged why the :colnames argument is not respected as to
> > removing the header from a table var.

> Thanks for taking the time to find the root of this problem.  I believe
> I've fixed this by change the `org-babel-merge-params' function so that
> when the value of a variable is update, then colname-names and
> rowname-names values saved for that variable are removed.
> 
> In my local tests the attached patch fixes this issue.  If it works for
> you as well then I'll apply it.

Works for me. Thanx, this will simplify a lot of code (e.g., the
example dot-from-tables that was floating around last week)

Aside... It's difficult for me to follow the code, so can you explain
the why the different results between:

#+call: emacs-echo(table=with-hline2) :colnames yes

#+results:
| B | C | D |
|---+---+---|
| 3 | 4 | 5 |
| 6 | 7 | 8 |

and

#+call: emacs-echo(table=with-hline2)[:colnames yes]

#+results:
| B | C | D |


thanx again,
rick

Re: [Bug] #+call does not respect :colnames argument

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> On Fri, Jul 26, 2013 at 11:53:33AM -0600, Eric Schulte wrote:
>> Rick Frankel <rick@rickster.com> writes:
>> 
>> > I have debugged why the :colnames argument is not respected as to
>> > removing the header from a table var.
>
>> Thanks for taking the time to find the root of this problem.  I believe
>> I've fixed this by change the `org-babel-merge-params' function so that
>> when the value of a variable is update, then colname-names and
>> rowname-names values saved for that variable are removed.
>> 
>> In my local tests the attached patch fixes this issue.  If it works for
>> you as well then I'll apply it.
>
> Works for me. Thanx, this will simplify a lot of code (e.g., the
> example dot-from-tables that was floating around last week)
>
> Aside... It's difficult for me to follow the code, so can you explain
> the why the different results between:
>
> #+call: emacs-echo(table=with-hline2) :colnames yes
>
> #+results:
> | B | C | D |
> |---+---+---|
> | 3 | 4 | 5 |
> | 6 | 7 | 8 |
>
> and
>
> #+call: emacs-echo(table=with-hline2)[:colnames yes]
>
> #+results:
> | B | C | D |
>

The later is not valid call line syntax, see [1] for a full description
of the call line syntax.  In effect what happens in the latter case, is
you set :colnames to "yes]".

I've just pushed up that patch.

Thanks for you help debugging this.

>
>
> thanx again,
> rick


Footnotes: 
[1]  (info "(org)Evaluating Code Blocks")

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [Bug] #+call does not respect :colnames argument

[Rick Frankel]

On Fri, Jul 26, 2013 at 06:54:08PM -0600, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:
> The later is not valid call line syntax, see [1] for a full description
> of the call line syntax.  In effect what happens in the latter case, is
> you set :colnames to "yes]".

Oh, ok. I just saw other people using that syntax on the  list so wondered.

> Thanks for you help debugging this.

my pleasure. ust wish i understood the code well enought to provide
patches instead :{.

rick

Re: [Bug] #+call does not respect :colnames argument

[Achim Gratz]

Eric Schulte writes:
> I've just pushed up that patch.

…which breaks testing:

10 unexpected results:
   FAILED  ob-exp/use-case-of-reading-entry-properties
   FAILED  test-ob-header-arg-defaults/global/call
   FAILED  test-ob-header-arg-defaults/global/noweb
   FAILED  test-ob-header-arg-defaults/tree/accumulate/call
   FAILED  test-ob-header-arg-defaults/tree/accumulate/noweb
   FAILED  test-ob-header-arg-defaults/tree/complex/call
   FAILED  test-ob-header-arg-defaults/tree/complex/noweb
   FAILED  test-ob-header-arg-defaults/tree/overwrite/call
   FAILED  test-ob-header-arg-defaults/tree/overwrite/noweb
   FAILED  test-ob-lob/call-with-header-arguments

with 

    (wrong-type-argument consp nil)

None of these tests deal with table arguments AFAICS.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] #+call does not respect :colnames argument

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> I've just pushed up that patch.
>
> …which breaks testing:
>
> 10 unexpected results:
>    FAILED  ob-exp/use-case-of-reading-entry-properties
>    FAILED  test-ob-header-arg-defaults/global/call
>    FAILED  test-ob-header-arg-defaults/global/noweb
>    FAILED  test-ob-header-arg-defaults/tree/accumulate/call
>    FAILED  test-ob-header-arg-defaults/tree/accumulate/noweb
>    FAILED  test-ob-header-arg-defaults/tree/complex/call
>    FAILED  test-ob-header-arg-defaults/tree/complex/noweb
>    FAILED  test-ob-header-arg-defaults/tree/overwrite/call
>    FAILED  test-ob-header-arg-defaults/tree/overwrite/noweb
>    FAILED  test-ob-lob/call-with-header-arguments
>
> with 
>
>     (wrong-type-argument consp nil)
>
> None of these tests deal with table arguments AFAICS.
>

I just pushed up a fix, I now only get 1 failing test locally, but it is
related to table alignment so I believe it must be unrelated to this
commit.

Best,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [Bug] #+call does not respect :colnames argument

[Achim Gratz]

Eric Schulte writes:
> I just pushed up a fix, I now only get 1 failing test locally, but it is
> related to table alignment so I believe it must be unrelated to this
> commit.

I confirm the fix.  As to your oither test fail, it doesn't show up for
me and it most certainly shouldn't show up for you if you started the
test suite via "make test-dirty".  Is that not working for you?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] #+call does not respect :colnames argument

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> I just pushed up a fix, I now only get 1 failing test locally, but it is
>> related to table alignment so I believe it must be unrelated to this
>> commit.
>
> I confirm the fix.  As to your oither test fail, it doesn't show up for
> me and it most certainly shouldn't show up for you if you started the
> test suite via "make test-dirty".  Is that not working for you?
>

With "make test-dirty" I get all tests passing as expected.  The table
alignment error was from running the test suite interactively with the
`org-test-run-all-tests' function, so it is probably due to something
specific to my config.

Thanks,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte