[PATCH] Process hlines in imported tables

[PATCH] Process hlines in imported tables

[Rick Frankel]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

Currently, there is no way to include an hline in an imported or
converted table. The `org-babel-import-elisp-from-file converts lines
starting with '-' (or '|-') to an integer 0, because, even though
`org-table-to-lisp' will correctly convert lines starting with "|-",
because `org-table-convert-region' always puts "| " at the begining of
each line.

This patch solves that by not putting a space after the pipe symbol if
the first character of the line is a dash ("-").

rick

[-- Attachment #2: 0001-When-importing-or-converting-table-convert-rows-star.patch --]
[-- Type: text/plain, Size: 1779 bytes --]

From 86ee5bfcaa7513769cc3e2939c5e0b1a1f3c7706 Mon Sep 17 00:00:00 2001
From: Rick Frankel <rick@rickster.com>
Date: Thu, 28 Mar 2013 21:34:06 -0400
Subject: [PATCH] When importing or converting table, convert rows starting
 with "-" to horizontal (hlines).

* lisp/ob-core.el (org-babel-import-elisp-from-file): Check if row in
  an array or 'hline.
* lisp/org-table.el (org-table-convert-region): Don't put space after
  pipe symbol if line starts with '-', so `org-table-to-lisp' will
  match the row against `org-table-hline-regexp'.
---
 lisp/ob-core.el   | 3 ++-
 lisp/org-table.el | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index a63f77e..93b12b2 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2537,7 +2537,8 @@ If the table is trivial, then return it as a scalar."
 	      (org-table-import file-name separator)
 	      (delete-file file-name)
 	      (setq result (mapcar (lambda (row)
-				     (mapcar #'org-babel-string-read row))
+				     (if (eq row 'hline) 'hline
+				       (mapcar #'org-babel-string-read row)))
 				   (org-table-to-lisp))))
 	  (error (message "Error reading results: %s" err) nil)))
       (if (null (cdr result)) ;; if result is trivial vector, then scalarize it
diff --git a/lisp/org-table.el b/lisp/org-table.el
index f087cf7..da923cc 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -605,7 +605,8 @@ nil      When nil, the command tries to be smart and figure out the
 		   (format "^ *\\| *\t *\\| \\{%d,\\}" separator)))
 		(t (error "This should not happen"))))
       (while (re-search-forward re end t)
-	(replace-match "| " t t)))
+	(replace-match
+	 (if (= (char-after) ?-) "|" "| ") t t)))
     (goto-char beg)
     (org-table-align)))
 
-- 
1.8.2

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> Currently, there is no way to include an hline in an imported or
> converted table. The `org-babel-import-elisp-from-file converts lines
> starting with '-' (or '|-') to an integer 0,

Oh, thanks for pointing this out, I've just pushed up a fix.  The single
"-" was tricking the number checker because it looks like a minus sign
about to be followed by numbers.  These characters are no longer
converted to 0s.

> because, even though `org-table-to-lisp' will correctly convert lines
> starting with "|-", because `org-table-convert-region' always puts "|
> " at the begining of each line.
>
> This patch solves that by not putting a space after the pipe symbol if
> the first character of the line is a dash ("-").
>

Users may want to insert a "-" in their tables, and I think it would be
surprising to magically replace floating "-" characters with hlines.
There are numerous existing options for inserting hlines into tables,
e.g., the :colnames header argument, using the raw, wrap and org result
types and printing literal Org-mode syntax from your block, additionally
any result could be passed through an elisp code block which may insert
hline symbols at will.

Is there a specific use case which isn't addressed by the existing
functionality?

Thanks,

>
> rick
>
> From 86ee5bfcaa7513769cc3e2939c5e0b1a1f3c7706 Mon Sep 17 00:00:00 2001
> From: Rick Frankel <rick@rickster.com>
> Date: Thu, 28 Mar 2013 21:34:06 -0400
> Subject: [PATCH] When importing or converting table, convert rows starting
>  with "-" to horizontal (hlines).
>
> * lisp/ob-core.el (org-babel-import-elisp-from-file): Check if row in
>   an array or 'hline.
> * lisp/org-table.el (org-table-convert-region): Don't put space after
>   pipe symbol if line starts with '-', so `org-table-to-lisp' will
>   match the row against `org-table-hline-regexp'.
> ---
>  lisp/ob-core.el   | 3 ++-
>  lisp/org-table.el | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/lisp/ob-core.el b/lisp/ob-core.el
> index a63f77e..93b12b2 100644
> --- a/lisp/ob-core.el
> +++ b/lisp/ob-core.el
> @@ -2537,7 +2537,8 @@ If the table is trivial, then return it as a scalar."
>  	      (org-table-import file-name separator)
>  	      (delete-file file-name)
>  	      (setq result (mapcar (lambda (row)
> -				     (mapcar #'org-babel-string-read row))
> +				     (if (eq row 'hline) 'hline
> +				       (mapcar #'org-babel-string-read row)))
>  				   (org-table-to-lisp))))
>  	  (error (message "Error reading results: %s" err) nil)))
>        (if (null (cdr result)) ;; if result is trivial vector, then scalarize it
> diff --git a/lisp/org-table.el b/lisp/org-table.el
> index f087cf7..da923cc 100644
> --- a/lisp/org-table.el
> +++ b/lisp/org-table.el
> @@ -605,7 +605,8 @@ nil      When nil, the command tries to be smart and figure out the
>  		   (format "^ *\\| *\t *\\| \\{%d,\\}" separator)))
>  		(t (error "This should not happen"))))
>        (while (re-search-forward re end t)
> -	(replace-match "| " t t)))
> +	(replace-match
> +	 (if (= (char-after) ?-) "|" "| ") t t)))
>      (goto-char beg)
>      (org-table-align)))

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Rick Frankel]

On Fri, Mar 29, 2013 at 09:04:42AM -0600, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:
> 
> Users may want to insert a "-" in their tables, and I think it would be
> surprising to magically replace floating "-" characters with hlines.
> There are numerous existing options for inserting hlines into tables,
> e.g., the :colnames header argument, using the raw, wrap and org result
> types and printing literal Org-mode syntax from your block, additionally
> any result could be passed through an elisp code block which may insert
> hline symbols at will.
> 
> Is there a specific use case which isn't addressed by the existing
> functionality?

Yes and no. :colnames works, but often the header comes from the
processing, so they may not be static (I use a lot of call:s). Also,
I've been having trouble using the output from raw results as input --
it seems that unless the results are cached (:cache yes), the table is
not parsed on input, but passed as a multiline string. I was hoping to
avoid this problem using value returns (now that  Achim has made the
perl parsing work better). Here's an example (btw, this breaks in 7.4
as well):

* Cache vs. uncached raw
#+name: uncached
#+begin_src elisp :results raw
  "|c1|c2|
  |-
  |a|1|
  |b|2|"
#+end_src

#+call: uncached()

#+results: uncached()
: |c1|c2|
: |-
: |a|1|
: |b|2|


#+name: cached
#+begin_src elisp :results raw :cache yes
  "|c1|c2|
  |-
  |a|1|
  |b|2|"
#+end_src

#+results[62ca3004bf7cb363e47635216b3289cfdc39684c]: cached
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |

#+call: cached()

#+results: cached()
| a | 1 |
| b | 2 |

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Rick Frankel <rick@rickster.com> writes:

> On Fri, Mar 29, 2013 at 09:04:42AM -0600, Eric Schulte wrote:
>> Rick Frankel <rick@rickster.com> writes:
>> 
>> Users may want to insert a "-" in their tables, and I think it would be
>> surprising to magically replace floating "-" characters with hlines.
>> There are numerous existing options for inserting hlines into tables,
>> e.g., the :colnames header argument, using the raw, wrap and org result
>> types and printing literal Org-mode syntax from your block, additionally
>> any result could be passed through an elisp code block which may insert
>> hline symbols at will.
>> 
>> Is there a specific use case which isn't addressed by the existing
>> functionality?
>
> Yes and no. :colnames works, but often the header comes from the
> processing, so they may not be static (I use a lot of call:s). Also,
> I've been having trouble using the output from raw results as input --
> it seems that unless the results are cached (:cache yes), the table is
> not parsed on input, but passed as a multiline string. I was hoping to
> avoid this problem using value returns (now that  Achim has made the
> perl parsing work better). Here's an example (btw, this breaks in 7.4
> as well):
>

Alright, I've just pushed up changes so that org and wrap results will
expand tables (not just raw).  With this change in place you can now use
":results wrap" to get the results you want, and since they are
delimited, you can then re-use these results in later code blocks.

Here's my version of your example run with the latest from git.  Notice
that it is necessary to add header arguments to call lines (because
these arguments control how results are inserted they must be associated
with the call).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cache.org --]
[-- Type: text/x-org, Size: 693 bytes --]

* Cache vs. uncached org
#+name: uncached
#+begin_src elisp :results wrap
  "|c1|c2|
  |-
  |a|1|
  |b|2|"
#+end_src

#+RESULTS: uncached
:RESULTS:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |
:END:

#+call: uncached() :results wrap

#+RESULTS: uncached():results wrap
:RESULTS:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |
:END:

#+name: cached
#+begin_src elisp :results wrap :cache yes
  "|c1|c2|
  |-
  |a|1|
  |b|2|"
#+end_src

#+RESULTS[0faf1fbb046c9e001622c49242322c225811b162]: cached
:RESULTS:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |
:END:

#+call: cached() :results wrap

#+RESULTS: cached():results wrap
:RESULTS:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |
:END:

[-- Attachment #3: Type: text/plain, Size: 55 bytes --]


Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Rick Frankel]

On Fri, Mar 29, 2013 at 06:01:21PM -0600, Eric Schulte wrote:
> > Yes and no. :colnames works, but often the header comes from the
> > processing, so they may not be static (I use a lot of call:s). Also,
> > I've been having trouble using the output from raw results as input --
> > it seems that unless the results are cached (:cache yes), the table is
> > not parsed on input, but passed as a multiline string. I was hoping to
> > avoid this problem using value returns (now that  Achim has made the
> > perl parsing work better). Here's an example (btw, this breaks in 7.4
> > as well):
> >
> 
> Alright, I've just pushed up changes so that org and wrap results will
> expand tables (not just raw).  With this change in place you can now use
> ":results wrap" to get the results you want, and since they are
> delimited, you can then re-use these results in later code blocks.
> 

Better for the elisp. But perl table processing is now totally wacky:

*Note* =wrap= and =raw= give same results
#+begin_src perl :results raw
  q[|c1|c2|
  |-
  |a|1|
  |b|2|];
#+end_src

#+results:
|   | c1 | c2 |
|   | -  |    |
|   | a  |  1 |
|   | b  |  2 |

#+begin_src perl :results raw
  q[c1|c2
  -
  a|1
  b|2];
#+end_src

#+results:
| c1 | c2 |
| -  |    |
| a  |  1 |
| b  |  2 |

#+begin_src perl :results raw output
  print q[|c1|c2|
  |-
  |a|1|
  |b|2|
  ];
#+end_src

#+results:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> On Fri, Mar 29, 2013 at 06:01:21PM -0600, Eric Schulte wrote:
>> > Yes and no. :colnames works, but often the header comes from the
>> > processing, so they may not be static (I use a lot of call:s). Also,
>> > I've been having trouble using the output from raw results as input --
>> > it seems that unless the results are cached (:cache yes), the table is
>> > not parsed on input, but passed as a multiline string. I was hoping to
>> > avoid this problem using value returns (now that  Achim has made the
>> > perl parsing work better). Here's an example (btw, this breaks in 7.4
>> > as well):
>> >
>> 
>> Alright, I've just pushed up changes so that org and wrap results will
>> expand tables (not just raw).  With this change in place you can now use
>> ":results wrap" to get the results you want, and since they are
>> delimited, you can then re-use these results in later code blocks.
>> 
>
> Better for the elisp. But perl table processing is now totally wacky:
>
> *Note* =wrap= and =raw= give same results
> #+begin_src perl :results raw
>   q[|c1|c2|
>   |-
>   |a|1|
>   |b|2|];
> #+end_src
>
> #+results:
> |   | c1 | c2 |
> |   | -  |    |
> |   | a  |  1 |
> |   | b  |  2 |
>

This is a problem in the results returned by ob-perl, not in the results
insertion mechanism.  Given what is actually being returned by that code
block the results make sense.

    #+name: perl-example
    #+begin_src perl :results raw
      q[|c1|c2|
      |-
      |a|1|
      |b|2|];
    #+end_src

    #+begin_src emacs-lisp :var data=perl-example :results verbatim
      (format "%S" data)
    #+end_src

    #+RESULTS:
    : "((\"\" \"c1\" \"c2\") (\"\" \"-\" \"\") (\"\" \"a\" 1) (\"\" \"b\" 2))"

If we add verbatim (which inhibits interpretation as a value, which can
often result in a list or table result), then we get what I assume you
expect.

    #+name: perl-example
    #+begin_src perl :results verbatim raw
      q[|c1|c2|
      |-
      |a|1|
      |b|2|];
    #+end_src

    #+RESULTS: perl-example
    | c1 | c2 |
    |----+----|
    | a  |  1 |
    | b  |  2 |

>
> #+begin_src perl :results raw
>   q[c1|c2
>   -
>   a|1
>   b|2];
> #+end_src
>
> #+results:
> | c1 | c2 |
> | -  |    |
> | a  |  1 |
> | b  |  2 |
>

This output above makes sense.  Maybe try the following (with verbatim)
instead.

    #+begin_src perl :results verbatim drawer
      q[|c1|c2
      |-
      |a|1
      |b|2];
    #+end_src

    #+results:
    :RESULTS:
    | c1 | c2 |
    |----+----|
    | a  |  1 |
    | b  |  2 |
    :END:

>
> #+begin_src perl :results raw output
>   print q[|c1|c2|
>   |-
>   |a|1|
>   |b|2|
>   ];
> #+end_src
>
> #+results:
> | c1 | c2 |
> |----+----|
> | a  |  1 |
> | b  |  2 |
>

This one looks good to me as is.  I added a note about verbatim to [1],
if you can think anything else from this discussion that could be of
general interest please place it there as well.

Thanks,

Footnotes: 
[1]  http://orgmode.org/worg/org-contrib/babel/header-args.html

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Rick Frankel]

On Sat, Mar 30, 2013 at 06:43:30PM -0600, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:

> > *Note* =wrap= and =raw= give same results
> > #+begin_src perl :results raw
> >   q[|c1|c2|
> >   |-
> >   |a|1|
> >   |b|2|];
> > #+end_src
> >
> > #+results:
> > |   | c1 | c2 |
> > |   | -  |    |
> > |   | a  |  1 |
> > |   | b  |  2 |
> >
> 
> This is a problem in the results returned by ob-perl, not in the results
> insertion mechanism.  Given what is actually being returned by that code
> block the results make sense.
> 
>     #+name: perl-example
>     #+begin_src perl :results raw
>       q[|c1|c2|
>       |-
>       |a|1|
>       |b|2|];
>     #+end_src

> If we add verbatim (which inhibits interpretation as a value, which can
> often result in a list or table result), then we get what I assume you
> expect.

>     #+name: perl-example
>     #+begin_src perl :results verbatim raw
>       q[|c1|c2|
>       |-
>       |a|1|
>       |b|2|];
>     #+end_src
> 
>     #+RESULTS: perl-example
>     | c1 | c2 |
>     |----+----|
>     | a  |  1 |
>     | b  |  2 |
> 
Missed verbatim. Thanks for the pointer, it works, but i think that
perl is double-processing returned values. If we do the same things in
elisp i get (my) expected results:

#+begin_src elisp :results raw
  "|c1|c2|
  |-
  |a|1|
  |b|2|";
#+end_src

#+results:
| c1 | c2 |
|----+----|
| a  |  1 |
| b  |  2 |

*NOTE* this will not properly clean-up the results, but keep inserting
more copies.
#+begin_src elisp :results raw verbatim
  "|c1|c2|
  |-
  |a|1|
  |b|2|";
#+end_src

#+results:
"|c1|c2|
|-
|a|1|
|b|2|"

> > #+begin_src perl :results raw output
> >   print q[|c1|c2|
> >   |-
> >   |a|1|
> >   |b|2|
> >   ];

Yes, this was included to show the inconsistency between value and
output returns. BTW, "raw output" and  "raw output verbatim" generate
the same results.

> 
> Footnotes: 
> [1]  http://orgmode.org/worg/org-contrib/babel/header-args.html

I will take a look and see if i come up with anything else. In
general, what I have found with babel processing is that there is
major inconsistency between the results processing in different
languages. Somehow, I would think that the same  output (modulo
sytnactic diferences) should generate the same results regardless of
language.

rick

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> On Sat, Mar 30, 2013 at 06:43:30PM -0600, Eric Schulte wrote:
>> Rick Frankel <rick@rickster.com> writes:
>
>> > *Note* =wrap= and =raw= give same results
>> > #+begin_src perl :results raw
>> >   q[|c1|c2|
>> >   |-
>> >   |a|1|
>> >   |b|2|];
>> > #+end_src
>> >
>> > #+results:
>> > |   | c1 | c2 |
>> > |   | -  |    |
>> > |   | a  |  1 |
>> > |   | b  |  2 |
>> >
>> 
>> This is a problem in the results returned by ob-perl, not in the results
>> insertion mechanism.  Given what is actually being returned by that code
>> block the results make sense.
>> 
>>     #+name: perl-example
>>     #+begin_src perl :results raw
>>       q[|c1|c2|
>>       |-
>>       |a|1|
>>       |b|2|];
>>     #+end_src
>
>> If we add verbatim (which inhibits interpretation as a value, which can
>> often result in a list or table result), then we get what I assume you
>> expect.
>
>>     #+name: perl-example
>>     #+begin_src perl :results verbatim raw
>>       q[|c1|c2|
>>       |-
>>       |a|1|
>>       |b|2|];
>>     #+end_src
>> 
>>     #+RESULTS: perl-example
>>     | c1 | c2 |
>>     |----+----|
>>     | a  |  1 |
>>     | b  |  2 |
>> 
> Missed verbatim. Thanks for the pointer, it works, but i think that
> perl is double-processing returned values. If we do the same things in
> elisp i get (my) expected results:
>

Yes, because Emacs Lisp is the language of Emacs, it's results aren't
interpreted as are the results of every other language.  This was an
intentional design decision, and it gives extra flexibility when working
with Emacs Lisp.

>
> #+begin_src elisp :results raw
>   "|c1|c2|
>   |-
>   |a|1|
>   |b|2|";
> #+end_src
>
> #+results:
> | c1 | c2 |
> |----+----|
> | a  |  1 |
> | b  |  2 |
>
> *NOTE* this will not properly clean-up the results, but keep inserting
> more copies.

Yes, this is why ":results drawer" exists.

> 
> #+begin_src elisp :results raw verbatim "|c1|c2|
>   |-
>   |a|1|
>   |b|2|";
> #+end_src
>
> #+results:
> "|c1|c2|
> |-
> |a|1|
> |b|2|"
>
>> > #+begin_src perl :results raw output
>> >   print q[|c1|c2|
>> >   |-
>> >   |a|1|
>> >   |b|2|
>> >   ];
>
> Yes, this was included to show the inconsistency between value and
> output returns. BTW, "raw output" and  "raw output verbatim" generate
> the same results.
>

Yes, output collects the code block result from STDOUT.  The strings
collected from STDOUT are not treated as values, but rather as raw
strings.  This should be true across every language.

>
>> 
>> Footnotes: 
>> [1]  http://orgmode.org/worg/org-contrib/babel/header-args.html
>
> I will take a look and see if i come up with anything else.

Please do, I think this page could become a great resource.

> In general, what I have found with babel processing is that there is
> major inconsistency between the results processing in different
> languages.

It is certainly true that Emacs Lisp is treated differently than all
other languages.  There are also significant differences between
languages, e.g., session evaluation doesn't make sense for some
languages, and for other languages session evaluation is the only type
of evaluation that does make sense.

In addition to that, with over 40 supported languages [1], There
undoubtedly are cases where we are doing a bad job of ensuring
inter-languages consistency.  If you can find concise examples which use
demonstrate significant inconsistencies between languages, then we
should be able to resolve those on a case by case basis.  In general I
think ob-sh is probably the most widely used code block language, and
can be treated as the gold standard against which other languages should
be compared.

> Somehow, I would think that the same output (modulo sytnactic
> diferences) should generate the same results regardless of language.
>

I agree, modulo what I said about emacs-lisp.  Please submit specific
reproducible examples and we can get to work fixing them.

Thanks,

>
> rick


Footnotes: 
[1]  Specifically, from the current master branch.
     ob-asymptote.el
     ob-awk.el
     ob-calc.el
     ob-C.el
     ob-clojure.el
     ob-css.el
     ob-ditaa.el
     ob-dot.el
     ob-emacs-lisp.el
     ob-fortran.el
     ob-gnuplot.el
     ob-haskell.el
     ob-io.el
     ob-java.el
     ob-js.el
     ob-keys.el
     ob-latex.el
     ob-ledger.el
     ob-lilypond.el
     ob-lisp.el
     ob-makefile.el
     ob-matlab.el
     ob-maxima.el
     ob-mscgen.el
     ob-ocaml.el
     ob-octave.el
     ob-org.el
     ob-perl.el
     ob-picolisp.el
     ob-plantuml.el
     ob-python.el
     ob-R.el
     ob-ruby.el
     ob-sass.el
     ob-scala.el
     ob-scheme.el
     ob-screen.el
     ob-sh.el
     ob-shen.el
     ob-sql.el
     ob-sqlite.el
     ob-tangle.el


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

babel results handling (was: Process hlines in imported tables)

[Rick Frankel]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On Sun, Mar 31, 2013 at 07:37:38AM -0600, Eric Schulte wrote:
> It is certainly true that Emacs Lisp is treated differently than all
> other languages.  There are also significant differences between
> languages, e.g., session evaluation doesn't make sense for some
> languages, and for other languages session evaluation is the only type
> of evaluation that does make sense.
> 
> In addition to that, with over 40 supported languages [1], There
> undoubtedly are cases where we are doing a bad job of ensuring
> inter-languages consistency.  If you can find concise examples which use
> demonstrate significant inconsistencies between languages, then we
> should be able to resolve those on a case by case basis.  In general I
> think ob-sh is probably the most widely used code block language, and
> can be treated as the gold standard against which other languages should
> be compared.

`sh' is probably not the best choice as a "gold standard" due to the
fact that it only supports STDOUT ("output" and not "value").

Many of the languages are obviously not general purpose, or do not
support (like shell), wrapped values (only STDOUT), or don't generate
text, so consistency does not matter (e.g., css, sass, sql, sqlite,
plantuml, dot).

Regardless, the attached org file is a first step an comparing the
result processing of some languages (specifically, sh, emacs-lisp,
perl and ruby), which, I think, covers a good portion of the babel use
of general purpose languages.

The upshot, is that perl value results match shell value/output
results and emacs-lisp, python and ruby all return about the same
results (elisp returns the quote characters from a verbatim string).

I still think that the scalar pipe-delimited processing from shell and
perl is wrong in that pipe-delimited data ends up w/ an extra column
if each line starts w/ a pipe, but not if the pipe is used like a
csv separator (between columns but not at the beginning or end of the
line).

Also, looking at the manual
(http://orgmode.org/manual/results.html#results) vs. the code, are
there are actually four classes of :results arguments with type broken
into type and format?

     - Type :: (file list vector table scalar verbatim)
     - Format :: (raw html latex org code pp drawer)

rick

[-- Attachment #2: results.org --]
[-- Type: text/plain, Size: 3142 bytes --]

* Value results procesing

The list of valid =:result= arguments
#+BEGIN_SRC elisp :results list
(cdr (assoc 'results org-babel-common-header-args-w-values))
#+END_SRC
#+results:
- (file list vector table scalar verbatim)
- (raw html latex org code pp drawer)
- (replace silent none append prepend)
- (output value)

You can add languages to this list to test them as well.
The genrator assumes that a double quoted string, with the escapes
=\n= and =\t= are valid in the language and that no statement
terminator is required. The "method" is the statement required for
the value to be returned to the wrapper code. (E.g., none for =perl=
and =ruby=, =sh= does not actually return values so needs an =echo=.)

#+NAME: languages
| language | method | array                      |
|----------+--------+----------------------------|
| sh       | echo   |                            |
| perl     |        | [[qw[h1 h2]],[qw[r1 r2]]]  |
| ruby     |        | [%w[h1 h2],%w[r1 r2]]      |
| elisp    |        | '((h1 h2) (r1 r2))         |
| python   | return | [["h1", "h2"],["r1","r2"]] |

** Generate test code                                             :noexport:
#+NAME: block
#+BEGIN_SRC elisp :eval never
  (defun block (lang body function handler type)
    (format
     "**** %s body, :results value %s %s
  ,#+BEGIN_SRC %s :results value %s %s :wrap example
    %s %s
  ,#+END_SRC\n"
     (cond 
      ((not (not (string-match-p "^\"|" body))) "pipe delimited")
      ((not (not (string-match-p "^\"" body))) "tab delimited")
      (t "array"))
     handler type
     lang handler type function body))
#+END_SRC

Run this block to generate the test code. You can then execute the
test code by running =org-babel-execute-subtree= on [[*All%20Tests][All Tests]]. The
file exports resonably nicely to either HTML or LaTeX.

#+HEADER: :var languages=languages
#+HEADER: :var types='("" "table" "scalar" "verbatim")
#+HEADER: :var formatting='("" "raw")
#+HEADER: :var scalar="\"|h1|h2|\\n|-\\n|r1|r2|\""
#+HEADER: :var tab-delim="\"h1\\th2\\t\\nr1\\tr2\""
#+BEGIN_SRC elisp :results raw :noweb yes   
  (require 'ob-perl)
  (require 'ob-sh)
  (require 'ob-python)
  <<block>>
  (save-excursion
    (condition-case nil
        (progn
          (goto-char (org-find-entry-with-id "ALL-TESTS"))
          (org-cut-subtree))
      (error t)))
  (concat
   "* All Tests\n"
   ":PROPERTIES:\n"
   ":ID:  ALL-TESTS\n"
   ":END:\n"
   (mapconcat
    (lambda (lang)
      (format 
       "** %s\n%s" (car lang)
       (mapconcat
        (lambda (formatter)
          (format "*** Result format: %s\n%s" 
                  (if (string= formatter "") "default" formatter)
                  (mapconcat 
                   (lambda (type)
                     (let ((l (car lang)) (pfx (nth 1 lang)) (array (nth 2 lang)))
                       (concat
                        (block l scalar pfx formatter type)
                        (block l tab-delim pfx formatter type)
                        (unless (string= "" array)
                          (block l array pfx formatter type))))) types "\n")))
        formatting ""))) languages ""))
#+END_SRC

Re: babel results handling

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> On Sun, Mar 31, 2013 at 07:37:38AM -0600, Eric Schulte wrote:
>> It is certainly true that Emacs Lisp is treated differently than all
>> other languages.  There are also significant differences between
>> languages, e.g., session evaluation doesn't make sense for some
>> languages, and for other languages session evaluation is the only type
>> of evaluation that does make sense.
>> 
>> In addition to that, with over 40 supported languages [1], There
>> undoubtedly are cases where we are doing a bad job of ensuring
>> inter-languages consistency.  If you can find concise examples which use
>> demonstrate significant inconsistencies between languages, then we
>> should be able to resolve those on a case by case basis.  In general I
>> think ob-sh is probably the most widely used code block language, and
>> can be treated as the gold standard against which other languages should
>> be compared.
>
> `sh' is probably not the best choice as a "gold standard" due to the
> fact that it only supports STDOUT ("output" and not "value").
>
> Many of the languages are obviously not general purpose, or do not
> support (like shell), wrapped values (only STDOUT), or don't generate
> text, so consistency does not matter (e.g., css, sass, sql, sqlite,
> plantuml, dot).
>
> Regardless, the attached org file is a first step an comparing the
> result processing of some languages (specifically, sh, emacs-lisp,
> perl and ruby), which, I think, covers a good portion of the babel use
> of general purpose languages.
>

This is a great file.  Thanks for generating it and sharing it.
Although I think it would be more useful if languages were the smallest
scale of organization rather than the largest to make cross-language
comparison easier.

Would it be difficult to add another set of code blocks which
automatically compare the output of these automatically generated code
blocks, indicating when there are differences.

>
> The upshot, is that perl value results match shell value/output
> results and emacs-lisp, python and ruby all return about the same
> results (elisp returns the quote characters from a verbatim string).
>

What are the perl-shell vs. python-ruby-elisp differences?

>
> I still think that the scalar pipe-delimited processing from shell and
> perl is wrong in that pipe-delimited data ends up w/ an extra column
> if each line starts w/ a pipe, but not if the pipe is used like a
> csv separator (between columns but not at the beginning or end of the
> line).
>

If you want to use pipes to delimit data, then I'd suggest *not*
interpreting the data as a value, but rather doing something like
":results verbatim drawer".  Generally pipes aren't considered to be
table column delimiters, I'd try tabs or spaces instead.

>
> Also, looking at the manual
> (http://orgmode.org/manual/results.html#results) vs. the code, are
> there are actually four classes of :results arguments with type broken
> into type and format?
>
>      - Type :: (file list vector table scalar verbatim)
>      - Format :: (raw html latex org code pp drawer)
>

Yes, this does seem to be more clear.  If you're willing to supply a
documentation patch I'd be very happy to apply it.

Thanks!

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: babel results handling

[Achim Gratz]

Eric Schulte writes:
> Would it be difficult to add another set of code blocks which
> automatically compare the output of these automatically generated code
> blocks, indicating when there are differences.

I'd lobby for integration into the test framework.

>> I still think that the scalar pipe-delimited processing from shell and
>> perl is wrong in that pipe-delimited data ends up w/ an extra column
>> if each line starts w/ a pipe, but not if the pipe is used like a
>> csv separator (between columns but not at the beginning or end of the
>> line).
>>
>
> If you want to use pipes to delimit data, then I'd suggest *not*
> interpreting the data as a value, but rather doing something like
> ":results verbatim drawer".  Generally pipes aren't considered to be
> table column delimiters, I'd try tabs or spaces instead.

Yes they are (by accident mostly as a side effect of how the table gets
imported), but only for Babel languages (except elisp) that can return
tables as values.  I only recently implemented the necessary processing
for Perl.  The returned string gets interpreted as a table if it is
multiline and/or has pipe symbol in it.  Normally the separator should
be a TAB character, but pipes work just the same (the actual
interpretation is done by org-table-convert-region, which also inserts
the leading "| " that Rick is complaining about).

If you want to have Org interpret the table the way Rick seems to want,
then the string must be a valid Org table that should be cycled after
insertion (type "org" or "wrap").  I think the only thing still missing
is interpreting "^[ \t]*|-" as hline.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: babel results handling

[Rick Frankel]

On Wed, Apr 03, 2013 at 08:18:09AM -0600, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:
> 
> > On Sun, Mar 31, 2013 at 07:37:38AM -0600, Eric Schulte wrote:
> > `sh' is probably not the best choice as a "gold standard" due to the
> > fact that it only supports STDOUT ("output" and not "value").
> >
> > Many of the languages are obviously not general purpose, or do not
> > support (like shell), wrapped values (only STDOUT), or don't generate
> > text, so consistency does not matter (e.g., css, sass, sql, sqlite,
> > plantuml, dot).
> >
> > Regardless, the attached org file is a first step an comparing the
> > result processing of some languages (specifically, sh, emacs-lisp,
> > perl and ruby), which, I think, covers a good portion of the babel use
> > of general purpose languages.
> >
> 
> This is a great file.  Thanks for generating it and sharing it.
> Although I think it would be more useful if languages were the smallest
> scale of organization rather than the largest to make cross-language
> comparison easier.

I will try it this way over the weekend. I kept going back-and-forth
with the nesting, never really liked any of the output...

> Would it be difficult to add another set of code blocks which
> automatically compare the output of these automatically generated code
> blocks, indicating when there are differences.

I'll try an add (i think Achim suggested adding it to the test
infrastructure in a later email, will take a look, but don't hold your
breath :).

> > The upshot, is that perl value results match shell value/output
> > results and emacs-lisp, python and ruby all return about the same
> > results (elisp returns the quote characters from a verbatim string).
> >
> 
> What are the perl-shell vs. python-ruby-elisp differences?

Again, Achim has covered it in a later email, but it's that sh (which
is using _output_ and not _value_), and perl (as _value_) are
post-processing the raw results through org-table-convert-region.

> If you want to use pipes to delimit data, then I'd suggest *not*
> interpreting the data as a value, but rather doing something like
> ":results verbatim drawer".  Generally pipes aren't considered to be
> table column delimiters, I'd try tabs or spaces instead.

Agreed, i was just setting up a test to compare results among
different processors.

> > Also, looking at the manual
> > (http://orgmode.org/manual/results.html#results) vs. the code, are
> > there are actually four classes of :results arguments with type broken
> > into type and format?
> >
> >      - Type :: (file list vector table scalar verbatim)
> >      - Format :: (raw html latex org code pp drawer)
> >
> 
> Yes, this does seem to be more clear.  If you're willing to supply a
> documentation patch I'd be very happy to apply it.

I will make the change this weekend -- I have commit privileges but
wanted to make sure my interpretation was correct.

rick

Re: [PATCH] Process hlines in imported tables

[Achim Gratz]

Rick Frankel writes:
> Missed verbatim. Thanks for the pointer, it works, but i think that
> perl is double-processing returned values. If we do the same things in
> elisp i get (my) expected results:
>
> #+begin_src elisp :results raw
>   "|c1|c2|
>   |-
>   |a|1|
>   |b|2|";
> #+end_src

Elisp is different from all other languages: it doesn't do any
processing of strings to begin with for value returns.  The reason that
Perl processes "raw" results is that org-babel-result-cond does not
switch to the "scalar" path for this condition, which is why you need
the extra "verbatim".  It probably should, though, so if Eric agrees
then I will push a change that does this.



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Hi Achim,

Achim Gratz wrote:
> Rick Frankel writes:
>> Missed verbatim. Thanks for the pointer, it works, but i think that
>> perl is double-processing returned values. If we do the same things in
>> elisp i get (my) expected results:
>>
>> #+begin_src elisp :results raw
>>   "|c1|c2|
>>   |-
>>   |a|1|
>>   |b|2|";
>> #+end_src
>
> Elisp is different from all other languages: it doesn't do any
> processing of strings to begin with for value returns.  The reason that
> Perl processes "raw" results is that org-babel-result-cond does not
> switch to the "scalar" path for this condition, which is why you need
> the extra "verbatim".  It probably should, though, so if Eric agrees
> then I will push a change that does this.

IIUC, wouldn't that be changing the default answer to "how to interpret the
results" just for Perl?  While the default answer for all languages seems to
be "table"?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:

> Hi Achim,
>
> Achim Gratz wrote:
>> Rick Frankel writes:
>>> Missed verbatim. Thanks for the pointer, it works, but i think that
>>> perl is double-processing returned values. If we do the same things in
>>> elisp i get (my) expected results:
>>>
>>> #+begin_src elisp :results raw
>>>   "|c1|c2|
>>>   |-
>>>   |a|1|
>>>   |b|2|";
>>> #+end_src
>>
>> Elisp is different from all other languages: it doesn't do any
>> processing of strings to begin with for value returns.  The reason that
>> Perl processes "raw" results is that org-babel-result-cond does not
>> switch to the "scalar" path for this condition, which is why you need
>> the extra "verbatim".  It probably should, though, so if Eric agrees
>> then I will push a change that does this.
>
> IIUC, wouldn't that be changing the default answer to "how to interpret the
> results" just for Perl?  While the default answer for all languages seems to
> be "table"?
>

I would agree that this (meaning raw implies scalar) should either occur
for all languages or for none.  If we do have such header argument
implications, then we'd want to put them into the weakest portion of the
default header argument hierarchy.  Currently this hierarchy looks
something like

1. default header arguments shipped with Org-mode
2. user-set default header arguments
3. default languages-specific header arguments shipped with Org-mode
4. user-set default language-specific header arguments
5. buffer or file level header arguments
6. subtree header arguments
7. code block header arguments

I think this raw implies verbatim action should probably take place
somewhere between 3 and 4, but there could be arguments for other
positions.  Also, without looking at the code, I'm not sure how
difficult adding such implications would be.

Are there other header argument implication rules which would make code
blocks "do what I mean" more naturally in more situations?

Cheers,

>
> Best regards,
>   Seb

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Eric,

Eric Schulte wrote:
> I would agree that this (meaning raw implies scalar) should either occur
> for all languages or for none.

I think this is something interesting, but I wonder now if we wouldn't loose
more than we would win. I mean: how would one be able to output a real "raw"
result, then, that is one where pipes are not interpreted as table field
separator which have to be aligned in some specific way.

Do we need another argument for that?

I mean: at the end, raw should really be raw (no interpretation). If we want
some cycling for table alignment purpose (BTW, do you have lots of such code
blocks?), maybe it'd be better to introduce a `cycle' argument or so?

> If we do have such header argument implications, then we'd want to put them
> into the weakest portion of the default header argument hierarchy. Currently
> this hierarchy looks something like
>
> 1. default header arguments shipped with Org-mode
> 2. user-set default header arguments
> 3. default languages-specific header arguments shipped with Org-mode
> 4. user-set default language-specific header arguments
> 5. buffer or file level header arguments
> 6. subtree header arguments
> 7. code block header arguments
>
> I think this raw implies verbatim action should probably take place
> somewhere between 3 and 4, but there could be arguments for other
> positions.  Also, without looking at the code, I'm not sure how
> difficult adding such implications would be.

Maybe I don't understand the problem correctly, but I'd think this "raw
implies verbatim" would have to take place after _each_ above step.

If between 3 and 4, then a raw specified on the block level (step 7) wouldn't
imply verbatim?  Does that make sense?

I think every raw (be it default, language, buffer, subtree or block-local)
would have to imply the same reasoning.

> Are there other header argument implication rules which would make code
> blocks "do what I mean" more naturally in more situations?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:

> Eric,
>
> Eric Schulte wrote:
>> I would agree that this (meaning raw implies scalar) should either occur
>> for all languages or for none.
>
> I think this is something interesting, but I wonder now if we wouldn't loose
> more than we would win. I mean: how would one be able to output a real "raw"
> result, then, that is one where pipes are not interpreted as table field
> separator which have to be aligned in some specific way.
>
> Do we need another argument for that?
>
> I mean: at the end, raw should really be raw (no interpretation). If we want
> some cycling for table alignment purpose (BTW, do you have lots of such code
> blocks?), maybe it'd be better to introduce a `cycle' argument or so?
>
>> If we do have such header argument implications, then we'd want to put them
>> into the weakest portion of the default header argument hierarchy. Currently
>> this hierarchy looks something like
>>
>> 1. default header arguments shipped with Org-mode
>> 2. user-set default header arguments
>> 3. default languages-specific header arguments shipped with Org-mode
>> 4. user-set default language-specific header arguments
>> 5. buffer or file level header arguments
>> 6. subtree header arguments
>> 7. code block header arguments
>>
>> I think this raw implies verbatim action should probably take place
>> somewhere between 3 and 4, but there could be arguments for other
>> positions.  Also, without looking at the code, I'm not sure how
>> difficult adding such implications would be.
>
> Maybe I don't understand the problem correctly, but I'd think this "raw
> implies verbatim" would have to take place after _each_ above step.
>

Actually I think I was confused when I wrote the above, so please
disregard, sorry for the noise.

Best,

>
> If between 3 and 4, then a raw specified on the block level (step 7)
> wouldn't imply verbatim?  Does that make sense?
>
> I think every raw (be it default, language, buffer, subtree or block-local)
> would have to imply the same reasoning.
>
>> Are there other header argument implication rules which would make code
>> blocks "do what I mean" more naturally in more situations?
>
> Best regards,
>   Seb

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Hi Eric,

"Sebastien Vauban" wrote:
> Eric Schulte wrote:
>> I would agree that this (meaning raw implies scalar) should either occur
>> for all languages or for none.
>
> I think this is something interesting, but I wonder now if we wouldn't loose
> more than we would win. I mean: how would one be able to output a real "raw"
> result, then, that is one where pipes are not interpreted as table field
> separator which have to be aligned in some specific way.
>
> Do we need another argument for that?
>
> I mean: at the end, raw should really be raw (no interpretation). If we want
> some cycling for table alignment purpose (BTW, do you have lots of such code
> blocks?), maybe it'd be better to introduce a `cycle' argument or so?

I think that this portion of my post has been ignored in your answers -- which
I still have to carefully look at.

Though, I don't think the above question should stay unanswered: if you now
"cycle" on all "raw" results, how do we insert real "raw" results for which we
don't want any interpretation (not even cycling tables, or what you be
confounded as tables)?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:

> Hi Eric,
>
> "Sebastien Vauban" wrote:
>> Eric Schulte wrote:
>>> I would agree that this (meaning raw implies scalar) should either occur
>>> for all languages or for none.
>>
>> I think this is something interesting, but I wonder now if we wouldn't loose
>> more than we would win. I mean: how would one be able to output a real "raw"
>> result, then, that is one where pipes are not interpreted as table field
>> separator which have to be aligned in some specific way.
>>
>> Do we need another argument for that?
>>
>> I mean: at the end, raw should really be raw (no interpretation). If we want
>> some cycling for table alignment purpose (BTW, do you have lots of such code
>> blocks?), maybe it'd be better to introduce a `cycle' argument or so?
>
> I think that this portion of my post has been ignored in your answers -- which
> I still have to carefully look at.
>
> Though, I don't think the above question should stay unanswered: if you now
> "cycle" on all "raw" results, how do we insert real "raw" results for which we
> don't want any interpretation (not even cycling tables, or what you be
> confounded as tables)?
>

Is this a hypothetical problem or do you have a use case which requires
non-cycling?

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Eric,

Eric Schulte wrote:
> "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> "Sebastien Vauban" wrote:
>>> Eric Schulte wrote:
>>>> I would agree that this (meaning raw implies scalar) should either occur
>>>> for all languages or for none.
>>>
>>> I think this is something interesting, but I wonder now if we wouldn't loose
>>> more than we would win. I mean: how would one be able to output a real "raw"
>>> result, then, that is one where pipes are not interpreted as table field
>>> separator which have to be aligned in some specific way.
>>>
>>> Do we need another argument for that?
>>>
>>> I mean: at the end, raw should really be raw (no interpretation). If we want
>>> some cycling for table alignment purpose (BTW, do you have lots of such code
>>> blocks?), maybe it'd be better to introduce a `cycle' argument or so?
>>
>> I think that this portion of my post has been ignored in your answers -- which
>> I still have to carefully look at.
>>
>> Though, I don't think the above question should stay unanswered: if you now
>> "cycle" on all "raw" results, how do we insert real "raw" results for which we
>> don't want any interpretation (not even cycling tables, or what you be
>> confounded as tables)?
>
> Is this a hypothetical problem or do you have a use case which requires
> non-cycling?

At this stage, completely hypothetical. It's just that we remove some
possibility which existed: from now on, we can't insert some types of data
anymore. And I feel that the name "raw" does not conform anymore to the
reality.

But, as said, I don't have a real problem at hand.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [PATCH] Process hlines in imported tables

[Rick Frankel]

On Thu, Apr 04, 2013 at 03:59:36PM +0200, Sebastien Vauban wrote:
> Hi Achim,
> 
> Achim Gratz wrote:
> > Rick Frankel writes:
> >> Missed verbatim. Thanks for the pointer, it works, but i think that
> >> perl is double-processing returned values. If we do the same things in
> >> elisp i get (my) expected results:
> >>
> >> #+begin_src elisp :results raw
> >>   "|c1|c2|
> >>   |-
> >>   |a|1|
> >>   |b|2|";
> >> #+end_src
> >
> > Elisp is different from all other languages: it doesn't do any
> > processing of strings to begin with for value returns.  The reason that
> > Perl processes "raw" results is that org-babel-result-cond does not
> > switch to the "scalar" path for this condition, which is why you need
> > the extra "verbatim".  It probably should, though, so if Eric agrees
> > then I will push a change that does this.
> 
> IIUC, wouldn't that be changing the default answer to "how to interpret the
> results" just for Perl?  While the default answer for all languages seems to
> be "table"?

It's not. only shell (which doesn't have _value_ results), and sql,
force the results to be interpreted as a table. elisp, ruby and python
seem to treat raw results as scalars.

rick

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Hi Rick,

Rick Frankel wrote:
>> IIUC, wouldn't that be changing the default answer to "how to interpret the
>> results" just for Perl?  While the default answer for all languages seems to
>> be "table"?
>
> It's not. only shell (which doesn't have _value_ results),

Is this statement really true, btw?  Wouldn't the value result of a sh code
block be the return code?

> and sql, force the results to be interpreted as a table.

Because table is the default option...

> elisp, ruby and python seem to treat raw results as scalars.

For ELisp, Eric told that it was an exception.

For Ruby and Python, I dunno, as I don't know the languages and have no
environment ready to test them (under Windows).

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [PATCH] Process hlines in imported tables

[Achim Gratz]

Sebastien Vauban writes:
> Achim Gratz wrote:
>> Elisp is different from all other languages: it doesn't do any
>> processing of strings to begin with for value returns.  The reason that
>> Perl processes "raw" results is that org-babel-result-cond does not
>> switch to the "scalar" path for this condition, which is why you need
>> the extra "verbatim".  It probably should, though, so if Eric agrees
>> then I will push a change that does this.
>
> IIUC, wouldn't that be changing the default answer to "how to
> interpret the results" just for Perl?  While the default answer for
> all languages seems to be "table"?

Again, org-babel-result-cond doesn't interpret "raw" at all at the
moment, which I think is a bug.  This leads to the interpretation of
multiline strings and strings with separators in the return value as
tables.  Not all Babel languages are using that macro, so these
implementations might have a different interpretation (which may or may
not be a bug in itself).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]

Achim Gratz <Stromeko@nexgo.de> writes:

> Sebastien Vauban writes:
>> Achim Gratz wrote:
>>> Elisp is different from all other languages: it doesn't do any
>>> processing of strings to begin with for value returns.  The reason that
>>> Perl processes "raw" results is that org-babel-result-cond does not
>>> switch to the "scalar" path for this condition, which is why you need
>>> the extra "verbatim".  It probably should, though, so if Eric agrees
>>> then I will push a change that does this.
>>
>> IIUC, wouldn't that be changing the default answer to "how to
>> interpret the results" just for Perl?  While the default answer for
>> all languages seems to be "table"?
>
> Again, org-babel-result-cond doesn't interpret "raw" at all at the
> moment, which I think is a bug.  This leads to the interpretation of
> multiline strings and strings with separators in the return value as
> tables.

The attached patch updates the org-babel-result-cond macro so that
"raw", "org" and "drawer" all imply verbatim results *unless* the
":results table" header argument is explicitly specified.  I think this
is an improvement and should (hopefully) be merged before the 8.0
release.  Could everyone on this thread test this patch and let me know
what you think before I apply it?  I'm nervous about such a patch at the
last minute.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-raw-org-and-drawer-imply-verbatim-results.patch --]
[-- Type: text/x-patch, Size: 2360 bytes --]

From c12003708f691862086aac30c675868cd69d6bb3 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Sat, 6 Apr 2013 10:18:35 -0600
Subject: [PATCH] raw org and drawer imply verbatim results

  This unifies the results handling across a number of different
  languages.  It is still possible to force tabular output by adding the
  ":results table" argument.

The following example demonstrates the results in shell python and perl.

** drawer and table
#+begin_src sh :results drawer table
 echo -e "1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
| 1 |
| 2 |
| 3 |
:END:

#+begin_src perl :results drawer table
"1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
| 1 |
| 2 |
| 3 |
:END:

#+begin_src python :results drawer table
return "1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
| 1\n2\n3 |
:END:

** drawer
#+begin_src sh :results drawer
 echo -e "1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
1
2
3
:END:

#+begin_src perl :results drawer
"1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
1
2
3
:END:

#+begin_src python :results drawer
return "1\n2\n3"
#+end_src

#+RESULTS:
:RESULTS:
1
2
3
:END:

** raw
#+begin_src sh :results raw
 echo -e "1\n2\n3"
#+end_src

#+RESULTS:
1
2
3

#+begin_src perl :results raw
"1\n2\n3"
#+end_src

#+RESULTS:
1
2
3

#+begin_src python :results raw
return "1\n2\n3"
#+end_src

#+RESULTS:
1
2
3

* lisp/ob-core.el (org-babel-result-cond): The "raw", "org" and "drawer"
  :results header argument values preclude table processing unless the
  "table" argument is given as well.
---
 lisp/ob-core.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bba2dfe..b8f863f 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2637,10 +2637,14 @@ Emacs shutdown."))
 	     (member "html" ,result-params)
 	     (member "code" ,result-params)
 	     (member "pp" ,result-params)
-	     (and (member "output" ,result-params)
+	     (and (or (member "output" ,result-params)
+		      (member "raw"    ,result-params)
+		      (member "org"    ,result-params)
+		      (member "drawer" ,result-params))
 		  (not (member "table" ,result-params))))
 	 ,scalar-form
        ,@table-forms)))
+(def-edebug-spec org-babel-result-cond (form form body))
 
 (defun org-babel-temp-file (prefix &optional suffix)
   "Create a temporary file in the `org-babel-temporary-directory'.
-- 
1.8.2


[-- Attachment #3: Type: text/plain, Size: 1465 bytes --]


> Not all Babel languages are using that macro, so these implementations
> might have a different interpretation (which may or may not be a bug
> in itself).
>

Every general purpose language should be using this macro.  The
following lists those ob*.el files which don't use this macro.

    $ grep -rc org-babel-result-cond lisp/ob*el|grep 0|sed 's/:.*$//'
    lisp/ob-asymptote.el
    lisp/ob-calc.el
    lisp/ob-comint.el
    lisp/ob-css.el
    lisp/ob-ditaa.el
    lisp/ob-dot.el
    lisp/ob.el
    lisp/ob-eval.el
    lisp/ob-exp.el
    lisp/ob-gnuplot.el
    lisp/ob-haskell.el
    lisp/ob-js.el
    lisp/ob-keys.el
    lisp/ob-latex.el
    lisp/ob-ledger.el
    lisp/ob-lilypond.el
    lisp/ob-lob.el
    lisp/ob-makefile.el
    lisp/ob-matlab.el
    lisp/ob-mscgen.el
    lisp/ob-ocaml.el
    lisp/ob-octave.el
    lisp/ob-org.el
    lisp/ob-plantuml.el
    lisp/ob-ref.el
    lisp/ob-ruby.el
    lisp/ob-sass.el
    lisp/ob-scheme.el
    lisp/ob-screen.el
    lisp/ob-table.el
    lisp/ob-tangle.el

Of these I would guess that the following 7 should be updated to use the
org-babel-result-cond macro.

  scheme ruby ocaml matlab js haskell asymptote

I don't know if we want to try to make these changes before the 8.0
release.  I personally could update and test js, scheme, ocaml and
haskell this weekend.  I do not have ruby, matlab or asymptote installed
on my laptop.

Thanks,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

>
> Of these I would guess that the following 7 should be updated to use the
> org-babel-result-cond macro.
>
>   scheme ruby ocaml matlab js haskell asymptote
>
> I don't know if we want to try to make these changes before the 8.0
> release.  I personally could update and test js, scheme, ocaml and
> haskell this weekend.

The attached tarball patches these four languages to use the
org-babel-result-cond macro.  I've tested them all and they seem to
work.  Any reason not to apply these changes?


[-- Attachment #2: patches.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 3163 bytes --]

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]


> I do not have ruby, matlab or asymptote installed on my laptop.
>
> Thanks,
>
>>
>>
>> Regards,
>> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Bastien]

Hi Eric,

Eric Schulte <schulte.eric@gmail.com> writes:

> I'm nervous about such a patch at the last minute.

Please go ahead and apply it, that's the best way to find problems.

-- 
 Bastien

Re: [PATCH] Process hlines in imported tables

[Eric Schulte]

Bastien <bzg@altern.org> writes:

> Hi Eric,
>
> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> I'm nervous about such a patch at the last minute.
>
> Please go ahead and apply it, that's the best way to find problems.

Done.  Thanks,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] Process hlines in imported tables

[Rick Frankel]

On Wed, Apr 03, 2013 at 08:21:01PM +0200, Achim Gratz wrote:
> Rick Frankel writes:
> > Missed verbatim. Thanks for the pointer, it works, but i think that
> > perl is double-processing returned values. If we do the same things in
> > elisp i get (my) expected results:
> >
> > #+begin_src elisp :results raw
> >   "|c1|c2|
> >   |-
> >   |a|1|
> >   |b|2|";
> > #+end_src
> 
> Elisp is different from all other languages: it doesn't do any
> processing of strings to begin with for value returns.  The reason that
> Perl processes "raw" results is that org-babel-result-cond does not
> switch to the "scalar" path for this condition, which is why you need
> the extra "verbatim".  It probably should, though, so if Eric agrees
> then I will push a change that does this.

I agree. "raw" results should probably be treated as scalar (with
cycling of the output to reformat an table :).

BTW, I am having problems wrapping my head around verbatim.
From testing, it seems that verbatim acts the same as scalar.

rick

Re: [PATCH] Process hlines in imported tables

[Sebastien Vauban]

Rick, Achim, Eric,

Rick Frankel wrote:
> I agree. "raw" results should probably be treated as scalar (with
> cycling of the output to reformat an table :).

I think (!) I fully agree with that as well. Just that I wouldn't like more
exceptions in the supported languages. So, if it's kindof the new default,
great.

> BTW, I am having problems wrapping my head around verbatim.
> From testing, it seems that verbatim acts the same as scalar.

100% true; they're just synonyms [1].

FYI, so are `table' and `vector'...

Best regards,
  Seb

[1] I hate synonyms. I'd be in favor of just keeping one syntax...

-- 
Sebastien Vauban