babel results handling (was: Process hlines in imported tables)

[Rick Frankel <rick@rickster.com>]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On Sun, Mar 31, 2013 at 07:37:38AM -0600, Eric Schulte wrote:
> It is certainly true that Emacs Lisp is treated differently than all
> other languages.  There are also significant differences between
> languages, e.g., session evaluation doesn't make sense for some
> languages, and for other languages session evaluation is the only type
> of evaluation that does make sense.
> 
> In addition to that, with over 40 supported languages [1], There
> undoubtedly are cases where we are doing a bad job of ensuring
> inter-languages consistency.  If you can find concise examples which use
> demonstrate significant inconsistencies between languages, then we
> should be able to resolve those on a case by case basis.  In general I
> think ob-sh is probably the most widely used code block language, and
> can be treated as the gold standard against which other languages should
> be compared.

`sh' is probably not the best choice as a "gold standard" due to the
fact that it only supports STDOUT ("output" and not "value").

Many of the languages are obviously not general purpose, or do not
support (like shell), wrapped values (only STDOUT), or don't generate
text, so consistency does not matter (e.g., css, sass, sql, sqlite,
plantuml, dot).

Regardless, the attached org file is a first step an comparing the
result processing of some languages (specifically, sh, emacs-lisp,
perl and ruby), which, I think, covers a good portion of the babel use
of general purpose languages.

The upshot, is that perl value results match shell value/output
results and emacs-lisp, python and ruby all return about the same
results (elisp returns the quote characters from a verbatim string).

I still think that the scalar pipe-delimited processing from shell and
perl is wrong in that pipe-delimited data ends up w/ an extra column
if each line starts w/ a pipe, but not if the pipe is used like a
csv separator (between columns but not at the beginning or end of the
line).

Also, looking at the manual
(http://orgmode.org/manual/results.html#results) vs. the code, are
there are actually four classes of :results arguments with type broken
into type and format?

     - Type :: (file list vector table scalar verbatim)
     - Format :: (raw html latex org code pp drawer)

rick

[-- Attachment #2: results.org --]
[-- Type: text/plain, Size: 3142 bytes --]

* Value results procesing

The list of valid =:result= arguments
#+BEGIN_SRC elisp :results list
(cdr (assoc 'results org-babel-common-header-args-w-values))
#+END_SRC
#+results:
- (file list vector table scalar verbatim)
- (raw html latex org code pp drawer)
- (replace silent none append prepend)
- (output value)

You can add languages to this list to test them as well.
The genrator assumes that a double quoted string, with the escapes
=\n= and =\t= are valid in the language and that no statement
terminator is required. The "method" is the statement required for
the value to be returned to the wrapper code. (E.g., none for =perl=
and =ruby=, =sh= does not actually return values so needs an =echo=.)

#+NAME: languages
| language | method | array                      |
|----------+--------+----------------------------|
| sh       | echo   |                            |
| perl     |        | [[qw[h1 h2]],[qw[r1 r2]]]  |
| ruby     |        | [%w[h1 h2],%w[r1 r2]]      |
| elisp    |        | '((h1 h2) (r1 r2))         |
| python   | return | [["h1", "h2"],["r1","r2"]] |

** Generate test code                                             :noexport:
#+NAME: block
#+BEGIN_SRC elisp :eval never
  (defun block (lang body function handler type)
    (format
     "**** %s body, :results value %s %s
  ,#+BEGIN_SRC %s :results value %s %s :wrap example
    %s %s
  ,#+END_SRC\n"
     (cond 
      ((not (not (string-match-p "^\"|" body))) "pipe delimited")
      ((not (not (string-match-p "^\"" body))) "tab delimited")
      (t "array"))
     handler type
     lang handler type function body))
#+END_SRC

Run this block to generate the test code. You can then execute the
test code by running =org-babel-execute-subtree= on [[*All%20Tests][All Tests]]. The
file exports resonably nicely to either HTML or LaTeX.

#+HEADER: :var languages=languages
#+HEADER: :var types='("" "table" "scalar" "verbatim")
#+HEADER: :var formatting='("" "raw")
#+HEADER: :var scalar="\"|h1|h2|\\n|-\\n|r1|r2|\""
#+HEADER: :var tab-delim="\"h1\\th2\\t\\nr1\\tr2\""
#+BEGIN_SRC elisp :results raw :noweb yes   
  (require 'ob-perl)
  (require 'ob-sh)
  (require 'ob-python)
  <<block>>
  (save-excursion
    (condition-case nil
        (progn
          (goto-char (org-find-entry-with-id "ALL-TESTS"))
          (org-cut-subtree))
      (error t)))
  (concat
   "* All Tests\n"
   ":PROPERTIES:\n"
   ":ID:  ALL-TESTS\n"
   ":END:\n"
   (mapconcat
    (lambda (lang)
      (format 
       "** %s\n%s" (car lang)
       (mapconcat
        (lambda (formatter)
          (format "*** Result format: %s\n%s" 
                  (if (string= formatter "") "default" formatter)
                  (mapconcat 
                   (lambda (type)
                     (let ((l (car lang)) (pfx (nth 1 lang)) (array (nth 2 lang)))
                       (concat
                        (block l scalar pfx formatter type)
                        (block l tab-delim pfx formatter type)
                        (unless (string= "" array)
                          (block l array pfx formatter type))))) types "\n")))
        formatting ""))) languages ""))
#+END_SRC