[babel] Babel as a test harness?

[babel] Babel as a test harness?

[Nathan Neff]

[-- Attachment #1.1: Type: text/plain, Size: 830 bytes --]

I think it would be cool to use Org-babel as a simple test harness.

I'd like to have an org file, with various sections that demonstrate how to
do
something in Groovy.  I'd like to be able to run all the code in the org
file and make sure
they all run successfully (return code 0).

In Groovy, the "assert" function will exit with a non-zero code if it
fails.  How
would I use org-babel to generate a "summary" table with the name of
each patch of code and whether or not it succeeded?

Something like this would be cool:

* Regular Expressions
** Here's how to match "foo"

#+source simple_regex
#+begin_src groovy
assert "foo" =~ /foo/
#+end_src

** Here's how to match "bar" or "baz"
#+source regex_or
#+begin_src groovy
assert "bar" =~ "ba(z|r)"
#+end_src

* Results:
| simple_regex | success |
| regex_or        | success |

[-- Attachment #1.2: Type: text/html, Size: 1027 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Babel as a test harness?

[Dan Davison]

Nathan Neff <nathan.neff@gmail.com> writes:

> I think it would be cool to use Org-babel as a simple test harness.
>
> I'd like to have an org file, with various sections that demonstrate how to do
> something in Groovy.  I'd like to be able to run all the code in the org file
> and make sure
> they all run successfully (return code 0).
>
> In Groovy, the "assert" function will exit with a non-zero code if it fails. 
> How
> would I use org-babel to generate a "summary" table with the name of
> each patch of code and whether or not it succeeded?

Like below? This is all Eric's doing. For quite a while now org-babel
has used a table like this to validate itself. It took me a while to
understand it, but basically Eric designed a special function (actually,
a macro) called sbe (source block evaluate) to be used in table formulas
to call org-babel source blocks. Use C-u C-c C-c to update the table.

* Tests
| functionality  | block        | arg | expected | results              | pass                                       |
|----------------+--------------+-----+----------+----------------------+--------------------------------------------|
| simple regexp  | simple_regex |     |          |                      | pass                                       |
| regexp with or | regex_or     |     |          | A pretend problem... | expected "" but was "A pretend problem..." |
#+TBLFM: $5='(if (= (length $3) 1) (sbe $2 (n $3)) (sbe $2)) :: $6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))

* Code blocks for tests
** Here's how to match "foo"
#+source: simple_regex
#+begin_src groovy
assert "foo" =~ /foo/
#+end_src

** Here's how to match "bar" or "baz"
#+source: regex_or
#+begin_src groovy
assert "bar" =~ "ba(z|r)"
println "A pretend problem"
#+end_src

You can put any arguments to your source blocks in the arg column. See

http://orgmode.org/worg/org-contrib/babel/intro.php#spreadsheet

(It often makes most sense to clone Worg and view the org files
themseleves)

And for a more complex example, including how to pass arguments to
source blocks to sbe, see our full test suite in the file
development.org in the devel repo

http://github.com/eschulte/babel-dev/

Dan

p.s. groovy seems to be funny about hyphens in the input file name. I
think you want to apply this:

diff --git a/contrib/babel/lisp/langs/org-babel-groovy.el b/contrib/babel/lisp/langs/org-babel-groovy.el
index 02b3272..d6a28a7 100644
--- a/contrib/babel/lisp/langs/org-babel-groovy.el
+++ b/contrib/babel/lisp/langs/org-babel-groovy.el
@@ -88,7 +88,7 @@ print o join(\"\\n\", @r), \"\\n\"")
 (defun org-babel-groovy-evaluate (session body &optional result-type)
   "Evaluate Groovy code in BODY"
  ;; external process evaluation
- (let ((infile (make-temp-file "org_babel_groovy_input_")))
+  (let ((infile (replace-regexp-in-string "-" "_" (make-temp-file "org_babel_groovy_input_"))))
    (save-excursion
      (with-temp-buffer
        (with-temp-file infile (insert body))



>
> Something like this would be cool:
>
> * Regular Expressions
> ** Here's how to match "foo"
>
> #+source simple_regex
> #+begin_src groovy
> assert "foo" =~ /foo/
> #+end_src
>
> ** Here's how to match "bar" or "baz"
> #+source regex_or
> #+begin_src groovy
> assert "bar" =~ "ba(z|r)"
> #+end_src
>
> * Results:
> | simple_regex | success |
> | regex_or        | success |
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Babel as a test harness?

[Nathan Neff]

[-- Attachment #1.1: Type: text/plain, Size: 4294 bytes --]

On Fri, Mar 19, 2010 at 7:45 PM, Dan Davison <davison@stats.ox.ac.uk> wrote:

> Nathan Neff <nathan.neff@gmail.com> writes:
>
> > I think it would be cool to use Org-babel as a simple test harness.
> >
> > I'd like to have an org file, with various sections that demonstrate how
> to do
> > something in Groovy.  I'd like to be able to run all the code in the org
> file
> > and make sure
> > they all run successfully (return code 0).
> >
> > In Groovy, the "assert" function will exit with a non-zero code if it
> fails.
> > How
> > would I use org-babel to generate a "summary" table with the name of
> > each patch of code and whether or not it succeeded?
>
> Like below? This is all Eric's doing. For quite a while now org-babel
> has used a table like this to validate itself. It took me a while to
> understand it, but basically Eric designed a special function (actually,
> a macro) called sbe (source block evaluate) to be used in table formulas
> to call org-babel source blocks. Use C-u C-c C-c to update the table.
>
> * Tests
> | functionality  | block        | arg | expected | results              |
> pass                                       |
>
> |----------------+--------------+-----+----------+----------------------+--------------------------------------------|
> | simple regexp  | simple_regex |     |          |                      |
> pass                                       |
> | regexp with or | regex_or     |     |          | A pretend problem... |
> expected "" but was "A pretend problem..." |
> #+TBLFM: $5='(if (= (length $3) 1) (sbe $2 (n $3)) (sbe $2)) :: $6='(if
> (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
>
> * Code blocks for tests
> ** Here's how to match "foo"
> #+source: simple_regex
> #+begin_src groovy
> assert "foo" =~ /foo/
> #+end_src
>
> ** Here's how to match "bar" or "baz"
> #+source: regex_or
> #+begin_src groovy
> assert "bar" =~ "ba(z|r)"
> println "A pretend problem"
> #+end_src
>
>
This is awesome.  I tried it and it works.

The table's "results" column says exactly what went wrong.
For example, if I have the following:

** Here's how to match "bar" or "baz"
#+source: regex_or
#+begin_src groovy
assert "bar" =~ "this ain't gonna match"
#+end_src

Then, my results table looks like this:
| results |
| Caught: java.lang.AssertionError: Expression: (bar =~ this ain't gonna |

really cool!

--Nate


> You can put any arguments to your source blocks in the arg column. See
>
> http://orgmode.org/worg/org-contrib/babel/intro.php#spreadsheet
>
> (It often makes most sense to clone Worg and view the org files
> themseleves)
>
> And for a more complex example, including how to pass arguments to
> source blocks to sbe, see our full test suite in the file
> development.org in the devel repo
>
> http://github.com/eschulte/babel-dev/
>
> Dan
>
> p.s. groovy seems to be funny about hyphens in the input file name. I
> think you want to apply this:
>
> diff --git a/contrib/babel/lisp/langs/org-babel-groovy.el
> b/contrib/babel/lisp/langs/org-babel-groovy.el
> index 02b3272..d6a28a7 100644
> --- a/contrib/babel/lisp/langs/org-babel-groovy.el
> +++ b/contrib/babel/lisp/langs/org-babel-groovy.el
> @@ -88,7 +88,7 @@ print o join(\"\\n\", @r), \"\\n\"")
>  (defun org-babel-groovy-evaluate (session body &optional result-type)
>   "Evaluate Groovy code in BODY"
>  ;; external process evaluation
> - (let ((infile (make-temp-file "org_babel_groovy_input_")))
> +  (let ((infile (replace-regexp-in-string "-" "_" (make-temp-file
> "org_babel_groovy_input_"))))
>    (save-excursion
>      (with-temp-buffer
>        (with-temp-file infile (insert body))
>
>
>
> >
> > Something like this would be cool:
> >
> > * Regular Expressions
> > ** Here's how to match "foo"
> >
> > #+source simple_regex
> > #+begin_src groovy
> > assert "foo" =~ /foo/
> > #+end_src
> >
> > ** Here's how to match "bar" or "baz"
> > #+source regex_or
> > #+begin_src groovy
> > assert "bar" =~ "ba(z|r)"
> > #+end_src
> >
> > * Results:
> > | simple_regex | success |
> > | regex_or        | success |
> >
> > _______________________________________________
> > Emacs-orgmode mailing list
> > Please use `Reply All' to send replies to the list.
> > Emacs-orgmode@gnu.org
> > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>

[-- Attachment #1.2: Type: text/html, Size: 6029 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] Babel as a test harness?

[Nathan Neff]

[-- Attachment #1.1: Type: text/plain, Size: 3545 bytes --]

On Fri, Mar 19, 2010 at 7:45 PM, Dan Davison <davison@stats.ox.ac.uk> wrote:

> Nathan Neff <nathan.neff@gmail.com> writes:
>
> > I think it would be cool to use Org-babel as a simple test harness.
> >
> > I'd like to have an org file, with various sections that demonstrate how
> to do
> > something in Groovy.  I'd like to be able to run all the code in the org
> file
> > and make sure
> > they all run successfully (return code 0).
> >
> > In Groovy, the "assert" function will exit with a non-zero code if it
> fails.
> > How
> > would I use org-babel to generate a "summary" table with the name of
> > each patch of code and whether or not it succeeded?
>
> Like below? This is all Eric's doing. For quite a while now org-babel
> has used a table like this to validate itself. It took me a while to
> understand it, but basically Eric designed a special function (actually,
> a macro) called sbe (source block evaluate) to be used in table formulas
> to call org-babel source blocks. Use C-u C-c C-c to update the table.
>
> * Tests
> | functionality  | block        | arg | expected | results              |
> pass                                       |
>
> |----------------+--------------+-----+----------+----------------------+--------------------------------------------|
> | simple regexp  | simple_regex |     |          |                      |
> pass                                       |
> | regexp with or | regex_or     |     |          | A pretend problem... |
> expected "" but was "A pretend problem..." |
> #+TBLFM: $5='(if (= (length $3) 1) (sbe $2 (n $3)) (sbe $2)) :: $6='(if
> (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
>
> * Code blocks for tests
> ** Here's how to match "foo"
> #+source: simple_regex
> #+begin_src groovy
> assert "foo" =~ /foo/
> #+end_src
>
> ** Here's how to match "bar" or "baz"
> #+source: regex_or
> #+begin_src groovy
> assert "bar" =~ "ba(z|r)"
> println "A pretend problem"
> #+end_src
>
> You can put any arguments to your source blocks in the arg column. See
>
> http://orgmode.org/worg/org-contrib/babel/intro.php#spreadsheet
>
> (It often makes most sense to clone Worg and view the org files
> themseleves)
>
> And for a more complex example, including how to pass arguments to
> source blocks to sbe, see our full test suite in the file
> development.org in the devel repo
>
> http://github.com/eschulte/babel-dev/
>
> Dan
>
>

This is really cool Dan!  I simplified the test table -- it runs the test in
column 1, then
prints the output to the second column, and puts "pass" or "fail" in the 3rd
column
if the 2nd column is blank.

* Tests, Simpler

Output from each source block should be nothing.
| block        |
output                                                        | result |
|--------------+---------------------------------------------------------------+--------|
| simple_regex
|                                                               | pass   |
| regex_or
|                                                               | pass   |
| regex_fail   | Caught: java.lang.AssertionError: Expression: (bar =~
foo)... | fail   |
#+TBLFM: $2='(sbe $1) :: $3='(if (string= $2 "") "pass" "fail")


* Code blocks for tests
** Here's how to match "foo"
#+source: simple_regex
#+begin_src groovy results: output
assert "foo" =~ /foo/
#+end_src

** Here's how to match "bar" or "baz"
#+source: regex_or
#+begin_src groovy
assert "bar" =~ "ba(z|r)"
#+end_src

** A failing test
#+source: regex_fail
#+begin_src groovy
assert "bar" =~ "foo"
#+end_src

[-- Attachment #1.2: Type: text/html, Size: 4775 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode