Global variables in Org mode document with source blocks

Global variables in Org mode document with source blocks

[Lennart C. Karssen]

[-- Attachment #1.1: Type: text/plain, Size: 1609 bytes --]

Dear list,

I am working on a dynamic report in Org mode, where I use source blocks
in various languages to process data. Several blocks produce text or
tables that become part of the PDF on export.

The final chapter should state whether all checks passed, or whether one
or more failed (it is not necessary to know which step failed).

In a single language environment, I would use a variable (called e.g.
nrChecksFailed) that would be incremented for each failing check. In a
single language Org document this could probably be done with a
:session, but given that I mix Awk, Bash, Emacs lisp and R that doesn't
look like the way to go. Do Org documents/source blocks have some
concept of a (global) variable that I can pass to my SRC blocks and
increment inside them?

E.g. after somehow initialising nrChecksFailed = 0, I would like to do:

#+header :var nFailed=nrChecksFailed :var someData=someData
#+begin_src R :results raw
do_some_check_here_on_someData

if (check_results_OK) {
  cat("check A passed\n")
} else {
  cat("check A *failed*\n")
  nFailed <- nFailed + 1
}
#+end_src

So that in my conclusion chapter I can do for example:

#+header: :var nFailed=nrChecksFailed
#+begin_src bash  :results raw
if [[ nFailed -eq 0 ]]; then
  echo "All checks passed
else
 echo "One or more checks *failed!*"
fi
#+end_src


Best regards,

Lennart.

-- 
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
L.C. Karssen
The Netherlands

lennart@karssen.org
http://blog.karssen.org
GPG key ID: A88F554A
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Global variables in Org mode document with source blocks

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 4041 bytes --]

Given all the different languages involved, I don't think there is a way to
use a common variable.

One way might be to have each block output some kind of string if it fails,
and then in the last block you could search for the buffer for that string.
Something like this:


* Section 1

#+BEGIN_SRC sh
false || echo "failed"-`date +'%s'`
#+END_SRC

#+RESULTS:
: failed-1621348872


#+BEGIN_SRC python
import time

if not False:
    print(f'failed-{time.time()}')
#+END_SRC

#+RESULTS:
: failed-1621348926.125608


* Final section

#+BEGIN_SRC emacs-lisp
(format "There were %s failed blocks" (count-matches "failed-[0-9]"
(point-min) (point-max)))
#+END_SRC

#+RESULTS:
: There were 2 failed blocks

Some alternatives include writing/appending to a file on error, and then
counting the number of lines.

Another route is to use a :post header and search for the string there.

#+BEGIN_SRC emacs-lisp
(setq n-failures 0)
#+END_SRC

#+RESULTS:
: 0

#+name: fail-capture
#+BEGIN_SRC emacs-lisp :var data=""
(when (string-match "failed-[0-9]" data)
  (incf n-failures)
  (message "captured a failure in %s. n-failures=%s" data n-failures))
#+END_SRC

#+BEGIN_SRC sh :post fail-capture(*this*)
false || echo "failed"-`date +'%s'`
#+END_SRC

#+RESULTS:
: captured a failure in failed-1621349398. n-failures=1


#+BEGIN_SRC python :post fail-capture(*this*)
print('This did not fail')
#+END_SRC

#+RESULTS:
: nil

#+BEGIN_SRC python :post fail-capture(*this*)
print('This failed-9')
#+END_SRC

#+RESULTS:
: captured a failure in This failed-9
: . n-failures=2

All these approaches need you to handle and catch the errors. I think other
kinds of errors would stop the export process. e.g. if a block errors out
from division by zero or something.

I don't know how easy it would be to check if a src block succeeded or not.
If it was easy, you might use the org-babel-after-execute-hook variable to
update something when a failure is detected. Some blocks create an
*Org-Babel Error Output buffer somewhere, but i don't know if this is 100%
reliable across all blocks.

John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Tue, May 18, 2021 at 9:58 AM Lennart C. Karssen <lennart@karssen.org>
wrote:

> Dear list,
>
> I am working on a dynamic report in Org mode, where I use source blocks
> in various languages to process data. Several blocks produce text or
> tables that become part of the PDF on export.
>
> The final chapter should state whether all checks passed, or whether one
> or more failed (it is not necessary to know which step failed).
>
> In a single language environment, I would use a variable (called e.g.
> nrChecksFailed) that would be incremented for each failing check. In a
> single language Org document this could probably be done with a
> :session, but given that I mix Awk, Bash, Emacs lisp and R that doesn't
> look like the way to go. Do Org documents/source blocks have some
> concept of a (global) variable that I can pass to my SRC blocks and
> increment inside them?
>
> E.g. after somehow initialising nrChecksFailed = 0, I would like to do:
>
> #+header :var nFailed=nrChecksFailed :var someData=someData
> #+begin_src R :results raw
> do_some_check_here_on_someData
>
> if (check_results_OK) {
>   cat("check A passed\n")
> } else {
>   cat("check A *failed*\n")
>   nFailed <- nFailed + 1
> }
> #+end_src
>
> So that in my conclusion chapter I can do for example:
>
> #+header: :var nFailed=nrChecksFailed
> #+begin_src bash  :results raw
> if [[ nFailed -eq 0 ]]; then
>   echo "All checks passed
> else
>  echo "One or more checks *failed!*"
> fi
> #+end_src
>
>
> Best regards,
>
> Lennart.
>
> --
> *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
> L.C. Karssen
> The Netherlands
>
> lennart@karssen.org
> http://blog.karssen.org
> GPG key ID: A88F554A
> -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
>
>

[-- Attachment #2: Type: text/html, Size: 5466 bytes --]

Re: Global variables in Org mode document with source blocks

[Lennart C. Karssen]

[-- Attachment #1.1: Type: text/plain, Size: 5746 bytes --]

Dear John,

Thanks a lot for your quick reply. My apologies for not replying
earlier. Some urgent things came up that took my time.

I tried your suggestions and settled on a small adaptation of your first
suggestion. Because the output of my source blocks is raw Org code (i.e.
the report text that explains the failure), I can't directly print
=failure-DIGITS= or something similar. My solution to that is to append
=# failure-BLOCK= to the output text (which is ignored during export)
and use your =count-matches= suggestion to count those. The text =BLOCK=
is different for each code block and should allow me to specify which
blocks fail in the Conclusion section.


Thanks a lot!

Best regards,

Lennart.

On 18-05-2021 17:03, John Kitchin wrote:
> Given all the different languages involved, I don't think there is a way
> to use a common variable. 
> 
> One way might be to have each block output some kind of string if it
> fails, and then in the last block you could search for the buffer for
> that string. Something like this:
> 
> 
> * Section 1
> 
> #+BEGIN_SRC sh
> false || echo "failed"-`date +'%s'`
> #+END_SRC
> 
> #+RESULTS:
> : failed-1621348872
> 
> 
> #+BEGIN_SRC python
> import time
> 
> if not False:
>     print(f'failed-{time.time()}')
> #+END_SRC
> 
> #+RESULTS:
> : failed-1621348926.125608
> 
> 
> * Final section
> 
> #+BEGIN_SRC emacs-lisp
> (format "There were %s failed blocks" (count-matches "failed-[0-9]"
> (point-min) (point-max)))
> #+END_SRC
> 
> #+RESULTS:
> : There were 2 failed blocks
> 
> Some alternatives include writing/appending to a file on error, and then
> counting the number of lines.
> 
> Another route is to use a :post header and search for the string there.
> 
> #+BEGIN_SRC emacs-lisp
> (setq n-failures 0)
> #+END_SRC
> 
> #+RESULTS:
> : 0
> 
> #+name: fail-capture
> #+BEGIN_SRC emacs-lisp :var data=""
> (when (string-match "failed-[0-9]" data)
>   (incf n-failures)
>   (message "captured a failure in %s. n-failures=%s" data n-failures))
> #+END_SRC
> 
> #+BEGIN_SRC sh :post fail-capture(*this*)
> false || echo "failed"-`date +'%s'`
> #+END_SRC
> 
> #+RESULTS:
> : captured a failure in failed-1621349398. n-failures=1
> 
> 
> #+BEGIN_SRC python :post fail-capture(*this*)
> print('This did not fail')
> #+END_SRC
> 
> #+RESULTS:
> : nil
> 
> #+BEGIN_SRC python :post fail-capture(*this*)
> print('This failed-9')
> #+END_SRC
> 
> #+RESULTS:
> : captured a failure in This failed-9
> : . n-failures=2
> 
> All these approaches need you to handle and catch the errors. I think
> other kinds of errors would stop the export process. e.g. if a block
> errors out from division by zero or something.
> 
> I don't know how easy it would be to check if a src block succeeded or
> not. If it was easy, you might use the org-babel-after-execute-hook
> variable to update something when a failure is detected. Some blocks
> create an *Org-Babel Error Output buffer somewhere, but i don't know if
> this is 100% reliable across all blocks.
> 
> John
> 
> -----------------------------------
> Professor John Kitchin (he/him/his)
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu <http://kitchingroup.cheme.cmu.edu>
> 
> 
> 
> On Tue, May 18, 2021 at 9:58 AM Lennart C. Karssen <lennart@karssen.org
> <mailto:lennart@karssen.org>> wrote:
> 
>     Dear list,
> 
>     I am working on a dynamic report in Org mode, where I use source blocks
>     in various languages to process data. Several blocks produce text or
>     tables that become part of the PDF on export.
> 
>     The final chapter should state whether all checks passed, or whether one
>     or more failed (it is not necessary to know which step failed).
> 
>     In a single language environment, I would use a variable (called e.g.
>     nrChecksFailed) that would be incremented for each failing check. In a
>     single language Org document this could probably be done with a
>     :session, but given that I mix Awk, Bash, Emacs lisp and R that doesn't
>     look like the way to go. Do Org documents/source blocks have some
>     concept of a (global) variable that I can pass to my SRC blocks and
>     increment inside them?
> 
>     E.g. after somehow initialising nrChecksFailed = 0, I would like to do:
> 
>     #+header :var nFailed=nrChecksFailed :var someData=someData
>     #+begin_src R :results raw
>     do_some_check_here_on_someData
> 
>     if (check_results_OK) {
>       cat("check A passed\n")
>     } else {
>       cat("check A *failed*\n")
>       nFailed <- nFailed + 1
>     }
>     #+end_src
> 
>     So that in my conclusion chapter I can do for example:
> 
>     #+header: :var nFailed=nrChecksFailed
>     #+begin_src bash  :results raw
>     if [[ nFailed -eq 0 ]]; then
>       echo "All checks passed
>     else
>      echo "One or more checks *failed!*"
>     fi
>     #+end_src
> 
> 
>     Best regards,
> 
>     Lennart.
> 
>     -- 
>     *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
>     L.C. Karssen
>     The Netherlands
> 
>     lennart@karssen.org <mailto:lennart@karssen.org>
>     http://blog.karssen.org <http://blog.karssen.org>
>     GPG key ID: A88F554A
>     -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
> 

-- 
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
L.C. Karssen
's-Hertogenbosch
The Netherlands

lennart@karssen.org
http://blog.karssen.org
GPG key ID: A88F554A
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Global variables in Org mode document with source blocks

[Greg Minshall]

Lennart,

John's idea seems good.  also, you could generate a separate RESULT for
each language, then :var each language's "failed" RESULT into your bash
block and fail if any of them are set?

cheers, Greg

Re: Global variables in Org mode document with source blocks

[Lennart C. Karssen]

[-- Attachment #1.1: Type: text/plain, Size: 1153 bytes --]

Hi Greg,

On 18-05-2021 19:25, Greg Minshall wrote:
> Lennart,
> 
> John's idea seems good.  also, you could generate a separate RESULT for
> each language, then :var each language's "failed" RESULT into your bash
> block and fail if any of them are set?

The main problem I see with that solution is that my code blocks print
raw Org code/text, e.g. tables of results or a paragraph of text that
depends on the computations done in the code blocks. I haven't tried,
but passing and parsing those multiline RESULT blocks into a final
'conclusion' block is probably not very easy.

I ended up going for a slightly adapted version of John's first idea:
Adding a # failure-CODE line to any block that fails. That line is
ignored on export to PDF, but can still be counted in a final code block
that searches the buffer for matching regexps.


Thanks for your input.

Lennart.

> 
> cheers, Greg
> 

-- 
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
L.C. Karssen
's-Hertogenbosch
The Netherlands

lennart@karssen.org
http://blog.karssen.org
GPG key ID: A88F554A
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]