source blocks DAG evaluation

source blocks DAG evaluation

[c4t0]

Hi,

Is it possible to have a dependency hierarchy of source blocks?

i.e.: in C, if you use libA from a compilation unit and that library needs libB, you don't need to include it in main.c

-> main.c
#include "libB.h"
-> libB.c
#include "libA.h"

you don't need to:
-> main.c
#include "libB.h"
#include "libA.h"

because library headers are closed under inclusion.

I haven't succeeded in doing the same in org-mode. Even after populating org-babel-library-of-babel.

Using #+call just doesn't work. Using :var is better, evaluates all, but there appears to lack session support, it doesn't check for cycles and it feels a little hacky

With #+call I need to do it like this:

#+name: libA
#+begin_src scheme :results none
(define hi "hello")
#+end_src

#+name: libB
#+begin_src scheme :results none
(define greetings (string-append hi ", " "to all the people!"))
#+end_src

here is my "main" I need to C-c C-c in each #+call line and write the :session that the code block uses in each one, and do it in the correct order. If I C-c C-c in libB first it won't eval because 'hi' is not defined.

#+call: libB[:session example]
#+call: libA[:session example]
#+begin_src scheme :session example :results output
(display greetings)
#+end_src

source blocks can be #+call(ed) but aren't closed under #+call (a source block can be called but then the callee won't) 

instead I would like to :

#+name: libA
#+begin_src scheme :results none
(define hi "hello")
#+end_src

#+call: libA
#+name: libB
#+begin_src scheme :results none
(define greetings (string-append hi ", " "to all the people!"))
#+end_src

#+call: libB
#+begin_src scheme :session example :results output
(display greetings)
#+end_src

- there shouldn't be needed to C-c C-c in the #+call line, evaluating the source block alone should suffice.
- there shouldn't be a need to write the :session
  - it should use the session of the user evaled block, unless specified otherwise

In the other hand, using :var with a dummy variable:

#+name: libA
#+begin_src scheme :results none
(define hi "hello")
#+end_src

#+name: libB
#+begin_src scheme :results none :var _=libA
(define greetings (string-append hi ", " "to all the people!"))
#+end_src

#+HEADER: :var _=libB
#+begin_src scheme :session example :results output 
(display greetings)
#+end_src

It evals libA then libB and finally the (display greetings) code.
But it fails, because the :session example is ignored. Even if I add a :session example to every source block (which would be really bad, sessión must be decided by the consumer) it doesn't work. I think that is because :var expects a value, so it just opens a new session to evaluate code every time.

Besides if there are any dependency cycles, it just fails with: Lisp nesting exceeds ‘max-lisp-eval-depth’

So if I'm right and there is not an implemented way to do this, how can we do it? Adding session support for :var? constructing a DAG of #+calls and then evaluating in order? maybe using a new header?

COD.

Re: source blocks DAG evaluation

[Thomas S. Dye]

Aloha c4to,

I would be tempted to use noweb expansion here.

#+name: libB
#+begin_src scheme :results none :noweb yes
<<libA>>
(define greetings (string-append hi ", " "to all the people!"))
#+end_src

#+begin_src scheme :session example :results output :noweb yes
<<libB>>
(display greetings)
#+end_src

Does this do what you want?

All the best,
Tom

c4t0 writes:

> Hi,
>
> Is it possible to have a dependency hierarchy of source blocks?
>
> i.e.: in C, if you use libA from a compilation unit and that 
> library needs libB, you don't need to include it in main.c
>
> -> main.c
> #include "libB.h"
> -> libB.c
> #include "libA.h"
>
> you don't need to:
> -> main.c
> #include "libB.h"
> #include "libA.h"
>
> because library headers are closed under inclusion.
>
> I haven't succeeded in doing the same in org-mode. Even after 
> populating org-babel-library-of-babel.
>
> Using #+call just doesn't work. Using :var is better, evaluates 
> all, but there appears to lack session support, it doesn't check 
> for cycles and it feels a little hacky
>
> With #+call I need to do it like this:
>
> #+name: libA
> #+begin_src scheme :results none
> (define hi "hello")
> #+end_src
>
> #+name: libB
> #+begin_src scheme :results none
> (define greetings (string-append hi ", " "to all the people!"))
> #+end_src
>
> here is my "main" I need to C-c C-c in each #+call line and 
> write the :session that the code block uses in each one, and do 
> it in the correct order. If I C-c C-c in libB first it won't 
> eval because 'hi' is not defined.
>
> #+call: libB[:session example]
> #+call: libA[:session example]
> #+begin_src scheme :session example :results output
> (display greetings)
> #+end_src
>
> source blocks can be #+call(ed) but aren't closed under #+call 
> (a source block can be called but then the callee won't)
>
> instead I would like to :
>
> #+name: libA
> #+begin_src scheme :results none
> (define hi "hello")
> #+end_src
>
> #+call: libA
> #+name: libB
> #+begin_src scheme :results none
> (define greetings (string-append hi ", " "to all the people!"))
> #+end_src
>
> #+call: libB
> #+begin_src scheme :session example :results output
> (display greetings)
> #+end_src
>
> - there shouldn't be needed to C-c C-c in the #+call line, 
> evaluating the source block alone should suffice.
> - there shouldn't be a need to write the :session
>   - it should use the session of the user evaled block, unless 
>   specified otherwise
>
> In the other hand, using :var with a dummy variable:
>
> #+name: libA
> #+begin_src scheme :results none
> (define hi "hello")
> #+end_src
>
> #+name: libB
> #+begin_src scheme :results none :var _=libA
> (define greetings (string-append hi ", " "to all the people!"))
> #+end_src
>
> #+HEADER: :var _=libB
> #+begin_src scheme :session example :results output
> (display greetings)
> #+end_src
>
> It evals libA then libB and finally the (display greetings) 
> code.
> But it fails, because the :session example is ignored. Even if I 
> add a :session example to every source block (which would be 
> really bad, sessión must be decided by the consumer) it doesn't 
> work. I think that is because :var expects a value, so it just 
> opens a new session to evaluate code every time.
>
> Besides if there are any dependency cycles, it just fails with: 
> Lisp nesting exceeds ‘max-lisp-eval-depth’
>
> So if I'm right and there is not an implemented way to do this, 
> how can we do it? Adding session support for :var? constructing 
> a DAG of #+calls and then evaluating in order? maybe using a new 
> header?
>
> COD.


--
Thomas S. Dye
https://tsdye.online/tsdye

Re: source blocks DAG evaluation

[c4t0]

Hello Thomas!

That is fairly nice, thanks! I can use it for the moment.

I see a minor problem, if you export it, the last block will show all the
code and that may be redundant or undesired. But that can be solved
fiddling with exporting options or code I guess. btw, using :noweb no-export,
just leaves the <<lib*>> sintax, so it's not what I want. 

In this sense it also doesn't do any cycle resolving, when a there is
one it just fails with
"org-at-heading-p: Variable binding depth exceeds max-specpdl-size"

That may be ok, you shouldn't have cycles. But there is a more realist
problem, you *can* have, diamonds dependencies.

Say
libA -> libB -> libD -> main
     \> libC /

If you <<libD>> from main, you will get two copies of libA source
code. That may be a problem in some languages and it's definitely bad at
export time.

The good about noweb expansion is that you can ignore the problem with
the sessions all together and it will work with things that doesn't have
sessions at all (now I remember a home work in assembler that I did in
org-mode with noweb expansion ...)

So, maybe if I try to improve anything, to address this issue, maybe is
the code for noweb expansion? Maybe some options to conditional output
the template expansion or just remove the <<*>> calls.  I don't know if
the semantics of what I want go along a template expansion ... I have
the feeling that we might need something particular for this, like a
(require ...) for org-mode source blocks.

thanks!

COD.

"Thomas S. Dye" <tsd@tsdye.online> writes:

> Aloha c4to,
>
> I would be tempted to use noweb expansion here.
>
> #+name: libB
> #+begin_src scheme :results none :noweb yes
>
> <<libA>>
> (define greetings (string-append hi ", " "to all the people!"))
> #+end_src
>
> #+begin_src scheme :session example :results output :noweb yes
> <<libB>>
> (display greetings)
> #+end_src
>
> Does this do what you want?
>
> All the best,
> Tom
>
> c4t0 writes:
>
>> Hi,
>>
>> Is it possible to have a dependency hierarchy of source blocks?
>>
>> i.e.: in C, if you use libA from a compilation unit and that 
>> library needs libB, you don't need to include it in main.c
>>
>> -> main.c
>> #include "libB.h"
>> -> libB.c
>> #include "libA.h"
>>
>> you don't need to:
>> -> main.c
>> #include "libB.h"
>> #include "libA.h"
>>
>> because library headers are closed under inclusion.
>>
>> I haven't succeeded in doing the same in org-mode. Even after 
>> populating org-babel-library-of-babel.
>>
>> Using #+call just doesn't work. Using :var is better, evaluates 
>> all, but there appears to lack session support, it doesn't check 
>> for cycles and it feels a little hacky
>>
>> With #+call I need to do it like this:
>>
>> #+name: libA
>> #+begin_src scheme :results none
>> (define hi "hello")
>> #+end_src
>>
>> #+name: libB
>> #+begin_src scheme :results none
>> (define greetings (string-append hi ", " "to all the people!"))
>> #+end_src
>>
>> here is my "main" I need to C-c C-c in each #+call line and 
>> write the :session that the code block uses in each one, and do 
>> it in the correct order. If I C-c C-c in libB first it won't 
>> eval because 'hi' is not defined.
>>
>> #+call: libB[:session example]
>> #+call: libA[:session example]
>> #+begin_src scheme :session example :results output
>> (display greetings)
>> #+end_src
>>
>> source blocks can be #+call(ed) but aren't closed under #+call 
>> (a source block can be called but then the callee won't)
>>
>> instead I would like to :
>>
>> #+name: libA
>> #+begin_src scheme :results none
>> (define hi "hello")
>> #+end_src
>>
>> #+call: libA
>> #+name: libB
>> #+begin_src scheme :results none
>> (define greetings (string-append hi ", " "to all the people!"))
>> #+end_src
>>
>> #+call: libB
>> #+begin_src scheme :session example :results output
>> (display greetings)
>> #+end_src
>>
>> - there shouldn't be needed to C-c C-c in the #+call line, 
>> evaluating the source block alone should suffice.
>> - there shouldn't be a need to write the :session
>>   - it should use the session of the user evaled block, unless 
>>   specified otherwise
>>
>> In the other hand, using :var with a dummy variable:
>>
>> #+name: libA
>> #+begin_src scheme :results none
>> (define hi "hello")
>> #+end_src
>>
>> #+name: libB
>> #+begin_src scheme :results none :var _=libA
>> (define greetings (string-append hi ", " "to all the people!"))
>> #+end_src
>>
>> #+HEADER: :var _=libB
>> #+begin_src scheme :session example :results output
>> (display greetings)
>> #+end_src
>>
>> It evals libA then libB and finally the (display greetings) 
>> code.
>> But it fails, because the :session example is ignored. Even if I 
>> add a :session example to every source block (which would be 
>> really bad, sessión must be decided by the consumer) it doesn't 
>> work. I think that is because :var expects a value, so it just 
>> opens a new session to evaluate code every time.
>>
>> Besides if there are any dependency cycles, it just fails with: 
>> Lisp nesting exceeds ‘max-lisp-eval-depth’
>>
>> So if I'm right and there is not an implemented way to do this, 
>> how can we do it? Adding session support for :var? constructing 
>> a DAG of #+calls and then evaluating in order? maybe using a new 
>> header?
>>
>> COD.
>
>
> --
> Thomas S. Dye
> https://tsdye.online/tsdye