ob-clojure: possible to display results as soon as something is written to STDOUT?

ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

Hi everybody,

I was wondering if it was possible for Org-mode to write results in 
#+RESULTS as soon as something is written to STDOUT?

Here is my issue: I am writing Notebooks in Clojure using org-mode 
(clojure). I have some long-running procedures in the notebook that 
output the current status of the process (like, 10000/4986000) to the 
REPL using printnl.

With now, I am using the header "clojure :session :results output.

I am getting the output, but *only once the function finished its 
processing*. What I would like to know if it is possible to have 
Org-mode to write things when they are written to STDOUT instead then 
just when it finished?

Note: I am not sure if this is something related to Org-mode, or 
ob-clojure or Cider.

Thanks,

Fred

[-- Attachment #2: Type: text/html, Size: 1140 bytes --]

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Eric S Fraga]

On Tuesday, 29 Mar 2016 at 13:44, Frederick Giasson wrote:

[...]

> Note: I am not sure if this is something related to Org-mode, or
> ob-clojure or Cider.

None of these directly.  It's Emacs, basically: it has very minimal
support for threading so the whole process (Emacs) is waiting for the
sub-process (Clojure) to finish before doing anything else.
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi Eric,

>> Note: I am not sure if this is something related to Org-mode, or
>> ob-clojure or Cider.
> None of these directly.  It's Emacs, basically: it has very minimal
> support for threading so the whole process (Emacs) is waiting for the
> sub-process (Clojure) to finish before doing anything else.


Ok good, thanks for confirming!

Take care,

Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[John Kitchin]

This might not be totally true.

In this post I use a few different asynchronous approaches to running
Python that do something like what you want, perhaps especially the last
bit of the post.

http://kitchingroup.cheme.cmu.edu/blog/2015/11/20/Asynchronously-running-python-blocks-in-org-mode/

Maybe you could adapt it for clojure.

Frederick Giasson writes:

> Hi Eric,
>
>>> Note: I am not sure if this is something related to Org-mode, or
>>> ob-clojure or Cider.
>> None of these directly.  It's Emacs, basically: it has very minimal
>> support for threading so the whole process (Emacs) is waiting for the
>> sub-process (Clojure) to finish before doing anything else.
>
>
> Ok good, thanks for confirming!
>
> Take care,
>
> Fred


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Eric S Fraga]

On Wednesday, 30 Mar 2016 at 08:30, John Kitchin wrote:
> This might not be totally true.

Which is why I said "minimal support" ;-)

But you are correct: there are mechanisms for asynchronous threading in
Emacs.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi John!

Interesting approach :)

I am really new with org-mode, but I will try to adapt for ob-clojure.el.

One question: this function "org-babel-async-execute:python", where 
should I register it? I guess it should replace "org-babel-execute:python"?

Thanks for this precision will work on that once I know how to plug that 
into org-mode to start testing the code I will adapt.

Thanks,

Fred


> This might not be totally true.
>
> In this post I use a few different asynchronous approaches to running
> Python that do something like what you want, perhaps especially the last
> bit of the post.
>
> http://kitchingroup.cheme.cmu.edu/blog/2015/11/20/Asynchronously-running-python-blocks-in-org-mode/
>
> Maybe you could adapt it for clojure.
>
> Frederick Giasson writes:
>
>> Hi Eric,
>>
>>>> Note: I am not sure if this is something related to Org-mode, or
>>>> ob-clojure or Cider.
>>> None of these directly.  It's Emacs, basically: it has very minimal
>>> support for threading so the whole process (Emacs) is waiting for the
>>> sub-process (Clojure) to finish before doing anything else.
>>
>> Ok good, thanks for confirming!
>>
>> Take care,
>>
>> Fred
>
> --
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
>

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[John Kitchin]

you should rename it org-babel-async-execute:clojure, and adapt it to
run clojure.

I wrote the function in the org-file that is that post, and executed the
code block (C-c C-c) which "registers" the function for that instance of
emacs. Later you could put it in an init file that is loaded when Emacs starts.

and use M-x to call it. Once I added an :async option to the header args
to make C-c C-c execute it, but until it works the way you want with M-x
that is just convenience ;)

Frederick Giasson writes:

> Hi John!
>
> Interesting approach :)
>
> I am really new with org-mode, but I will try to adapt for ob-clojure.el.
>
> One question: this function "org-babel-async-execute:python", where
> should I register it? I guess it should replace "org-babel-execute:python"?
>
> Thanks for this precision will work on that once I know how to plug that
> into org-mode to start testing the code I will adapt.
>
> Thanks,
>
> Fred
>
>
>> This might not be totally true.
>>
>> In this post I use a few different asynchronous approaches to running
>> Python that do something like what you want, perhaps especially the last
>> bit of the post.
>>
>> http://kitchingroup.cheme.cmu.edu/blog/2015/11/20/Asynchronously-running-python-blocks-in-org-mode/
>>
>> Maybe you could adapt it for clojure.
>>
>> Frederick Giasson writes:
>>
>>> Hi Eric,
>>>
>>>>> Note: I am not sure if this is something related to Org-mode, or
>>>>> ob-clojure or Cider.
>>>> None of these directly.  It's Emacs, basically: it has very minimal
>>>> support for threading so the whole process (Emacs) is waiting for the
>>>> sub-process (Clojure) to finish before doing anything else.
>>>
>>> Ok good, thanks for confirming!
>>>
>>> Take care,
>>>
>>> Fred
>>
>> --
>> Professor John Kitchin
>> Doherty Hall A207F
>> Department of Chemical Engineering
>> Carnegie Mellon University
>> Pittsburgh, PA 15213
>> 412-268-7803
>> @johnkitchin
>> http://kitchingroup.cheme.cmu.edu
>>


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi John,

> you should rename it org-babel-async-execute:clojure, and adapt it to
> run clojure.

Yes

> I wrote the function in the org-file that is that post, and executed the
> code block (C-c C-c) which "registers" the function for that instance of
> emacs.

I am not sure I understand here. Once you execute the elisp block with 
that function, the function becomes defined for that instance. But when 
you execute your other block, the example python block using C-c C-c, 
how does org-mode knows to use "org-babel-async-execute:python" instead 
of "org-babel-execute:python"??

You put the cursor in the block, and then call it using M-x?

> Later you could put it in an init file that is loaded when Emacs starts.
>
> and use M-x to call it. Once I added an :async option to the header args
> to make C-c C-c execute it, but until it works the way you want with M-x
> that is just convenience ;)

Yes, this is that :async option that would be great and necessary once 
it works using M-x :)

How can this be done? (Is there an extension mechanism in org-mode for 
that, or it needs to hack the core code?)

Thanks,


Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[John Kitchin]

a
Frederick Giasson writes:

> Hi John,
>
>> you should rename it org-babel-async-execute:clojure, and adapt it to
>> run clojure.
>
> Yes
>
>> I wrote the function in the org-file that is that post, and executed the
>> code block (C-c C-c) which "registers" the function for that instance of
>> emacs.
>
> I am not sure I understand here. Once you execute the elisp block with
> that function, the function becomes defined for that instance. But when
> you execute your other block, the example python block using C-c C-c,
> how does org-mode knows to use "org-babel-async-execute:python" instead
> of "org-babel-execute:python"??
>
> You put the cursor in the block, and then call it using M-x?

Right, with the cursor in the block type M-x org-babel-async-execute:clojure
>
>> Later you could put it in an init file that is loaded when Emacs starts.
>>
>> and use M-x to call it. Once I added an :async option to the header args
>> to make C-c C-c execute it, but until it works the way you want with M-x
>> that is just convenience ;)
>
> Yes, this is that :async option that would be great and necessary once
> it works using M-x :)
>
> How can this be done? (Is there an extension mechanism in org-mode for
> that, or it needs to hack the core code?)

You can see a commented out version for shell blocks here:
https://github.com/jkitchin/jmax/blob/master/jmax-org.el#L936

It looks like I just redefined the org-babel-execute:sh function after
it was loaded. That may not be recommended good practice, but it works
;)

I am not sure why it is commented out, maybe because I don't use it alot
and its not well tested.

You could always bind your own key sequence instead of C-c C-c.


>
> Thanks,
>
>
> Fred


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi Jon,

> You can see a commented out version for shell blocks here:
> https://github.com/jkitchin/jmax/blob/master/jmax-org.el#L936
>
> It looks like I just redefined the org-babel-execute:sh function after
> it was loaded. That may not be recommended good practice, but it works
> ;)
>
> I am not sure why it is commented out, maybe because I don't use it alot
> and its not well tested.
>
> You could always bind your own key sequence instead of C-c C-c.


Yes this is right, probably better/easier/cleaner


I am almost here with my code. In fact, it is quite different than yours 
since the nrepl does async communication. So far, I am really just using 
the Buffer in a new window idea from your code since I don't need to do 
anything with external files.

Will keep you updated!

Thanks,

Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi John,

Here is a blog post that explains the changes I did. I still have one 
improvement to do (but don't how it could be done or even if it can be 
done) which is explained at the end of the post:

http://fgiasson.com/blog/index.php/2016/04/05/using-clojure-in-org-mode-and-implementing-asynchronous-processing/

Thanks for your help, it is now working the way I need it to.


Take care,

Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Nicolas Goaziou]

Hello,

Frederick Giasson <fred@fgiasson.com> writes:

> Here is a blog post that explains the changes I did. I still have one
> improvement to do (but don't how it could be done or even if it can be
> done) which is explained at the end of the post:
>
> http://fgiasson.com/blog/index.php/2016/04/05/using-clojure-in-org-mode-and-implementing-asynchronous-processing/
>
> Thanks for your help, it is now working the way I need it to.

This looks interesting. Thank you for sharing your result.

Org Babel would benefit from a generic asynchronous execution mechanism.
See for example <http://permalink.gmane.org/gmane.emacs.orgmode/101757>.

So, if anyone feels like discussing the details of an implementation of
such a thing, please chime in.

Regards,

-- 
Nicolas Goaziou

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi Nicolas,

>> Here is a blog post that explains the changes I did. I still have one
>> improvement to do (but don't how it could be done or even if it can be
>> done) which is explained at the end of the post:
>>
>> http://fgiasson.com/blog/index.php/2016/04/05/using-clojure-in-org-mode-and-implementing-asynchronous-processing/
>>
>> Thanks for your help, it is now working the way I need it to.
> This looks interesting. Thank you for sharing your result.
>
> Org Babel would benefit from a generic asynchronous execution mechanism.
> See for example <http://permalink.gmane.org/gmane.emacs.orgmode/101757>.

Indeed, it would be interesting to generalize such a generic mechanism 
directly in Babel, and not in each OB plugins.

However, I am not sure neither how generic such a solution could be, and 
what would still be required to implement in each OB plugin. My (very 
little) experience so far is the Python prototype that John wrote, and 
then my analysis of the Clojure OB plugin. As I was explaining in this 
blog post, things are quite different between the two. In Python, it 
appears that one has to run a sub-process, in Clojure this is not 
necessary since we benefit the nREPL that handle that.

So far, the best idea about how to present async process I think is what 
John was proposing with the new window. Otherwise I still wonder about 
having a real background process for this thing to run such that one can 
continue to write things in the Org buffer while a code block is running 
in background. But as I explained, I will have to seriously look into 
Elisp in order to assess the full scope of such a task.

> So, if anyone feels like discussing the details of an implementation of
> such a thing, please chime in.

So far this is the only thing I have to say about that. In mean time I 
will fix my other patch, and then I will submit another one for this 
async feature (which will probably require much more review from your part).


Thanks,

Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Nicolas Goaziou]

Hello,

Frederick Giasson <fred@fgiasson.com> writes:

> However, I am not sure neither how generic such a solution could be,
> and what would still be required to implement in each OB plugin. My
> (very little) experience so far is the Python prototype that John
> wrote, and then my analysis of the Clojure OB plugin. As I was
> explaining in this blog post, things are quite different between the
> two. In Python, it appears that one has to run a sub-process, in
> Clojure this is not necessary since we benefit the nREPL that handle
> that.

The link I gave is a first approximation of the generic behaviour.
Basically, we could spawn a new Emacs process which could then
evaluation the block synchronously.

One difficulty is to grab the "closure" of the block. Another one is to
collect and insert the results back into the buffer without being
disruptive. The thread above gave hints about that second point.

Regards,

-- 
Nicolas Goaziou

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi Nicolas,

>> However, I am not sure neither how generic such a solution could be,
>> and what would still be required to implement in each OB plugin. My
>> (very little) experience so far is the Python prototype that John
>> wrote, and then my analysis of the Clojure OB plugin. As I was
>> explaining in this blog post, things are quite different between the
>> two. In Python, it appears that one has to run a sub-process, in
>> Clojure this is not necessary since we benefit the nREPL that handle
>> that.
> The link I gave is a first approximation of the generic behaviour.
> Basically, we could spawn a new Emacs process which could then
> evaluation the block synchronously.

Ok good.

> One difficulty is to grab the "closure" of the block. Another one is to
> collect and insert the results back into the buffer without being
> disruptive. The thread above gave hints about that second point.

Humm, from the page you provided, I don't see any threads, but just the 
initial message, am I looking at the right thing?

Thanks,

Fred

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Nicolas Goaziou]

Frederick Giasson <fred@fgiasson.com> writes:

> Humm, from the page you provided, I don't see any threads, but just
> the initial message, am I looking at the right thing?

Yes, you are. There is a "go to the topic" action on the left panel.
Anyway, there is not much to see, just some pointers to start
a discussion.

Regards,

Re: ob-clojure: possible to display results as soon as something is written to STDOUT?

[Frederick Giasson]

Hi,

> Yes, you are. There is a "go to the topic" action on the left panel.
> Anyway, there is not much to see, just some pointers to start
> a discussion.

Haa great thanks, see it now :)

Let me revise a few things, suggest another patch, and then will start 
to think about these things.

Thanks,

Fred