* asynchronous python code blocks in org-mode
@ 2015-11-20 16:52 John Kitchin
2015-11-30 11:38 ` Karl Voit
0 siblings, 1 reply; 5+ messages in thread
From: John Kitchin @ 2015-11-20 16:52 UTC (permalink / raw)
To: emacs-orgmode@gnu.org; +Cc: Rebecca Taylor
Hi everyone,
I worked out some ideas on running Python blocks asynchronously in
org-mode, while seeing the output of the code, /and/ still capturing the
output in org-mode!
The post is here:
http://kitchingroup.cheme.cmu.edu/blog/2015/11/20/Asynchronously-running-python-blocks-in-org-mode/
It seems to work pretty well as a proof of concept, but the present
approach has some limitations, e.g. no sessions, no src-block :var, and
maybe others. These might be fixable at some point.
Anyway, I thought I would share it, to see if anyone has thoughts on
it, and in case it looks useful to you.
--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: asynchronous python code blocks in org-mode
2015-11-20 16:52 asynchronous python code blocks in org-mode John Kitchin
@ 2015-11-30 11:38 ` Karl Voit
2015-11-30 21:06 ` John Kitchin
0 siblings, 1 reply; 5+ messages in thread
From: Karl Voit @ 2015-11-30 11:38 UTC (permalink / raw)
To: emacs-orgmode
* John Kitchin <jkitchin@andrew.cmu.edu> wrote:
> Hi everyone,
Hi John!
> I worked out some ideas on running Python blocks asynchronously in
> org-mode, while seeing the output of the code, /and/ still capturing the
> output in org-mode!
Found the link to your video + blog on Twitter. On the weekend, I
watched it with big eyes and tried your code on my side as well:
great work!
> It seems to work pretty well as a proof of concept, but the present
> approach has some limitations, e.g. no sessions, no src-block :var, and
> maybe others. These might be fixable at some point.
>
> Anyway, I thought I would share it, to see if anyone has thoughts on
> it, and in case it looks useful to you.
I'm loving it.
I'd love to see this async method for shell script as well. Many
shell snippets I am using do take some time: backup, audio
conversion, bulk image conversion, ...
1) How big is the chance that we're going to see async babel block
execution by default in a future org-mode release?
2) In the meantime: Wouldn't it be nice to have additional
source-block definitions like python + apython (for asynchronous
python) or sh + ash?
--
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
> get Memacs from https://github.com/novoid/Memacs <
https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: asynchronous python code blocks in org-mode
2015-11-30 11:38 ` Karl Voit
@ 2015-11-30 21:06 ` John Kitchin
2015-12-01 20:19 ` Tom
0 siblings, 1 reply; 5+ messages in thread
From: John Kitchin @ 2015-11-30 21:06 UTC (permalink / raw)
To: Karl Voit; +Cc: emacs-orgmode
Karl Voit writes:
> * John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>> Hi everyone,
>
> Hi John!
>
>> I worked out some ideas on running Python blocks asynchronously in
>> org-mode, while seeing the output of the code, /and/ still capturing the
>> output in org-mode!
>
> Found the link to your video + blog on Twitter. On the weekend, I
> watched it with big eyes and tried your code on my side as well:
> great work!
>
>> It seems to work pretty well as a proof of concept, but the present
>> approach has some limitations, e.g. no sessions, no src-block :var, and
>> maybe others. These might be fixable at some point.
>>
>> Anyway, I thought I would share it, to see if anyone has thoughts on
>> it, and in case it looks useful to you.
>
> I'm loving it.
>
> I'd love to see this async method for shell script as well. Many
> shell snippets I am using do take some time: backup, audio
> conversion, bulk image conversion, ...
It turns out to be pretty trivial to modify this for sh. Change change
org-babel-variable-assignments:python to
org-babel-variable-assignments:sh and python
to bash, and rename the function to org-babel-async-execute:sh, and you
should have it.
If you don't mind redefining an org command, this will let you put
:async in the header to trigger the asynchronous mode, otherwise, run
the usual way:
(defun org-babel-execute:sh (body params)
"Execute a block of Shell commands with Babel.
This function is called by `org-babel-execute-src-block'."
(let* ((session (org-babel-sh-initiate-session
(cdr (assoc :session params))))
(async (assoc :async params))
(stdin (let ((stdin (cdr (assoc :stdin params))))
(when stdin (org-babel-sh-var-to-string
(org-babel-ref-resolve stdin)))))
(full-body (org-babel-expand-body:generic
body params (org-babel-variable-assignments:sh params))))
(if async
;; run asynchronously
(org-babel-async-execute:sh)
;; else run regularly
(org-babel-reassemble-table
(org-babel-sh-evaluate session full-body params stdin)
(org-babel-pick-name
(cdr (assoc :colname-names params)) (cdr (assoc :colnames params)))
(org-babel-pick-name
(cdr (assoc :rowname-names params)) (cdr (assoc :rownames
params)))))))
I have not thought about sessions and asynchronous execution. It would
mean a queue and a different way to pass code to the process, which I
have not thought through yet. What to do when there are dependencies for
example,...
>
> 1) How big is the chance that we're going to see async babel block
> execution by default in a future org-mode release?
>
> 2) In the meantime: Wouldn't it be nice to have additional
> source-block definitions like python + apython (for asynchronous
> python) or sh + ash?
--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: asynchronous python code blocks in org-mode
2015-11-30 21:06 ` John Kitchin
@ 2015-12-01 20:19 ` Tom
2015-12-01 23:07 ` John Kitchin
0 siblings, 1 reply; 5+ messages in thread
From: Tom @ 2015-12-01 20:19 UTC (permalink / raw)
To: John Kitchin; +Cc: Karl Voit, emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]
On Mon, Nov 30, 2015 at 1:06 PM, John Kitchin <jkitchin@andrew.cmu.edu>
wrote:
> I have not thought about sessions and asynchronous execution. It would
> mean a queue and a different way to pass code to the process, which I
> have not thought through yet. What to do when there are dependencies for
> example,...
>
A good way of converting synchronous into asynchronous code is to use
futures/promises. Usually, that's done via data structures, but within
Emacs buffers, it could be done via text strings.
How might that work? org-babel-execute:python could wait for, say, 0.1 sec
for an immediate result. If the computation doesn't finish within that
time, it returns a "future", a magic string like
"org_mode_future_result(1234) ###MAGIC###". This would then get inserted as
output into the org-mode buffer. Later, when the actual result becomes
available from the subprocess, that invokes a callback on the org-python
mode buffer and replaces tihs magic string with the actual result, and
dequeues and executes the next command if necessary.
(Picking a Python-parseable expression would even allow futures to be used
in some other Python computations as if it were an actual value.)
I think that would allow much of the current API to remain in place.
Obviously, some things simply can't work. For example,
org-babel-reassemble-table expects an actual result, not a future; such
post-processing would have to move to a hook function, which probably would
be cleaner anyway.
Tom
[-- Attachment #2: Type: text/html, Size: 2074 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: asynchronous python code blocks in org-mode
2015-12-01 20:19 ` Tom
@ 2015-12-01 23:07 ` John Kitchin
0 siblings, 0 replies; 5+ messages in thread
From: John Kitchin @ 2015-12-01 23:07 UTC (permalink / raw)
To: Tom; +Cc: Karl Voit, emacs-orgmode
Tom writes:
> On Mon, Nov 30, 2015 at 1:06 PM, John Kitchin <jkitchin@andrew.cmu.edu>
> wrote:
>
>> I have not thought about sessions and asynchronous execution. It would
>> mean a queue and a different way to pass code to the process, which I
>> have not thought through yet. What to do when there are dependencies for
>> example,...
>>
>
> A good way of converting synchronous into asynchronous code is to use
> futures/promises. Usually, that's done via data structures, but within
> Emacs buffers, it could be done via text strings.
>
> How might that work? org-babel-execute:python could wait for, say, 0.1 sec
> for an immediate result. If the computation doesn't finish within that
> time, it returns a "future", a magic string like
> "org_mode_future_result(1234) ###MAGIC###". This would then get inserted as
> output into the org-mode buffer. Later, when the actual result becomes
> available from the subprocess, that invokes a callback on the org-python
> mode buffer and replaces tihs magic string with the actual result, and
> dequeues and executes the next command if necessary.
This is basically what happens now, I insert a uuid that gets replaced by
a callback function. A session is doable I think as you describe. Its
not a high-priority for me though. I feel like it would eventually be
necessary to have some queue management functions, e.g. what to do if
you run a block twice? or want to cancel a run, etc... How to make sure
dependencies are correct, e.g. you can start a second block before the
first block finishes, but it is queued. etc...
> (Picking a Python-parseable expression would even allow futures to be used
> in some other Python computations as if it were an actual value.)
>
> I think that would allow much of the current API to remain in place.
> Obviously, some things simply can't work. For example,
> org-babel-reassemble-table expects an actual result, not a future; such
> post-processing would have to move to a hook function, which probably would
> be cleaner anyway.
>
> Tom
--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-12-01 23:07 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-20 16:52 asynchronous python code blocks in org-mode John Kitchin
2015-11-30 11:38 ` Karl Voit
2015-11-30 21:06 ` John Kitchin
2015-12-01 20:19 ` Tom
2015-12-01 23:07 ` John Kitchin
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).