asynchronous code evaluation

asynchronous code evaluation

[Johannes Rainer]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

hi all!

I have some pretty calculation intense R code in a code block and when I
execute the code (C-c C-c) my emacs freezes and waits for the code to
finish. Is there a way to evaluate the code asynchronously, so that I could
edit the file (obviously not the code block itself) while the code is being
evaluated?

thanks, jo

[-- Attachment #2: Type: text/html, Size: 429 bytes --]

Re: asynchronous code evaluation

[Suvayu Ali]

On Wed, Sep 04, 2013 at 07:27:01AM +0200, Johannes Rainer wrote:
> hi all!
> 
> I have some pretty calculation intense R code in a code block and when I
> execute the code (C-c C-c) my emacs freezes and waits for the code to
> finish. Is there a way to evaluate the code asynchronously, so that I could
> edit the file (obviously not the code block itself) while the code is being
> evaluated?

Not sure that is a possibility yet.  For the moment you could try
asynchronous export as a workaround.

That said, I am curious why you are knowingly tying a computationally
intensive (possibly buggy, during development i.e.) process with your
editor.  I have to work on some fairly computationally intensive
analysis with Python.  I always keep them in separate modules and
scripts run from the shell.

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: asynchronous code evaluation

[Johannes Rainer]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Wed, Sep 4, 2013 at 10:00 AM, Suvayu Ali <fatkasuvayu+linux@gmail.com>wrote:

> On Wed, Sep 04, 2013 at 07:27:01AM +0200, Johannes Rainer wrote:
> > hi all!
> >
> > I have some pretty calculation intense R code in a code block and when I
> > execute the code (C-c C-c) my emacs freezes and waits for the code to
> > finish. Is there a way to evaluate the code asynchronously, so that I
> could
> > edit the file (obviously not the code block itself) while the code is
> being
> > evaluated?
>
> Not sure that is a possibility yet.  For the moment you could try
> asynchronous export as a workaround.
>
> yes, I will do that. I just realized that I have to set
ess-ask-for-ess-directory to nil, otherwise the async export hangs because
it tries to ask where to start R.


> That said, I am curious why you are knowingly tying a computationally
> intensive (possibly buggy, during development i.e.) process with your
> editor.  I have to work on some fairly computationally intensive
> analysis with Python.  I always keep them in separate modules and
> scripts run from the shell.
>
> well, I'm using emacs/org for my data analyses in R. I thus combine
documentation (i.e. the conclusions drawn from e.g. plots created in R) and
the R code to perform the analysis in my org file. Since I'm analysing high
throughput data some tasks to handle the data are quite time consuming.

cheers, jo


> Hope this helps,
>
> --
> Suvayu
>
> Open source is the future. It sets us free.
>
>


-- 
Johannes Rainer, PhD
Applied Bioinformatics Group,
Division Molecular Pathophysiology,
Biocenter, Medical University Innsbruck,
Innrain 80/82 II, 6020 Innsbruck, Austria
and
Tyrolean Cancer Research Institute
Innrain 66, 6020 Innsbruck, Austria

Tel.:     +43 (0)512 9003 70961
Email:  johannes.rainer@i-med.ac.at
            johannes.rainer@tcri.at
URL:   http://bioinfo.i-med.ac.at

[-- Attachment #2: Type: text/html, Size: 2946 bytes --]

Re: asynchronous code evaluation

[Suvayu Ali]

On Wed, Sep 04, 2013 at 03:23:03PM +0200, Johannes Rainer wrote:
> On Wed, Sep 4, 2013 at 10:00 AM, Suvayu Ali <fatkasuvayu+linux@gmail.com>wrote:
> 
> > That said, I am curious why you are knowingly tying a computationally
> > intensive (possibly buggy, during development i.e.) process with your
> > editor.  I have to work on some fairly computationally intensive
> > analysis with Python.  I always keep them in separate modules and
> > scripts run from the shell.
> >
> well, I'm using emacs/org for my data analyses in R. I thus combine
> documentation (i.e. the conclusions drawn from e.g. plots created in R) and
> the R code to perform the analysis in my org file. Since I'm analysing high
> throughput data some tasks to handle the data are quite time consuming.

Okay, in that case I would try to achieve the asynchronicity in R.  I
think you would have better luck that way.

-- 
Suvayu

Open source is the future. It sets us free.

Re: asynchronous code evaluation

[Allen S. Rout]

On 09/04/2013 09:23 AM, Johannes Rainer wrote:
>
> well, I'm using emacs/org for my data analyses in R. I thus combine
> documentation (i.e. the conclusions drawn from e.g. plots created in R)
> and the R code to perform the analysis in my org file. Since I'm
> analysing high throughput data some tasks to handle the data are quite
> time consuming.
> 

In the past, I have solved that problem by adding a makefile in the
directory, which produces a separate fragment to be consumed/included by
the org file.  Make based on the updated data, publish based on the
static outuput of the make process.

If you let the sim run "in the background" while you edit, you'll likely
get to a point, whilst tweaking or such, that you have dispatched
several long-running jobs in parallel, and only care about the last
result.  That's probably counterproductive. :)


- Allen S. Rout

Re: asynchronous code evaluation

[Brett Viren]

[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]

Hi Johannes,

Johannes Rainer <johannes.rainer@i-med.ac.at> writes:

> well, I'm using emacs/org for my data analyses in R. I thus combine
> documentation (i.e. the conclusions drawn from e.g. plots created in
> R) and the R code to perform the analysis in my org file. Since I'm
> analysing high throughput data some tasks to handle the data are quite
> time consuming.

A variation on Allen's Makefile approach, I structure my (Python) code
to be idempotent.  The "real" code never (re)runs unless its output
either doesn't exist or is older than the input.  Normally when org runs
the code's higher level interface it's essentially a no-op and very
fast.  If something does change it's usually because I'm doing
development and will be running the code outside of org enough that it's
reached its "no-op state" before I go back into emacs to refresh/edit
the org document.  Also, I keep things synchronous to avoid having to
multiple, competing processes running at once.

What I want is to take this approach and generalize into a more formal
workflow system.  One which handles caching files and in-memory objects
in a versioned store to provide data provenance and idempotent running
in a general sense.  There's already been some work on this ("bein",
"pyutilib workflow" and "sumatra" to name some Python oriented ones) but
so far nothing that "just fits".  I'm in the middle of trying my hand at
something better, but it's not yet usable.

-Brett.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: asynchronous code evaluation

[Charles Berry]

Johannes Rainer <johannes.rainer <at> gmail.com> writes:

> 
> 
> hi all!
> 
> I have some pretty calculation intense R code in a code block and when I 
> execute the code (C-c C-c) my emacs freezes and waits for the code to 
> finish. Is there a way to evaluate the code asynchronously, so that I 
> could edit the file (obviously not the code block itself) while the code 
> is being evaluated?
> 
> thanks, jo
>

Try this:

C-c ' in the R src block. 

Then use C-c C-c (ess-eval-region-or-function-or-paragraph-and-step) to 
execute the long running code from the ESS edit buffer.

You may need to select the region if there are several paragraphs, or 
delete any blank lines.

==

Another alternative is to use ravel. 

  https://github.com/chasberry/orgmode-accessories

Create a knitr or Sweave doc from the org doc or a suitable subtree of the 
computationally intensive parts. 

Then you can knit() the doc from the R session and switch back to
the buffer you want to edit. Also, you might want to turn on caching in the
knitr doc to enable a quick restart when you log in again.

Re: asynchronous code evaluation

[Eric Schulte]

Johannes Rainer <johannes.rainer@gmail.com> writes:

> hi all!
>
> I have some pretty calculation intense R code in a code block and when I
> execute the code (C-c C-c) my emacs freezes and waits for the code to
> finish. Is there a way to evaluate the code asynchronously, so that I could
> edit the file (obviously not the code block itself) while the code is being
> evaluated?
>

Hi Jo,

This is not currently possible, but it would certainly be a nice feature
and is something that has been requested before.

This would require some general Org-mode support before any language
specific code is written (ideally no language specific code would be
required).  For anyone who feels like tackling such a project the first
step would be to look at `org-babel-comint-with-output' and
`org-babel-eval'.  I'm not sure what the best way is to wait until the
external process returns to handle the code block results, I'd guess
we'd either want to pass closures to those functions, or somehow yield
while the external process runs.

Some careful thought of the ramifications of asynchronous code block
evaluation may be required.  E.g., would every piece of code which calls
a code block (from "C-c C-c" to :var resolution) need to be updated to
call the code block in an asynchronous manner?

Best,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D