emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: Brett Viren <bv@bnl.gov>
To: Johannes Rainer <johannes.rainer@i-med.ac.at>
Cc: emacs-orgmode@gnu.org
Subject: Re: asynchronous code evaluation
Date: Wed, 04 Sep 2013 15:20:26 -0400	[thread overview]
Message-ID: <ir4fvtkz951.fsf@lycastus.phy.bnl.gov> (raw)
In-Reply-To: <CABxYo6oPjzA7p+UZNkdzqw5YEsOWBCupS7V7Fn=0-1ipr5tE_Q@mail.gmail.com> (Johannes Rainer's message of "Wed, 4 Sep 2013 15:23:03 +0200")

[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]

Hi Johannes,

Johannes Rainer <johannes.rainer@i-med.ac.at> writes:

> well, I'm using emacs/org for my data analyses in R. I thus combine
> documentation (i.e. the conclusions drawn from e.g. plots created in
> R) and the R code to perform the analysis in my org file. Since I'm
> analysing high throughput data some tasks to handle the data are quite
> time consuming.

A variation on Allen's Makefile approach, I structure my (Python) code
to be idempotent.  The "real" code never (re)runs unless its output
either doesn't exist or is older than the input.  Normally when org runs
the code's higher level interface it's essentially a no-op and very
fast.  If something does change it's usually because I'm doing
development and will be running the code outside of org enough that it's
reached its "no-op state" before I go back into emacs to refresh/edit
the org document.  Also, I keep things synchronous to avoid having to
multiple, competing processes running at once.

What I want is to take this approach and generalize into a more formal
workflow system.  One which handles caching files and in-memory objects
in a versioned store to provide data provenance and idempotent running
in a general sense.  There's already been some work on this ("bein",
"pyutilib workflow" and "sumatra" to name some Python oriented ones) but
so far nothing that "just fits".  I'm in the middle of trying my hand at
something better, but it's not yet usable.

-Brett.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

  parent reply	other threads:[~2013-09-04 19:21 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-09-04  5:27 asynchronous code evaluation Johannes Rainer
2013-09-04  8:00 ` Suvayu Ali
2013-09-04 13:23   ` Johannes Rainer
2013-09-04 15:25     ` Suvayu Ali
2013-09-04 16:59     ` Allen S. Rout
2013-09-04 19:20     ` Brett Viren [this message]
2013-09-04 16:08 ` Charles Berry
2013-09-06 17:54 ` Eric Schulte

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ir4fvtkz951.fsf@lycastus.phy.bnl.gov \
    --to=bv@bnl.gov \
    --cc=emacs-orgmode@gnu.org \
    --cc=johannes.rainer@i-med.ac.at \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).