emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: John Kitchin <jkitchin@andrew.cmu.edu>
To: Christoph Groth <christoph@grothesque.org>
Cc: "emacs-orgmode@gnu.org" <emacs-orgmode@gnu.org>
Subject: Re: babel and long-running computations
Date: Fri, 18 Apr 2014 09:20:46 -0400	[thread overview]
Message-ID: <CAJ51ETqwnOLzFbT9dqRUgjqZ_dXEdpunmWT6BS-LLcE=q_fK1Q@mail.gmail.com> (raw)
In-Reply-To: <87lhv4qr7l.fsf@grothesque.org>

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

we routinely do this, in the following way. We run jobs that may take up to
a week to finish, and they are usually run on a cluster. Our setup relies
on the following behavior for a script.

1. you can run the script anytime you want, and it can tell the state of
the calculation by some means. If the script has never been run before, it
submits the job to a queue and exits. If the job is still in the queue, it
exits, and if the job is done, it gives you the result. We manage this in
python using exceptions to control the workflow. You can even have the
script output an org-task marked TODO so you get an item on your agenda to
check it later to see if it is done.

Essentially the way it works is:
when you run a script there is some code that checks the state, which in
our case is looking for certain output files, and flag files. If the output
files do not exist, we submit the job to a queue, and create a jobid file
containing the output from the queue submission command. Then the script
raises a JobSubmitted exception which we can handle if we want to.

Later, I run the code block again. A  jobid file is found, so the script
gets the jobid, and checks if the job is still in the queue. If so a
JobQueued exception is raised, which we can handle if we want to. If not,
we delete the jobid file, and proceed to analyze the output files.

All of that is done behind the scenes.

You can see how this is done at
https://github.com/jkitchin/jasp/blob/master/jasp/jasp_extensions.py#L416

this is a highly customized approach to a very specific code, but you could
adapt the ideas to other codes and computing environments I think.

John

-----------------------------------
John Kitchin
Associate Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu



On Thu, Apr 17, 2014 at 4:45 AM, Christoph Groth
<christoph@grothesque.org>wrote:

> Dear all,
>
> I’m considering the use of org-babel for “reproducible science”.  Is
> anyone successfully using org-babel for research that involves
> calculations that typically take a few minutes up to several hours on a
> single CPU (i.e. up to a few minutes on a cluster)?  It seems to me that
> currently org-babel is not well suited for such an usage profile, but
> I’d love to be taught otherwise.
>
> Many thanks,
> Christoph
>
>
>

[-- Attachment #2: Type: text/html, Size: 3113 bytes --]

  parent reply	other threads:[~2014-04-18 13:20 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-04-17  8:45 babel and long-running computations Christoph Groth
2014-04-17  9:47 ` Julian Gehring
2014-04-17 10:52   ` Christoph Groth
2014-04-17 12:06     ` Thorsten Jolitz
2014-04-17 12:45       ` Christoph Groth
2014-04-17 14:58         ` Waldemar Quevedo
2014-04-17 15:29         ` Eric Schulte
2014-04-17 16:29           ` Christoph Groth
2014-04-17 16:44             ` Nick Dokos
2014-04-17 15:33     ` Eric S Fraga
2014-04-20 13:29     ` John Kitchin
2014-04-18 13:20 ` John Kitchin [this message]
2014-04-19 21:49   ` Christoph Groth
2014-04-20 13:21     ` John Kitchin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJ51ETqwnOLzFbT9dqRUgjqZ_dXEdpunmWT6BS-LLcE=q_fK1Q@mail.gmail.com' \
    --to=jkitchin@andrew.cmu.edu \
    --cc=christoph@grothesque.org \
    --cc=emacs-orgmode@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).