From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Kitchin Subject: Re: babel and long-running computations Date: Sun, 20 Apr 2014 09:29:33 -0400 Message-ID: References: <87lhv4qr7l.fsf@grothesque.org> <534FA332.1080304@gmail.com> <87lhv4ut1n.fsf@grothesque.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7b6d7fb491578c04f7795faa Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:51916) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WbroO-00016I-1Y for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:29:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WbroM-0003Kp-NT for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:29:35 -0400 Received: from mail-pd0-x229.google.com ([2607:f8b0:400e:c02::229]:64767) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WbroM-0003Kk-BY for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:29:34 -0400 Received: by mail-pd0-f169.google.com with SMTP id fp1so2875919pdb.0 for ; Sun, 20 Apr 2014 06:29:33 -0700 (PDT) In-Reply-To: <87lhv4ut1n.fsf@grothesque.org> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Christoph Groth Cc: "emacs-orgmode@gnu.org" --047d7b6d7fb491578c04f7795faa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You are describing what I see as a need to run processes asynchronously. We use the Torque queue system for this. It takes care of scheduling and running jobs on a cluster. On Thu, Apr 17, 2014 at 6:52 AM, Christoph Groth wrote: > Hi Julian, > > > Generally, org-babel is suited for long running computations. Its > > caching can help you avoid rerunning code chunks. And long runtime doe= s > > not conflict with the idea of reproducible research, it just may be not > > very comfortable for the user. > > I agree of course that it=E2=80=99s generally a good idea to structure th= e > analysis in small logical steps and to save intermediate results. Let > me narrow down my question then: does org-babel support working with > tasks that take several minutes to execute? By this I mean (most > important first): > > - Not freezing the editor during the execution of a task > You code-block should exit if a job is submitted, and record the job-id so that you can check it later. Alternatively, you could do this in ipython notebooks if you have python functions that return jobids. > - Being able to execute multiple independent tasks in parallel > No problem. this is what queue systems were designed for. > - Being able to interrupt a running task > you do this with the queue commands, e.g. qdel jobid > - Being able to inspect the incomplete output of a running task > This is just checking the output files in the running jobs directories. > > I=E2=80=99d love to hear about any frameworks or workflows that fulfill t= hese > requirements. > > Cheers > > Christoph > > > PS. > > I=E2=80=99m using ipython notebooks but I=E2=80=99m not happy with them b= ecause of the > freezing problem and the complete lack of isolation of tasks within a > single notebook (they live in a common mutable namespace). I think a > useful framework must be =E2=80=9Cfunctional=E2=80=9D at the highest leve= l for caching > and dependencies to be useful. > > > --047d7b6d7fb491578c04f7795faa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You are describing what I see a= s a need to run processes asynchronously.

We use the Torque queue system for this. It takes care of schedulin= g and running jobs on a cluster.

On Thu, Apr= 17, 2014 at 6:52 AM, Christoph Groth <christoph@grothesque.org= > wrote:
Hi Julian,

> Generally, org-babel is suited for long running computations. =C2=A0It= s
> caching can help you avoid rerunning code chunks. =C2=A0And long runti= me does
> not conflict with the idea of reproducible research, it just may be no= t
> very comfortable for the user.

I agree of course that it=E2=80=99s generally a good idea to structur= e the
analysis in small logical steps and to save intermediate results. =C2=A0Let=
me narrow down my question then: does org-babel support working with
tasks that take several minutes to execute? =C2=A0By this I mean (most
important first):

- Not freezing the editor during the execution of a task

You code-block should exit if a job is submitted, and rec= ord the job-id so that you can check it later. Alternatively, you could do = this in ipython notebooks if you have python functions that return jobids.<= br>
=C2=A0
- Being able to execute multiple independent tasks in parallel

No problem. this is what queue systems were designe= d for.
=C2=A0
- Being able to interrupt a running task

you do this with the queue commands, e.g. qdel jobid
=C2=A0=
- Being able to inspect the incomplete output of a running task

This is just checking the output files in the runn= ing jobs directories.
=C2=A0

I=E2=80=99d love to hear about any frameworks or workflows that fulfill the= se
requirements.

Cheers

Christoph


PS.

I=E2=80=99m using ipython notebooks but I=E2=80=99m not happy with them bec= ause of the
freezing problem and the complete lack of isolation of tasks within a
single notebook (they live in a common mutable namespace). =C2=A0I think a<= br> useful framework must be =E2=80=9Cfunctional=E2=80=9D at the highest level = for caching
and dependencies to be useful.



--047d7b6d7fb491578c04f7795faa--