From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Kitchin Subject: Re: babel and long-running computations Date: Sun, 20 Apr 2014 09:21:09 -0400 Message-ID: References: <87lhv4qr7l.fsf@grothesque.org> <874n1pouqd.fsf@grothesque.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7ba977a09a4c8404f779416c Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50933) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WbrgK-0005hn-RN for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:21:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WbrgG-0000vu-2Q for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:21:16 -0400 Received: from mail-pd0-x22e.google.com ([2607:f8b0:400e:c02::22e]:54442) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WbrgF-0000vn-NY for emacs-orgmode@gnu.org; Sun, 20 Apr 2014 09:21:12 -0400 Received: by mail-pd0-f174.google.com with SMTP id y13so2857468pdi.33 for ; Sun, 20 Apr 2014 06:21:10 -0700 (PDT) In-Reply-To: <874n1pouqd.fsf@grothesque.org> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Christoph Groth , "emacs-orgmode@gnu.org" --047d7ba977a09a4c8404f779416c Content-Type: text/plain; charset=UTF-8 On Sat, Apr 19, 2014 at 5:49 PM, Christoph Groth wrote: > The following message is a courtesy copy of an article > that has been posted to gmane.emacs.orgmode as well. > > Thank you, John, for your detailed reply. > > > we routinely do this, in the following way. We run jobs that may take > > up to a week to finish, and they are usually run on a cluster. Our > > setup relies on the following behavior for a script. > > > > 1. you can run the script anytime you want, and it can tell the state > > of the calculation by some means. If the script has never been run > > before, it submits the job to a queue and exits. If the job is still > > in the queue, it exits, and if the job is done, it gives you the > > result. > > Returning immediately with whatever state the long-running computation > is in currently seems indeed to be a good solution. I think I will > setup something similar. Would you share your experience on the > following issues? > > - How do you interface such jobs from orgmode? With org-babel, do > you execute Python code, or do you run shell commands? > We just have code blocks in org-mode. They are usually python blocks, but we can also do shell, emacs-lisp, etc... Anything that can run a system command, and get the output will do. > > - Do you run your Emacs on the master node of the cluster? Or does your > setup involve running emacs on the machine you are working on and > talking to the cluster over the network? > Currently, we run emacs on the master node. Once upon a time I had a sophisticated ssh setup that would allow me to do this on a local machine, rsync the necessary files to the cluster, ssh some commands to run the jobs, and then when they were done to rsync the files back. It was pretty sweet, but I have stopped used and maintaining it. > > Cheers, > Christoph > --047d7ba977a09a4c8404f779416c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On S= at, Apr 19, 2014 at 5:49 PM, Christoph Groth <christoph@grothesque= .org> wrote:
The following message is a courtesy copy of = an article
that has been posted to gmane.emacs.orgmode as well.

Thank you, John, for your detailed reply.

> we routinely do this, in the following way. We run jobs that may take<= br> > up to a week to finish, and they are usually run on a cluster. Our
> setup relies on the following behavior for a script.
>
> 1. you can run the script anytime you want, and it can tell the state<= br> > of the calculation by some means. If the script has never been run
> before, it submits the job to a queue and exits. If the job is still > in the queue, it exits, and if the job is done, it gives you the
> result.

Returning immediately with whatever state the long-running computatio= n
is in currently seems indeed to be a good solution. =C2=A0I think I will setup something similar. =C2=A0Would you share your experience on the
following issues?

- How do you interface such jobs from orgmode? =C2=A0With org-babel, do
=C2=A0 you execute Python code, or do you run shell commands?

We just have code blocks in org-mode. They are usual= ly python blocks, but we can also do shell, emacs-lisp, etc... Anything tha= t can run a system command, and get the output will do.
=C2=A0

- Do you run your Emacs on the master node of the cluster? =C2=A0Or does yo= ur
=C2=A0 setup involve running emacs on the machine you are working on and =C2=A0 talking to the cluster over the network?

Currently, we run emacs on the master node. Once upon a time I had= a sophisticated ssh setup that would allow me to do this on a local machin= e, rsync the necessary files to the cluster, ssh some commands to run the j= obs, and then when they were done to rsync the files back. It was pretty sw= eet, but I have stopped used and maintaining it.
=C2=A0

Cheers,
Christoph

--047d7ba977a09a4c8404f779416c--