From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xebar Saram Subject: Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Date: Thu, 18 Jun 2015 14:39:18 +0300 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7b33d7f0fedd6c0518c9416b Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37062) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5YAD-0003wx-7s for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 07:39:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z5YAB-0003tV-PR for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 07:39:21 -0400 Received: from mail-pa0-x229.google.com ([2607:f8b0:400e:c03::229]:36707) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5YAB-0003tF-BA for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 07:39:19 -0400 Received: by paceq1 with SMTP id eq1so35254760pac.3 for ; Thu, 18 Jun 2015 04:39:18 -0700 (PDT) In-Reply-To: List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: "Charles C. Berry" , org mode --047d7b33d7f0fedd6c0518c9416b Content-Type: text/plain; charset=UTF-8 Thx Chuck this sounds great. could you perhaps point us to some documentation on this, or perhpas consider sharing a detailed overview of your workflow? this seems it could really fit my needs. thx so much in advance Z On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry wrote: > On Wed, 17 Jun 2015, William Denton wrote: > > On 17 June 2015, Xebar Saram wrote: >> >> I do alot of modeling work that involves using huge datasets and run >>> process intensive R processes (such as complex mixed models, Gamms etc). >>> in >>> R studio all works well yet when i use the orgmode eval on R code blocks >>> it >>> works well for small simple process but 90% of the time when dealing with >>> complex models and bug data (up to 256GB) it will just freeze emacs/ess. >>> sometimes i can C-c or C-g it and other times i need to physically kill >>> emacs. >>> >> >> I've been having the same problem for a while, but wasn't able to isolate >> it any more than large data sets, lack of memory, and heavy CPU usage. >> Sometimes everything hangs and I need to power cycle the computer. :( >> >> > And you (both) have `ess-eval-visibly' set to nil, right? > > I do statistical genomics, which can be compute intensive. Sometimes > processes need to run for a while, and I get impatient having to wait. > > I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in > org-mode. > > Basically, ravel will export Org mode to a format that knitr (and the > like) can run - turning src blocks into `code chunks'. That allows me to > set the cache=TRUE chunk option, etc. I run knitr on the exported document > to initialize objects for long running computations or to produce a > finished report. > > When I start a session, I run knitr in the R session, then all the cached > objects are loaded in and ready to use. > > If I write a src block I know will take a long time to export, I export > from org mode to update the knitr document and re-knit it to refresh the > cache. > > Mostly, I work in org-mode adding src blocks, revising existing ones, or > editing text and graphics. > > If you decide to try ravel I recommend the `ravel-lang' branch[2] as that > will soon replace master. > > HTH, > > Chuck > > > [1] https://github.com/chasberry/orgmode-accessories > [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang > --047d7b33d7f0fedd6c0518c9416b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thx Chuck

this sounds great. could you = perhaps point us to some documentation on this, or perhpas consider sharing= a detailed overview of your workflow? this seems it could really fit my ne= eds.

thx so much in advance

Z


On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry <ccberry@= ucsd.edu> wrote:
On Wed, 17 Jun 2015, William Denton wrote:<= br>
On 17 June 2015, Xebar Saram wrote:

I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in=
R studio all works well yet when i use the orgmode eval on R code blocks it=
works well for small simple process but 90% of the time when dealing with complex models and bug data (up to 256GB) it will just freeze emacs/ess. sometimes i can C-c or C-g it and other times i need to physically kill
emacs.

I've been having the same problem for a while, but wasn't able to i= solate it any more than large data sets, lack of memory, and heavy CPU usag= e. Sometimes everything hangs and I need to power cycle the computer. :(

And you (both) have `ess-eval-visibly' set to nil, right?

I do statistical genomics, which can be compute intensive. Sometimes proces= ses need to run for a while, and I get impatient having to wait.

I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in org-= mode.

Basically, ravel will export Org mode to a format that knitr (and the like)= can run - turning src blocks into `code chunks'. That allows me to set= the cache=3DTRUE chunk option, etc. I run knitr on the exported document t= o initialize objects for long running computations or to produce a finished= report.

When I start a session, I run knitr in the R session, then all the cached o= bjects are loaded in and ready to use.

If I write a src block I know will take a long time to export, I export fro= m org mode to update the knitr document and re-knit it to refresh the cache= .

Mostly, I work in org-mode adding src blocks, revising existing ones, or ed= iting text and graphics.

If you decide to try ravel I recommend the `ravel-lang' branch[2] as th= at will soon replace master.

HTH,

Chuck


[1] https://github.com/chasberry/orgmode-accessories=
[2] https://github.com/chasberry/org= mode-accessories/tree/ravel-lang

--047d7b33d7f0fedd6c0518c9416b--