Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

["Charles C. Berry" <ccberry@ucsd.edu>]

On Thu, 18 Jun 2015, Rainer M Krug wrote:

> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>

[snip: ox-ravel and how it might solve OP's problem]

>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:

[snip: how and why to structure an analysis as an R package]


> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>

And by writing the paper in the form of a vignette that Sweave or knitr 
can render you have an R package that when installed processes the data 
and reproduces the paper in pdf format.

ox-ravel will produce that vignette from Org mode.

For example, the bioConductor package geneRxCluster [1] comes from an Org 
mode document that contains the C and R code as src blocks and a subtree 
with the vignette (Using geneRxCluster) that analyzes data, produces 
graphics, etc. tangle-ing the src blocks and exporting the vignette 
creates the package.

HTH,

Chuck

[1] http://www.bioconductor.org/packages/geneRxCluster