Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Rainer M Krug <Rainer@krugs.de>]

[-- Attachment #1: Type: text/plain, Size: 4898 bytes --]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Wed, 17 Jun 2015, William Denton wrote:
>
>> On 17 June 2015, Xebar Saram wrote:
>>
>>> I do alot of modeling work that involves using huge datasets and run
>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>> works well for small simple process but 90% of the time when dealing with
>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>> emacs.
>>
>> I've been having the same problem for a while, but wasn't able to
>> isolate it any more than large data sets, lack of memory, and heavy
>> CPU usage. Sometimes everything hangs and I need to power cycle the
>> computer. :(
>>
>
> And you (both) have `ess-eval-visibly' set to nil, right?
>
> I do statistical genomics, which can be compute intensive. Sometimes
> processes need to run for a while, and I get impatient having to wait.
>
> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
> org-mode.
>
> Basically, ravel will export Org mode to a format that knitr (and the
> like) can run - turning src blocks into `code chunks'. That allows me
> to set the cache=TRUE chunk option, etc. I run knitr on the exported
> document to initialize objects for long running computations or to
> produce a finished report.
>
> When I start a session, I run knitr in the R session, then all the
> cached objects are loaded in and ready to use.
>
> If I write a src block I know will take a long time to export, I
> export from org mode to update the knitr document and re-knit it to
> refresh the cache.

I have a similar workflow, only that I use a package like
approach, i.e. I tangle function definitions in a folder ./R, data into
./data (which makes it possible to share org defined variables with R
running outside org) and scripts, i.e. the things which do a analysis,
import data, ... i.e. which might take long, into a folder ./scripts/. I
then add the usual R package infrastructure files (DESCRIPTION,
NAMESPACE, ...).
Then I have one file tangled into ./scripts/init.R:

#+begin_src R :tangle ./scripts/init.R  
library(devtools)
load_all()
#+end_src

and one for the analysis:

#+begin_src R :tangle ./scripts/myAnalysis.R  
## Do some really time intensive and horribly complicated and important
## stuff here
save(
    fileNames,
    bw,
    cols,
    labels,
    fit,
    dens,
    gof,
    gofPerProf,
    file = "./cache/results.myAnalysis.rds"
)
#+end_src

Now after tangling, I have my code easily available in a new R session:

1) start R in the directory in which the DESCRIPTION file is, 
2) run source("./scripts/init.R")

and I have all my functions and data available.

To run a analysis, I do

3) source("./scripts/myAnalysis.R")

and the results are saved in a file fn

To analyse the data further, I can then simply use

#+begin_src R :tangle ./scripts/myAnalysis.R
fitSing <- attach("./cache/results.myAnalysis.rds")
#+end_src

so they won't interfere with my environment in R.

I can finally remove the attached environment by doing

#+begin_src R :tangle ./scripts/myAnalysis.R  
detach(
    name = attr(fitSing, "name"),
    character.only = TRUE
)
#+end_src

Through these caching and compartmentalizing, I can easily do some
things outside org and some inside, and easily combine all the data.

Further advantage: I can actually create the package and send it to
somebody for testing and review and it should run out of the box, as in
the DESCRIPTION file all dependencies are defined.

I am using this approach at the moment for a paper and which will also
result in a paper. By executing all the scripts, one will be able to do
import the raw data, do the analysis and create all graphs used in the
paper.

Hope this gives you another idea how one can handle long running
analysis in R in org,

Cheers,

Rainer

>
> Mostly, I work in org-mode adding src blocks, revising existing ones,
> or editing text and graphics.
>
> If you decide to try ravel I recommend the `ravel-lang' branch[2] as
> that will soon replace master.
>
> HTH,
>
> Chuck
>
>
> [1] https://github.com/chasberry/orgmode-accessories
> [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]