* Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
@ 2015-06-17 19:09 Xebar Saram
2015-06-17 19:19 ` William Denton
0 siblings, 1 reply; 13+ messages in thread
From: Xebar Saram @ 2015-06-17 19:09 UTC (permalink / raw)
To: org mode
[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]
Hi all
im not sure if this is absolutely the correct forum to raise this and if
not would be also happy to get input on where to persue this issue.
i recently dived into using orgmode, ESS, Babel etc to run code and im
really love it. The problem i have (and im not sure if its a org,Emacs or
ess issue) is that emacs sometimes (and mostly when dealing with R
processes involving HUGE databases) will just hang/freeze.
I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in
R studio all works well yet when i use the orgmode eval on R code blocks it
works well for small simple process but 90% of the time when dealing with
complex models and bug data (up to 256GB) it will just freeze emacs/ess.
sometimes i can C-c or C-g it and other times i need to physically kill
emacs.
here is an example of such process that hangs
**** lmer
run the lmer part regressing stage 2 pred Vs mean pm
#+BEGIN_SRC R :session Rorg :results none
m2.smooth = lme(pred.m2 ~ meanPM25,random = list(aodid= ~1 +
meanPM25),control=lmeControl(opt = "optim"), data= mod2 )
#correlate to see everything from mod2 and the mpm works
mod2[, pred.t31 := predict(m2.smooth)]
mod2[, resid := residuals(m2.smooth)]
print(summary(lm(pred.m2~pred.t31,data=mod2))$r.squared)
#+END_SRC
i usually issue org-babel-execute-subtree to eval several subsections under
a main header.
again i dont know if its an org mode isse perse but would love to hear from
people that have experience using R/org with big data/RAM and maybe point
me to where to raise these issues
best
Z
[-- Attachment #2: Type: text/html, Size: 2005 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-17 19:09 Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Xebar Saram
@ 2015-06-17 19:19 ` William Denton
2015-06-18 3:17 ` Charles C. Berry
0 siblings, 1 reply; 13+ messages in thread
From: William Denton @ 2015-06-17 19:19 UTC (permalink / raw)
To: Xebar Saram; +Cc: org mode
[-- Attachment #1: Type: TEXT/PLAIN, Size: 811 bytes --]
On 17 June 2015, Xebar Saram wrote:
> I do alot of modeling work that involves using huge datasets and run
> process intensive R processes (such as complex mixed models, Gamms etc). in
> R studio all works well yet when i use the orgmode eval on R code blocks it
> works well for small simple process but 90% of the time when dealing with
> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
> sometimes i can C-c or C-g it and other times i need to physically kill
> emacs.
I've been having the same problem for a while, but wasn't able to isolate it any
more than large data sets, lack of memory, and heavy CPU usage. Sometimes
everything hangs and I need to power cycle the computer. :(
Bill
--
William Denton ↔ Toronto, Canada ↔ https://www.miskatonic.org/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-17 19:19 ` William Denton
@ 2015-06-18 3:17 ` Charles C. Berry
2015-06-18 11:39 ` Xebar Saram
2015-06-18 12:34 ` Rainer M Krug
0 siblings, 2 replies; 13+ messages in thread
From: Charles C. Berry @ 2015-06-18 3:17 UTC (permalink / raw)
To: William Denton; +Cc: Xebar Saram, org mode
On Wed, 17 Jun 2015, William Denton wrote:
> On 17 June 2015, Xebar Saram wrote:
>
>> I do alot of modeling work that involves using huge datasets and run
>> process intensive R processes (such as complex mixed models, Gamms etc). in
>> R studio all works well yet when i use the orgmode eval on R code blocks it
>> works well for small simple process but 90% of the time when dealing with
>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>> sometimes i can C-c or C-g it and other times i need to physically kill
>> emacs.
>
> I've been having the same problem for a while, but wasn't able to isolate it
> any more than large data sets, lack of memory, and heavy CPU usage.
> Sometimes everything hangs and I need to power cycle the computer. :(
>
And you (both) have `ess-eval-visibly' set to nil, right?
I do statistical genomics, which can be compute intensive. Sometimes
processes need to run for a while, and I get impatient having to wait.
I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
org-mode.
Basically, ravel will export Org mode to a format that knitr (and the
like) can run - turning src blocks into `code chunks'. That allows me to
set the cache=TRUE chunk option, etc. I run knitr on the exported document
to initialize objects for long running computations or to produce a
finished report.
When I start a session, I run knitr in the R session, then all the cached
objects are loaded in and ready to use.
If I write a src block I know will take a long time to export, I export
from org mode to update the knitr document and re-knit it to refresh the
cache.
Mostly, I work in org-mode adding src blocks, revising existing ones, or
editing text and graphics.
If you decide to try ravel I recommend the `ravel-lang' branch[2] as that
will soon replace master.
HTH,
Chuck
[1] https://github.com/chasberry/orgmode-accessories
[2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 3:17 ` Charles C. Berry
@ 2015-06-18 11:39 ` Xebar Saram
2015-06-18 18:45 ` Charles C. Berry
2015-06-18 12:34 ` Rainer M Krug
1 sibling, 1 reply; 13+ messages in thread
From: Xebar Saram @ 2015-06-18 11:39 UTC (permalink / raw)
To: Charles C. Berry, org mode
[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]
Thx Chuck
this sounds great. could you perhaps point us to some documentation on
this, or perhpas consider sharing a detailed overview of your workflow?
this seems it could really fit my needs.
thx so much in advance
Z
On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry <ccberry@ucsd.edu> wrote:
> On Wed, 17 Jun 2015, William Denton wrote:
>
> On 17 June 2015, Xebar Saram wrote:
>>
>> I do alot of modeling work that involves using huge datasets and run
>>> process intensive R processes (such as complex mixed models, Gamms etc).
>>> in
>>> R studio all works well yet when i use the orgmode eval on R code blocks
>>> it
>>> works well for small simple process but 90% of the time when dealing with
>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>> emacs.
>>>
>>
>> I've been having the same problem for a while, but wasn't able to isolate
>> it any more than large data sets, lack of memory, and heavy CPU usage.
>> Sometimes everything hangs and I need to power cycle the computer. :(
>>
>>
> And you (both) have `ess-eval-visibly' set to nil, right?
>
> I do statistical genomics, which can be compute intensive. Sometimes
> processes need to run for a while, and I get impatient having to wait.
>
> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
> org-mode.
>
> Basically, ravel will export Org mode to a format that knitr (and the
> like) can run - turning src blocks into `code chunks'. That allows me to
> set the cache=TRUE chunk option, etc. I run knitr on the exported document
> to initialize objects for long running computations or to produce a
> finished report.
>
> When I start a session, I run knitr in the R session, then all the cached
> objects are loaded in and ready to use.
>
> If I write a src block I know will take a long time to export, I export
> from org mode to update the knitr document and re-knit it to refresh the
> cache.
>
> Mostly, I work in org-mode adding src blocks, revising existing ones, or
> editing text and graphics.
>
> If you decide to try ravel I recommend the `ravel-lang' branch[2] as that
> will soon replace master.
>
> HTH,
>
> Chuck
>
>
> [1] https://github.com/chasberry/orgmode-accessories
> [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
>
[-- Attachment #2: Type: text/html, Size: 3320 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 3:17 ` Charles C. Berry
2015-06-18 11:39 ` Xebar Saram
@ 2015-06-18 12:34 ` Rainer M Krug
2015-06-18 20:20 ` Charles C. Berry
2015-06-19 22:31 ` Andreas Leha
1 sibling, 2 replies; 13+ messages in thread
From: Rainer M Krug @ 2015-06-18 12:34 UTC (permalink / raw)
To: Charles C. Berry; +Cc: Xebar Saram, William Denton, org mode
[-- Attachment #1: Type: text/plain, Size: 4898 bytes --]
"Charles C. Berry" <ccberry@ucsd.edu> writes:
> On Wed, 17 Jun 2015, William Denton wrote:
>
>> On 17 June 2015, Xebar Saram wrote:
>>
>>> I do alot of modeling work that involves using huge datasets and run
>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>> works well for small simple process but 90% of the time when dealing with
>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>> emacs.
>>
>> I've been having the same problem for a while, but wasn't able to
>> isolate it any more than large data sets, lack of memory, and heavy
>> CPU usage. Sometimes everything hangs and I need to power cycle the
>> computer. :(
>>
>
> And you (both) have `ess-eval-visibly' set to nil, right?
>
> I do statistical genomics, which can be compute intensive. Sometimes
> processes need to run for a while, and I get impatient having to wait.
>
> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
> org-mode.
>
> Basically, ravel will export Org mode to a format that knitr (and the
> like) can run - turning src blocks into `code chunks'. That allows me
> to set the cache=TRUE chunk option, etc. I run knitr on the exported
> document to initialize objects for long running computations or to
> produce a finished report.
>
> When I start a session, I run knitr in the R session, then all the
> cached objects are loaded in and ready to use.
>
> If I write a src block I know will take a long time to export, I
> export from org mode to update the knitr document and re-knit it to
> refresh the cache.
I have a similar workflow, only that I use a package like
approach, i.e. I tangle function definitions in a folder ./R, data into
./data (which makes it possible to share org defined variables with R
running outside org) and scripts, i.e. the things which do a analysis,
import data, ... i.e. which might take long, into a folder ./scripts/. I
then add the usual R package infrastructure files (DESCRIPTION,
NAMESPACE, ...).
Then I have one file tangled into ./scripts/init.R:
#+begin_src R :tangle ./scripts/init.R
library(devtools)
load_all()
#+end_src
and one for the analysis:
#+begin_src R :tangle ./scripts/myAnalysis.R
## Do some really time intensive and horribly complicated and important
## stuff here
save(
fileNames,
bw,
cols,
labels,
fit,
dens,
gof,
gofPerProf,
file = "./cache/results.myAnalysis.rds"
)
#+end_src
Now after tangling, I have my code easily available in a new R session:
1) start R in the directory in which the DESCRIPTION file is,
2) run source("./scripts/init.R")
and I have all my functions and data available.
To run a analysis, I do
3) source("./scripts/myAnalysis.R")
and the results are saved in a file fn
To analyse the data further, I can then simply use
#+begin_src R :tangle ./scripts/myAnalysis.R
fitSing <- attach("./cache/results.myAnalysis.rds")
#+end_src
so they won't interfere with my environment in R.
I can finally remove the attached environment by doing
#+begin_src R :tangle ./scripts/myAnalysis.R
detach(
name = attr(fitSing, "name"),
character.only = TRUE
)
#+end_src
Through these caching and compartmentalizing, I can easily do some
things outside org and some inside, and easily combine all the data.
Further advantage: I can actually create the package and send it to
somebody for testing and review and it should run out of the box, as in
the DESCRIPTION file all dependencies are defined.
I am using this approach at the moment for a paper and which will also
result in a paper. By executing all the scripts, one will be able to do
import the raw data, do the analysis and create all graphs used in the
paper.
Hope this gives you another idea how one can handle long running
analysis in R in org,
Cheers,
Rainer
>
> Mostly, I work in org-mode adding src blocks, revising existing ones,
> or editing text and graphics.
>
> If you decide to try ravel I recommend the `ravel-lang' branch[2] as
> that will soon replace master.
>
> HTH,
>
> Chuck
>
>
> [1] https://github.com/chasberry/orgmode-accessories
> [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
>
--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa
Tel : +33 - (0)9 53 10 27 44
Cell: +33 - (0)6 85 62 59 98
Fax : +33 - (0)9 58 10 27 44
Fax (D): +49 - (0)3 21 21 25 22 44
email: Rainer@krugs.de
Skype: RMkrug
PGP: 0x0F52F982
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 11:39 ` Xebar Saram
@ 2015-06-18 18:45 ` Charles C. Berry
0 siblings, 0 replies; 13+ messages in thread
From: Charles C. Berry @ 2015-06-18 18:45 UTC (permalink / raw)
To: Xebar Saram; +Cc: org mode
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1906 bytes --]
On Thu, 18 Jun 2015, Xebar Saram wrote:
> Thx Chuck
>
> this sounds great. could you perhaps point us to some documentation on
> this, or perhpas consider sharing a detailed overview of your workflow?
> this seems it could really fit my needs.
Start with Sections 1-3 of ox-ravel.org for basic info.
There are a couple of examples on my github repo. Looking at the
ravel-lang branch[1], there are these files:
- example-1-Rnw.org :: contains an org mode translation of the
“example-1.Rnw’ (Sweave) file from the R distribution and
instructions on how to export it.
- knitr-minimal-rhtml.org :: contains the “knitr-minimal.Rhtml’ file
from the knitr demos page, but modified to *.org format.
- demos.org :: contains a variety of examples. It is best viewed
online using 'raw' (or by downloading and viewing it in Org mode),
as github formatting masks the #+BEGIN_EXAMPLE ... #+END_EXAMPLE
sections that show what the output should be.
As far as caching goes this file (named cache.org, say)
--8<---------------cut here---------------start------------->8---
* cache this chunk
#+ATTR_RAVEL: cache=TRUE
#+NAME: show-time
#+BEGIN_SRC R
firstTime <- date()
firstTime
#+END_SRC
--8<---------------cut here---------------end--------------->8---
when exported as ravel-latex (i.e. C-c C-e r l, if you have ox-ravel
up and running, see section 2 of ox-ravel.org) creates cache.Rnw with
one chunk for which caching is specified. Loading knitr and running
knit("cache.Rnw") in R produces cache.tex and 3 ./cache/show-time*
files. Rerunning cache.Rnw will not update those files even if
cache.Rnw is updated as long as the code in the show-time block is
unchanged. You can start a fresh session, run knit("cache.Rnw"), and
the value of firstTime will be loaded into it.
HTH,
Chuck
[1] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 12:34 ` Rainer M Krug
@ 2015-06-18 20:20 ` Charles C. Berry
2015-06-18 21:20 ` Andreas Leha
2015-06-19 22:31 ` Andreas Leha
1 sibling, 1 reply; 13+ messages in thread
From: Charles C. Berry @ 2015-06-18 20:20 UTC (permalink / raw)
To: Rainer M Krug; +Cc: Xebar Saram, William Denton, org mode
On Thu, 18 Jun 2015, Rainer M Krug wrote:
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>
[snip: ox-ravel and how it might solve OP's problem]
>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:
[snip: how and why to structure an analysis as an R package]
> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>
And by writing the paper in the form of a vignette that Sweave or knitr
can render you have an R package that when installed processes the data
and reproduces the paper in pdf format.
ox-ravel will produce that vignette from Org mode.
For example, the bioConductor package geneRxCluster [1] comes from an Org
mode document that contains the C and R code as src blocks and a subtree
with the vignette (Using geneRxCluster) that analyzes data, produces
graphics, etc. tangle-ing the src blocks and exporting the vignette
creates the package.
HTH,
Chuck
[1] http://www.bioconductor.org/packages/geneRxCluster
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 20:20 ` Charles C. Berry
@ 2015-06-18 21:20 ` Andreas Leha
2015-06-19 22:13 ` Charles C. Berry
0 siblings, 1 reply; 13+ messages in thread
From: Andreas Leha @ 2015-06-18 21:20 UTC (permalink / raw)
To: emacs-orgmode
Hi Chuck,
[snip: all context about workflows for R projects]
> For example, the bioConductor package geneRxCluster [1] comes from an
> Org mode document that contains the C and R code as src blocks and a
> subtree with the vignette (Using geneRxCluster) that analyzes data,
> produces graphics, etc. tangle-ing the src blocks and exporting the
> vignette creates the package.
>
I'd love to see that org document, but I do not seem to be able to find
it. Is it available anywhere?
Thanks,
Andreas
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 21:20 ` Andreas Leha
@ 2015-06-19 22:13 ` Charles C. Berry
2015-06-19 22:25 ` Andreas Leha
0 siblings, 1 reply; 13+ messages in thread
From: Charles C. Berry @ 2015-06-19 22:13 UTC (permalink / raw)
To: Andreas Leha; +Cc: emacs-orgmode
On Thu, 18 Jun 2015, Andreas Leha wrote:
> Hi Chuck,
>
> [snip: all context about workflows for R projects]
>
>> For example, the bioConductor package geneRxCluster [1] comes from an
>> Org mode document that contains the C and R code as src blocks and a
>> subtree with the vignette (Using geneRxCluster) that analyzes data,
>> produces graphics, etc. tangle-ing the src blocks and exporting the
>> vignette creates the package.
>>
>
> I'd love to see that org document, but I do not seem to be able to find
> it. Is it available anywhere?
Andreas,
It is now, or at least the relevant subset of it is at:
https://github.com/chasberry/geneRx/blob/master/Rpackage.org
It is best viewed in Org mode as there are internal links that github does
not honor (so you get `404 This is not...' messages if you click on them).
And it is best to use those internal links as the file was not designed to
be read from top to bottom - it has lots of snippets I used to develop and
check code along the way that are not in the actual package and I have
left them in place.
Best,
Chuck
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-19 22:13 ` Charles C. Berry
@ 2015-06-19 22:25 ` Andreas Leha
0 siblings, 0 replies; 13+ messages in thread
From: Andreas Leha @ 2015-06-19 22:25 UTC (permalink / raw)
To: emacs-orgmode
"Charles C. Berry" <ccberry@ucsd.edu> writes:
> On Thu, 18 Jun 2015, Andreas Leha wrote:
>
>> Hi Chuck,
>>
>> [snip: all context about workflows for R projects]
>>
>>> For example, the bioConductor package geneRxCluster [1] comes from an
>>> Org mode document that contains the C and R code as src blocks and a
>>> subtree with the vignette (Using geneRxCluster) that analyzes data,
>>> produces graphics, etc. tangle-ing the src blocks and exporting the
>>> vignette creates the package.
>>>
>>
>> I'd love to see that org document, but I do not seem to be able to find
>> it. Is it available anywhere?
>
> Andreas,
>
> It is now, or at least the relevant subset of it is at:
>
> https://github.com/chasberry/geneRx/blob/master/Rpackage.org
>
> It is best viewed in Org mode as there are internal links that github
> does not honor (so you get `404 This is not...' messages if you click
> on them).
>
> And it is best to use those internal links as the file was not
> designed to be read from top to bottom - it has lots of snippets I
> used to develop and check code along the way that are not in the
> actual package and I have left them in place.
>
That is awesome! Really appreciated! I will take a good look.
Thanks,
Andreas
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-18 12:34 ` Rainer M Krug
2015-06-18 20:20 ` Charles C. Berry
@ 2015-06-19 22:31 ` Andreas Leha
2015-06-20 15:05 ` Rainer M Krug
1 sibling, 1 reply; 13+ messages in thread
From: Andreas Leha @ 2015-06-19 22:31 UTC (permalink / raw)
To: emacs-orgmode
Hi Rainer,
Rainer M Krug <Rainer@krugs.de> writes:
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>
>> I do statistical genomics, which can be compute intensive. Sometimes
>> processes need to run for a while, and I get impatient having to wait.
>>
>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>> org-mode.
>>
>> Basically, ravel will export Org mode to a format that knitr (and the
>> like) can run - turning src blocks into `code chunks'. That allows me
>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>> document to initialize objects for long running computations or to
>> produce a finished report.
>>
>> When I start a session, I run knitr in the R session, then all the
>> cached objects are loaded in and ready to use.
>>
>> If I write a src block I know will take a long time to export, I
>> export from org mode to update the knitr document and re-knit it to
>> refresh the cache.
>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:
>
> #+begin_src R :tangle ./scripts/init.R
> library(devtools)
> load_all()
> #+end_src
>
>
> and one for the analysis:
>
> #+begin_src R :tangle ./scripts/myAnalysis.R
> ## Do some really time intensive and horribly complicated and important
> ## stuff here
> save(
> fileNames,
> bw,
> cols,
> labels,
> fit,
> dens,
> gof,
> gofPerProf,
> file = "./cache/results.myAnalysis.rds"
> )
> #+end_src
>
>
> Now after tangling, I have my code easily available in a new R session:
>
> 1) start R in the directory in which the DESCRIPTION file is,
> 2) run source("./scripts/init.R")
>
> and I have all my functions and data available.
>
> To run a analysis, I do
>
> 3) source("./scripts/myAnalysis.R")
>
> and the results are saved in a file fn
>
> To analyse the data further, I can then simply use
>
> #+begin_src R :tangle ./scripts/myAnalysis.R
> fitSing <- attach("./cache/results.myAnalysis.rds")
> #+end_src
>
>
> so they won't interfere with my environment in R.
>
> I can finally remove the attached environment by doing
>
> #+begin_src R :tangle ./scripts/myAnalysis.R
> detach(
> name = attr(fitSing, "name"),
> character.only = TRUE
> )
> #+end_src
>
> Through these caching and compartmentalizing, I can easily do some
> things outside org and some inside, and easily combine all the data.
>
> Further advantage: I can actually create the package and send it to
> somebody for testing and review and it should run out of the box, as in
> the DESCRIPTION file all dependencies are defined.
>
> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>
> Hope this gives you another idea how one can handle long running
> analysis in R in org,
>
> Cheers,
>
> Rainer
>
That is a cool workflow. I especially like the fact that you end up
with an R package.
So, I'll try my again. Is there there any chance to see working
example of this? I'd love to see that.
Thanks,
Andreas
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-19 22:31 ` Andreas Leha
@ 2015-06-20 15:05 ` Rainer M Krug
2015-06-20 21:20 ` Andreas Leha
0 siblings, 1 reply; 13+ messages in thread
From: Rainer M Krug @ 2015-06-20 15:05 UTC (permalink / raw)
To: Andreas Leha; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 5638 bytes --]
Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
> Hi Rainer,
Hi Andreas,
>
> Rainer M Krug <Rainer@krugs.de> writes:
>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>>
>>> On Wed, 17 Jun 2015, William Denton wrote:
>>>
>>>> On 17 June 2015, Xebar Saram wrote:
>>>>
>>>>> I do alot of modeling work that involves using huge datasets and run
>>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>>> works well for small simple process but 90% of the time when dealing with
>>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>>> emacs.
>>>>
>>>> I've been having the same problem for a while, but wasn't able to
>>>> isolate it any more than large data sets, lack of memory, and heavy
>>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>>> computer. :(
>>>>
>>>
>>> And you (both) have `ess-eval-visibly' set to nil, right?
>>>
>>> I do statistical genomics, which can be compute intensive. Sometimes
>>> processes need to run for a while, and I get impatient having to wait.
>>>
>>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>>> org-mode.
>>>
>>> Basically, ravel will export Org mode to a format that knitr (and the
>>> like) can run - turning src blocks into `code chunks'. That allows me
>>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>>> document to initialize objects for long running computations or to
>>> produce a finished report.
>>>
>>> When I start a session, I run knitr in the R session, then all the
>>> cached objects are loaded in and ready to use.
>>>
>>> If I write a src block I know will take a long time to export, I
>>> export from org mode to update the knitr document and re-knit it to
>>> refresh the cache.
>>
>> I have a similar workflow, only that I use a package like
>> approach, i.e. I tangle function definitions in a folder ./R, data into
>> ./data (which makes it possible to share org defined variables with R
>> running outside org) and scripts, i.e. the things which do a analysis,
>> import data, ... i.e. which might take long, into a folder ./scripts/. I
>> then add the usual R package infrastructure files (DESCRIPTION,
>> NAMESPACE, ...).
>> Then I have one file tangled into ./scripts/init.R:
>>
>> #+begin_src R :tangle ./scripts/init.R
>> library(devtools)
>> load_all()
>> #+end_src
>>
>>
>> and one for the analysis:
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R
>> ## Do some really time intensive and horribly complicated and important
>> ## stuff here
>> save(
>> fileNames,
>> bw,
>> cols,
>> labels,
>> fit,
>> dens,
>> gof,
>> gofPerProf,
>> file = "./cache/results.myAnalysis.rds"
>> )
>> #+end_src
>>
>>
>> Now after tangling, I have my code easily available in a new R session:
>>
>> 1) start R in the directory in which the DESCRIPTION file is,
>> 2) run source("./scripts/init.R")
>>
>> and I have all my functions and data available.
>>
>> To run a analysis, I do
>>
>> 3) source("./scripts/myAnalysis.R")
>>
>> and the results are saved in a file fn
>>
>> To analyse the data further, I can then simply use
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R
>> fitSing <- attach("./cache/results.myAnalysis.rds")
>> #+end_src
>>
>>
>> so they won't interfere with my environment in R.
>>
>> I can finally remove the attached environment by doing
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R
>> detach(
>> name = attr(fitSing, "name"),
>> character.only = TRUE
>> )
>> #+end_src
>>
>> Through these caching and compartmentalizing, I can easily do some
>> things outside org and some inside, and easily combine all the data.
>>
>> Further advantage: I can actually create the package and send it to
>> somebody for testing and review and it should run out of the box, as in
>> the DESCRIPTION file all dependencies are defined.
>>
>> I am using this approach at the moment for a paper and which will also
>> result in a paper. By executing all the scripts, one will be able to do
>> import the raw data, do the analysis and create all graphs used in the
>> paper.
>>
>> Hope this gives you another idea how one can handle long running
>> analysis in R in org,
>>
>> Cheers,
>>
>> Rainer
>>
>
> That is a cool workflow. I especially like the fact that you end up
> with an R package.
Thanks. Yes - the idea of having a package at the end was one main
reason why I am using this approach.
>
> So, I'll try my again. Is there there any chance to see working
> example of this? I'd love to see that.
Let's say I am working on it. I am working on a project which is using
this workflow and when it is finished, the package will be available as
an electronic appendix to the paper.
But I will see if I can condense an example and blog it - I'll let you
kow when it is done.
Cheers,
Rainer
>
> Thanks,
> Andreas
>
>
--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa
Tel : +33 - (0)9 53 10 27 44
Cell: +33 - (0)6 85 62 59 98
Fax : +33 - (0)9 58 10 27 44
Fax (D): +49 - (0)3 21 21 25 22 44
email: Rainer@krugs.de
Skype: RMkrug
PGP: 0x0F52F982
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
2015-06-20 15:05 ` Rainer M Krug
@ 2015-06-20 21:20 ` Andreas Leha
0 siblings, 0 replies; 13+ messages in thread
From: Andreas Leha @ 2015-06-20 21:20 UTC (permalink / raw)
To: emacs-orgmode
Rainer M Krug <Rainer@krugs.de> writes:
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> Hi Rainer,
>
> Hi Andreas,
>
>>
>> Rainer M Krug <Rainer@krugs.de> writes:
>>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>>>
>>>> On Wed, 17 Jun 2015, William Denton wrote:
>>>>
>>>>> On 17 June 2015, Xebar Saram wrote:
>>>>>
>>>>>> I do alot of modeling work that involves using huge datasets and run
>>>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>>>> works well for small simple process but 90% of the time when dealing with
>>>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>>>> emacs.
>>>>>
>>>>> I've been having the same problem for a while, but wasn't able to
>>>>> isolate it any more than large data sets, lack of memory, and heavy
>>>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>>>> computer. :(
>>>>>
>>>>
>>>> And you (both) have `ess-eval-visibly' set to nil, right?
>>>>
>>>> I do statistical genomics, which can be compute intensive. Sometimes
>>>> processes need to run for a while, and I get impatient having to wait.
>>>>
>>>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>>>> org-mode.
>>>>
>>>> Basically, ravel will export Org mode to a format that knitr (and the
>>>> like) can run - turning src blocks into `code chunks'. That allows me
>>>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>>>> document to initialize objects for long running computations or to
>>>> produce a finished report.
>>>>
>>>> When I start a session, I run knitr in the R session, then all the
>>>> cached objects are loaded in and ready to use.
>>>>
>>>> If I write a src block I know will take a long time to export, I
>>>> export from org mode to update the knitr document and re-knit it to
>>>> refresh the cache.
>>>
>>> I have a similar workflow, only that I use a package like
>>> approach, i.e. I tangle function definitions in a folder ./R, data into
>>> ./data (which makes it possible to share org defined variables with R
>>> running outside org) and scripts, i.e. the things which do a analysis,
>>> import data, ... i.e. which might take long, into a folder ./scripts/. I
>>> then add the usual R package infrastructure files (DESCRIPTION,
>>> NAMESPACE, ...).
>>> Then I have one file tangled into ./scripts/init.R:
>>>
>>> #+begin_src R :tangle ./scripts/init.R
>>> library(devtools)
>>> load_all()
>>> #+end_src
>>>
>>>
>>> and one for the analysis:
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R
>>> ## Do some really time intensive and horribly complicated and important
>>> ## stuff here
>>> save(
>>> fileNames,
>>> bw,
>>> cols,
>>> labels,
>>> fit,
>>> dens,
>>> gof,
>>> gofPerProf,
>>> file = "./cache/results.myAnalysis.rds"
>>> )
>>> #+end_src
>>>
>>>
>>> Now after tangling, I have my code easily available in a new R session:
>>>
>>> 1) start R in the directory in which the DESCRIPTION file is,
>>> 2) run source("./scripts/init.R")
>>>
>>> and I have all my functions and data available.
>>>
>>> To run a analysis, I do
>>>
>>> 3) source("./scripts/myAnalysis.R")
>>>
>>> and the results are saved in a file fn
>>>
>>> To analyse the data further, I can then simply use
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R
>>> fitSing <- attach("./cache/results.myAnalysis.rds")
>>> #+end_src
>>>
>>>
>>> so they won't interfere with my environment in R.
>>>
>>> I can finally remove the attached environment by doing
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R
>>> detach(
>>> name = attr(fitSing, "name"),
>>> character.only = TRUE
>>> )
>>> #+end_src
>>>
>>> Through these caching and compartmentalizing, I can easily do some
>>> things outside org and some inside, and easily combine all the data.
>>>
>>> Further advantage: I can actually create the package and send it to
>>> somebody for testing and review and it should run out of the box, as in
>>> the DESCRIPTION file all dependencies are defined.
>>>
>>> I am using this approach at the moment for a paper and which will also
>>> result in a paper. By executing all the scripts, one will be able to do
>>> import the raw data, do the analysis and create all graphs used in the
>>> paper.
>>>
>>> Hope this gives you another idea how one can handle long running
>>> analysis in R in org,
>>>
>>> Cheers,
>>>
>>> Rainer
>>>
>>
>> That is a cool workflow. I especially like the fact that you end up
>> with an R package.
>
> Thanks. Yes - the idea of having a package at the end was one main
> reason why I am using this approach.
>
>
>>
>> So, I'll try my again. Is there there any chance to see working
>> example of this? I'd love to see that.
>
> Let's say I am working on it. I am working on a project which is using
> this workflow and when it is finished, the package will be available as
> an electronic appendix to the paper.
>
> But I will see if I can condense an example and blog it - I'll let you
> kow when it is done.
>
Thanks! Either way, I am really looking forward to this.
Regards,
Andreas
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-06-20 21:20 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-17 19:09 Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Xebar Saram
2015-06-17 19:19 ` William Denton
2015-06-18 3:17 ` Charles C. Berry
2015-06-18 11:39 ` Xebar Saram
2015-06-18 18:45 ` Charles C. Berry
2015-06-18 12:34 ` Rainer M Krug
2015-06-18 20:20 ` Charles C. Berry
2015-06-18 21:20 ` Andreas Leha
2015-06-19 22:13 ` Charles C. Berry
2015-06-19 22:25 ` Andreas Leha
2015-06-19 22:31 ` Andreas Leha
2015-06-20 15:05 ` Rainer M Krug
2015-06-20 21:20 ` Andreas Leha
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).