From: "Charles C. Berry" <ccberry@ucsd.edu>
To: Rainer M Krug <Rainer@krugs.de>
Cc: Xebar Saram <zeltakc@gmail.com>, William Denton <wtd@pobox.com>,
org mode <emacs-orgmode@gnu.org>
Subject: Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
Date: Thu, 18 Jun 2015 13:20:18 -0700 [thread overview]
Message-ID: <alpine.OSX.2.11.1506181243570.694@charles-berrys-macbook.local> (raw)
In-Reply-To: <m2lhfhyz3c.fsf@krugs.de>
On Thu, 18 Jun 2015, Rainer M Krug wrote:
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>
[snip: ox-ravel and how it might solve OP's problem]
>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:
[snip: how and why to structure an analysis as an R package]
> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>
And by writing the paper in the form of a vignette that Sweave or knitr
can render you have an R package that when installed processes the data
and reproduces the paper in pdf format.
ox-ravel will produce that vignette from Org mode.
For example, the bioConductor package geneRxCluster [1] comes from an Org
mode document that contains the C and R code as src blocks and a subtree
with the vignette (Using geneRxCluster) that analyzes data, produces
graphics, etc. tangle-ing the src blocks and exporting the vignette
creates the package.
HTH,
Chuck
[1] http://www.bioconductor.org/packages/geneRxCluster
next prev parent reply other threads:[~2015-06-18 20:20 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-17 19:09 Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Xebar Saram
2015-06-17 19:19 ` William Denton
2015-06-18 3:17 ` Charles C. Berry
2015-06-18 11:39 ` Xebar Saram
2015-06-18 18:45 ` Charles C. Berry
2015-06-18 12:34 ` Rainer M Krug
2015-06-18 20:20 ` Charles C. Berry [this message]
2015-06-18 21:20 ` Andreas Leha
2015-06-19 22:13 ` Charles C. Berry
2015-06-19 22:25 ` Andreas Leha
2015-06-19 22:31 ` Andreas Leha
2015-06-20 15:05 ` Rainer M Krug
2015-06-20 21:20 ` Andreas Leha
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.OSX.2.11.1506181243570.694@charles-berrys-macbook.local \
--to=ccberry@ucsd.edu \
--cc=Rainer@krugs.de \
--cc=emacs-orgmode@gnu.org \
--cc=wtd@pobox.com \
--cc=zeltakc@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).