emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed

* Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
@ 2015-06-17 19:09 Xebar Saram
  2015-06-17 19:19 ` William Denton
  0 siblings, 1 reply; 13+ messages in thread
From: Xebar Saram @ 2015-06-17 19:09 UTC (permalink / raw)
  To: org mode

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

Hi all

im not sure if this is absolutely the correct forum to raise this and if
not would be also happy to get input on where to persue this issue.

i recently dived into using orgmode, ESS, Babel etc to run code and im
really love it. The problem i have (and im not sure if its a org,Emacs or
ess issue) is that emacs sometimes (and mostly when dealing with R
processes involving HUGE databases) will just hang/freeze.

I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in
R studio all works well yet when i use the orgmode eval on R code blocks it
works well for small simple process but 90% of the time when dealing with
complex models and bug data (up to 256GB) it will just freeze emacs/ess.
sometimes i can C-c or C-g it and other times i need to physically kill
emacs.

here is an example of such process that hangs


**** lmer
run the lmer part regressing stage 2 pred Vs mean pm

#+BEGIN_SRC R  :session Rorg  :results none
m2.smooth = lme(pred.m2 ~ meanPM25,random = list(aodid= ~1 +
meanPM25),control=lmeControl(opt = "optim"), data= mod2 )
#correlate to see everything from mod2 and the mpm works
mod2[, pred.t31 := predict(m2.smooth)]
mod2[, resid  := residuals(m2.smooth)]
print(summary(lm(pred.m2~pred.t31,data=mod2))$r.squared)
#+END_SRC

i usually issue org-babel-execute-subtree to eval several subsections under
a main header.

again i dont know if its an org mode isse perse but would love to hear from
people that have experience using R/org with big data/RAM and maybe point
me to where to raise these issues

best

Z

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

• * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
    2015-06-17 19:09 Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Xebar Saram
  @ 2015-06-17 19:19 ` William Denton
    2015-06-18  3:17   ` Charles C. Berry
    0 siblings, 1 reply; 13+ messages in thread
  From: William Denton @ 2015-06-17 19:19 UTC (permalink / raw)
    To: Xebar Saram; +Cc: org mode
  
  [-- Attachment #1: Type: TEXT/PLAIN, Size: 811 bytes --]
  
  On 17 June 2015, Xebar Saram wrote:
  
  > I do alot of modeling work that involves using huge datasets and run
  > process intensive R processes (such as complex mixed models, Gamms etc). in
  > R studio all works well yet when i use the orgmode eval on R code blocks it
  > works well for small simple process but 90% of the time when dealing with
  > complex models and bug data (up to 256GB) it will just freeze emacs/ess.
  > sometimes i can C-c or C-g it and other times i need to physically kill
  > emacs.
  
  I've been having the same problem for a while, but wasn't able to isolate it any 
  more than large data sets, lack of memory, and heavy CPU usage.  Sometimes 
  everything hangs and I need to power cycle the computer. :(
  
  Bill
  -- 
  William Denton ↔  Toronto, Canada ↔  https://www.miskatonic.org/
  
  ^ permalink raw reply	[flat|nested] 13+ messages in thread

• • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
      2015-06-17 19:19 ` William Denton
    @ 2015-06-18  3:17   ` Charles C. Berry
      2015-06-18 11:39     ` Xebar Saram
      2015-06-18 12:34     ` Rainer M Krug
      0 siblings, 2 replies; 13+ messages in thread
    From: Charles C. Berry @ 2015-06-18  3:17 UTC (permalink / raw)
      To: William Denton; +Cc: Xebar Saram, org mode
    
    On Wed, 17 Jun 2015, William Denton wrote:
    
    > On 17 June 2015, Xebar Saram wrote:
    >
    >> I do alot of modeling work that involves using huge datasets and run
    >> process intensive R processes (such as complex mixed models, Gamms etc). in
    >> R studio all works well yet when i use the orgmode eval on R code blocks it
    >> works well for small simple process but 90% of the time when dealing with
    >> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
    >> sometimes i can C-c or C-g it and other times i need to physically kill
    >> emacs.
    >
    > I've been having the same problem for a while, but wasn't able to isolate it 
    > any more than large data sets, lack of memory, and heavy CPU usage. 
    > Sometimes everything hangs and I need to power cycle the computer. :(
    >
    
    And you (both) have `ess-eval-visibly' set to nil, right?
    
    I do statistical genomics, which can be compute intensive. Sometimes 
    processes need to run for a while, and I get impatient having to wait.
    
    I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in 
    org-mode.
    
    Basically, ravel will export Org mode to a format that knitr (and the 
    like) can run - turning src blocks into `code chunks'. That allows me to 
    set the cache=TRUE chunk option, etc. I run knitr on the exported document 
    to initialize objects for long running computations or to produce a 
    finished report.
    
    When I start a session, I run knitr in the R session, then all the cached 
    objects are loaded in and ready to use.
    
    If I write a src block I know will take a long time to export, I export 
    from org mode to update the knitr document and re-knit it to refresh the 
    cache.
    
    Mostly, I work in org-mode adding src blocks, revising existing ones, or 
    editing text and graphics.
    
    If you decide to try ravel I recommend the `ravel-lang' branch[2] as that 
    will soon replace master.
    
    HTH,
    
    Chuck
    
    
    [1] https://github.com/chasberry/orgmode-accessories
    [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
    
    ^ permalink raw reply	[flat|nested] 13+ messages in thread

  • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
        2015-06-18  3:17   ` Charles C. Berry
      @ 2015-06-18 11:39     ` Xebar Saram
        2015-06-18 18:45       ` Charles C. Berry
        2015-06-18 12:34     ` Rainer M Krug
        1 sibling, 1 reply; 13+ messages in thread
      From: Xebar Saram @ 2015-06-18 11:39 UTC (permalink / raw)
        To: Charles C. Berry, org mode
      
      [-- Attachment #1: Type: text/plain, Size: 2361 bytes --]
      
      Thx Chuck
      
      this sounds great. could you perhaps point us to some documentation on
      this, or perhpas consider sharing a detailed overview of your workflow?
      this seems it could really fit my needs.
      
      thx so much in advance
      
      Z
      
      
      On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry <ccberry@ucsd.edu> wrote:
      
      > On Wed, 17 Jun 2015, William Denton wrote:
      >
      >  On 17 June 2015, Xebar Saram wrote:
      >>
      >>  I do alot of modeling work that involves using huge datasets and run
      >>> process intensive R processes (such as complex mixed models, Gamms etc).
      >>> in
      >>> R studio all works well yet when i use the orgmode eval on R code blocks
      >>> it
      >>> works well for small simple process but 90% of the time when dealing with
      >>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
      >>> sometimes i can C-c or C-g it and other times i need to physically kill
      >>> emacs.
      >>>
      >>
      >> I've been having the same problem for a while, but wasn't able to isolate
      >> it any more than large data sets, lack of memory, and heavy CPU usage.
      >> Sometimes everything hangs and I need to power cycle the computer. :(
      >>
      >>
      > And you (both) have `ess-eval-visibly' set to nil, right?
      >
      > I do statistical genomics, which can be compute intensive. Sometimes
      > processes need to run for a while, and I get impatient having to wait.
      >
      > I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
      > org-mode.
      >
      > Basically, ravel will export Org mode to a format that knitr (and the
      > like) can run - turning src blocks into `code chunks'. That allows me to
      > set the cache=TRUE chunk option, etc. I run knitr on the exported document
      > to initialize objects for long running computations or to produce a
      > finished report.
      >
      > When I start a session, I run knitr in the R session, then all the cached
      > objects are loaded in and ready to use.
      >
      > If I write a src block I know will take a long time to export, I export
      > from org mode to update the knitr document and re-knit it to refresh the
      > cache.
      >
      > Mostly, I work in org-mode adding src blocks, revising existing ones, or
      > editing text and graphics.
      >
      > If you decide to try ravel I recommend the `ravel-lang' branch[2] as that
      > will soon replace master.
      >
      > HTH,
      >
      > Chuck
      >
      >
      > [1] https://github.com/chasberry/orgmode-accessories
      > [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
      >
      
      [-- Attachment #2: Type: text/html, Size: 3320 bytes --]
      
      ^ permalink raw reply	[flat|nested] 13+ messages in thread

    • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
          2015-06-18 11:39     ` Xebar Saram
        @ 2015-06-18 18:45       ` Charles C. Berry
          0 siblings, 0 replies; 13+ messages in thread
        From: Charles C. Berry @ 2015-06-18 18:45 UTC (permalink / raw)
          To: Xebar Saram; +Cc: org mode
        
        [-- Attachment #1: Type: TEXT/PLAIN, Size: 1906 bytes --]
        
        On Thu, 18 Jun 2015, Xebar Saram wrote:
        
        > Thx Chuck
        >
        > this sounds great. could you perhaps point us to some documentation on
        > this, or perhpas consider sharing a detailed overview of your workflow?
        > this seems it could really fit my needs.
        
        Start with Sections 1-3 of ox-ravel.org for basic info.
        
        There are a couple of examples on my github repo. Looking at the 
        ravel-lang branch[1], there are these files:
        
        - example-1-Rnw.org :: contains an org mode translation of the
              “example-1.Rnw’ (Sweave) file from the R distribution and
              instructions on how to export it.
        
        
        - knitr-minimal-rhtml.org :: contains the “knitr-minimal.Rhtml’ file
              from the knitr demos page, but modified to *.org format.
        
        
        - demos.org :: contains a variety of examples. It is best viewed
              online using 'raw' (or by downloading and viewing it in Org mode),
              as github formatting masks the #+BEGIN_EXAMPLE ... #+END_EXAMPLE
              sections that show what the output should be.
        
        As far as caching goes this file (named cache.org, say)
        
        --8<---------------cut here---------------start------------->8---
        
        * cache this chunk
        
        #+ATTR_RAVEL: cache=TRUE
        #+NAME: show-time
        #+BEGIN_SRC R
           firstTime <- date()
           firstTime
        #+END_SRC
        
        --8<---------------cut here---------------end--------------->8---
        
        when exported as ravel-latex (i.e. C-c C-e r l, if you have ox-ravel
        up and running, see section 2 of ox-ravel.org) creates cache.Rnw with
        one chunk for which caching is specified. Loading knitr and running
        knit("cache.Rnw") in R produces cache.tex and 3 ./cache/show-time*
        files. Rerunning cache.Rnw will not update those files even if
        cache.Rnw is updated as long as the code in the show-time block is
        unchanged. You can start a fresh session, run knit("cache.Rnw"), and
        the value of firstTime will be loaded into it.
        
        HTH,
        
        Chuck
        
        [1] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

    • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
        2015-06-18  3:17   ` Charles C. Berry
        2015-06-18 11:39     ` Xebar Saram
      @ 2015-06-18 12:34     ` Rainer M Krug
        2015-06-18 20:20       ` Charles C. Berry
        2015-06-19 22:31       ` Andreas Leha
        1 sibling, 2 replies; 13+ messages in thread
      From: Rainer M Krug @ 2015-06-18 12:34 UTC (permalink / raw)
        To: Charles C. Berry; +Cc: Xebar Saram, William Denton, org mode
      
      [-- Attachment #1: Type: text/plain, Size: 4898 bytes --]
      
      "Charles C. Berry" <ccberry@ucsd.edu> writes:
      
      > On Wed, 17 Jun 2015, William Denton wrote:
      >
      >> On 17 June 2015, Xebar Saram wrote:
      >>
      >>> I do alot of modeling work that involves using huge datasets and run
      >>> process intensive R processes (such as complex mixed models, Gamms etc). in
      >>> R studio all works well yet when i use the orgmode eval on R code blocks it
      >>> works well for small simple process but 90% of the time when dealing with
      >>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
      >>> sometimes i can C-c or C-g it and other times i need to physically kill
      >>> emacs.
      >>
      >> I've been having the same problem for a while, but wasn't able to
      >> isolate it any more than large data sets, lack of memory, and heavy
      >> CPU usage. Sometimes everything hangs and I need to power cycle the
      >> computer. :(
      >>
      >
      > And you (both) have `ess-eval-visibly' set to nil, right?
      >
      > I do statistical genomics, which can be compute intensive. Sometimes
      > processes need to run for a while, and I get impatient having to wait.
      >
      > I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
      > org-mode.
      >
      > Basically, ravel will export Org mode to a format that knitr (and the
      > like) can run - turning src blocks into `code chunks'. That allows me
      > to set the cache=TRUE chunk option, etc. I run knitr on the exported
      > document to initialize objects for long running computations or to
      > produce a finished report.
      >
      > When I start a session, I run knitr in the R session, then all the
      > cached objects are loaded in and ready to use.
      >
      > If I write a src block I know will take a long time to export, I
      > export from org mode to update the knitr document and re-knit it to
      > refresh the cache.
      
      I have a similar workflow, only that I use a package like
      approach, i.e. I tangle function definitions in a folder ./R, data into
      ./data (which makes it possible to share org defined variables with R
      running outside org) and scripts, i.e. the things which do a analysis,
      import data, ... i.e. which might take long, into a folder ./scripts/. I
      then add the usual R package infrastructure files (DESCRIPTION,
      NAMESPACE, ...).
      Then I have one file tangled into ./scripts/init.R:
      
      #+begin_src R :tangle ./scripts/init.R  
      library(devtools)
      load_all()
      #+end_src
      
      and one for the analysis:
      
      #+begin_src R :tangle ./scripts/myAnalysis.R  
      ## Do some really time intensive and horribly complicated and important
      ## stuff here
      save(
          fileNames,
          bw,
          cols,
          labels,
          fit,
          dens,
          gof,
          gofPerProf,
          file = "./cache/results.myAnalysis.rds"
      )
      #+end_src
      
      Now after tangling, I have my code easily available in a new R session:
      
      1) start R in the directory in which the DESCRIPTION file is, 
      2) run source("./scripts/init.R")
      
      and I have all my functions and data available.
      
      To run a analysis, I do
      
      3) source("./scripts/myAnalysis.R")
      
      and the results are saved in a file fn
      
      To analyse the data further, I can then simply use
      
      #+begin_src R :tangle ./scripts/myAnalysis.R
      fitSing <- attach("./cache/results.myAnalysis.rds")
      #+end_src
      
      so they won't interfere with my environment in R.
      
      I can finally remove the attached environment by doing
      
      #+begin_src R :tangle ./scripts/myAnalysis.R  
      detach(
          name = attr(fitSing, "name"),
          character.only = TRUE
      )
      #+end_src
      
      Through these caching and compartmentalizing, I can easily do some
      things outside org and some inside, and easily combine all the data.
      
      Further advantage: I can actually create the package and send it to
      somebody for testing and review and it should run out of the box, as in
      the DESCRIPTION file all dependencies are defined.
      
      I am using this approach at the moment for a paper and which will also
      result in a paper. By executing all the scripts, one will be able to do
      import the raw data, do the analysis and create all graphs used in the
      paper.
      
      Hope this gives you another idea how one can handle long running
      analysis in R in org,
      
      Cheers,
      
      Rainer
      
      >
      > Mostly, I work in org-mode adding src blocks, revising existing ones,
      > or editing text and graphics.
      >
      > If you decide to try ravel I recommend the `ravel-lang' branch[2] as
      > that will soon replace master.
      >
      > HTH,
      >
      > Chuck
      >
      >
      > [1] https://github.com/chasberry/orgmode-accessories
      > [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
      >
      
      -- 
      Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
      
      Centre of Excellence for Invasion Biology
      Stellenbosch University
      South Africa
      
      Tel :       +33 - (0)9 53 10 27 44
      Cell:       +33 - (0)6 85 62 59 98
      Fax :       +33 - (0)9 58 10 27 44
      
      Fax (D):    +49 - (0)3 21 21 25 22 44
      
      email:      Rainer@krugs.de
      
      Skype:      RMkrug
      
      PGP: 0x0F52F982
      
      [-- Attachment #2: signature.asc --]
      [-- Type: application/pgp-signature, Size: 480 bytes --]
      
      ^ permalink raw reply	[flat|nested] 13+ messages in thread

    • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
          2015-06-18 12:34     ` Rainer M Krug
        @ 2015-06-18 20:20       ` Charles C. Berry
          2015-06-18 21:20         ` Andreas Leha
          2015-06-19 22:31       ` Andreas Leha
          1 sibling, 1 reply; 13+ messages in thread
        From: Charles C. Berry @ 2015-06-18 20:20 UTC (permalink / raw)
          To: Rainer M Krug; +Cc: Xebar Saram, William Denton, org mode
        
        On Thu, 18 Jun 2015, Rainer M Krug wrote:
        
        > "Charles C. Berry" <ccberry@ucsd.edu> writes:
        >
        >> On Wed, 17 Jun 2015, William Denton wrote:
        >>
        >>> On 17 June 2015, Xebar Saram wrote:
        >>>
        >>>> I do alot of modeling work that involves using huge datasets and run
        >>>> process intensive R processes (such as complex mixed models, Gamms etc). in
        >>>> R studio all works well yet when i use the orgmode eval on R code blocks it
        >>>> works well for small simple process but 90% of the time when dealing with
        >>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
        >>>> sometimes i can C-c or C-g it and other times i need to physically kill
        >>>> emacs.
        >>>
        >>> I've been having the same problem for a while, but wasn't able to
        >>> isolate it any more than large data sets, lack of memory, and heavy
        >>> CPU usage. Sometimes everything hangs and I need to power cycle the
        >>> computer. :(
        >>>
        >>
        >> And you (both) have `ess-eval-visibly' set to nil, right?
        >>
        
        [snip: ox-ravel and how it might solve OP's problem]
        
        >
        > I have a similar workflow, only that I use a package like
        > approach, i.e. I tangle function definitions in a folder ./R, data into
        > ./data (which makes it possible to share org defined variables with R
        > running outside org) and scripts, i.e. the things which do a analysis,
        > import data, ... i.e. which might take long, into a folder ./scripts/. I
        > then add the usual R package infrastructure files (DESCRIPTION,
        > NAMESPACE, ...).
        > Then I have one file tangled into ./scripts/init.R:
        
        [snip: how and why to structure an analysis as an R package]
        
        
        > I am using this approach at the moment for a paper and which will also
        > result in a paper. By executing all the scripts, one will be able to do
        > import the raw data, do the analysis and create all graphs used in the
        > paper.
        >
        
        And by writing the paper in the form of a vignette that Sweave or knitr 
        can render you have an R package that when installed processes the data 
        and reproduces the paper in pdf format.
        
        ox-ravel will produce that vignette from Org mode.
        
        For example, the bioConductor package geneRxCluster [1] comes from an Org 
        mode document that contains the C and R code as src blocks and a subtree 
        with the vignette (Using geneRxCluster) that analyzes data, produces 
        graphics, etc. tangle-ing the src blocks and exporting the vignette 
        creates the package.
        
        HTH,
        
        Chuck
        
        [1] http://www.bioconductor.org/packages/geneRxCluster
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
            2015-06-18 20:20       ` Charles C. Berry
          @ 2015-06-18 21:20         ` Andreas Leha
            2015-06-19 22:13           ` Charles C. Berry
            0 siblings, 1 reply; 13+ messages in thread
          From: Andreas Leha @ 2015-06-18 21:20 UTC (permalink / raw)
            To: emacs-orgmode
          
          Hi Chuck,
          
          [snip: all context about workflows for R projects]
          
          > For example, the bioConductor package geneRxCluster [1] comes from an
          > Org mode document that contains the C and R code as src blocks and a
          > subtree with the vignette (Using geneRxCluster) that analyzes data,
          > produces graphics, etc. tangle-ing the src blocks and exporting the
          > vignette creates the package.
          >
          
          I'd love to see that org document, but I do not seem to be able to find
          it.  Is it available anywhere?
          
          Thanks,
          Andreas
          
          ^ permalink raw reply	[flat|nested] 13+ messages in thread

        • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
              2015-06-18 21:20         ` Andreas Leha
            @ 2015-06-19 22:13           ` Charles C. Berry
              2015-06-19 22:25             ` Andreas Leha
              0 siblings, 1 reply; 13+ messages in thread
            From: Charles C. Berry @ 2015-06-19 22:13 UTC (permalink / raw)
              To: Andreas Leha; +Cc: emacs-orgmode
            
            On Thu, 18 Jun 2015, Andreas Leha wrote:
            
            > Hi Chuck,
            >
            > [snip: all context about workflows for R projects]
            >
            >> For example, the bioConductor package geneRxCluster [1] comes from an
            >> Org mode document that contains the C and R code as src blocks and a
            >> subtree with the vignette (Using geneRxCluster) that analyzes data,
            >> produces graphics, etc. tangle-ing the src blocks and exporting the
            >> vignette creates the package.
            >>
            >
            > I'd love to see that org document, but I do not seem to be able to find
            > it.  Is it available anywhere?
            
            Andreas,
            
            It is now, or at least the relevant subset of it is at:
            
            https://github.com/chasberry/geneRx/blob/master/Rpackage.org
            
            It is best viewed in Org mode as there are internal links that github does 
            not honor (so you get `404 This is not...' messages if you click on them).
            
            And it is best to use those internal links as the file was not designed to 
            be read from top to bottom - it has lots of snippets I used to develop and 
            check code along the way that are not in the actual package and I have 
            left them in place.
            
            Best,
            
            Chuck
            
            ^ permalink raw reply	[flat|nested] 13+ messages in thread

          • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
                2015-06-19 22:13           ` Charles C. Berry
              @ 2015-06-19 22:25             ` Andreas Leha
                0 siblings, 0 replies; 13+ messages in thread
              From: Andreas Leha @ 2015-06-19 22:25 UTC (permalink / raw)
                To: emacs-orgmode
              
              "Charles C. Berry" <ccberry@ucsd.edu> writes:
              > On Thu, 18 Jun 2015, Andreas Leha wrote:
              >
              >> Hi Chuck,
              >>
              >> [snip: all context about workflows for R projects]
              >>
              >>> For example, the bioConductor package geneRxCluster [1] comes from an
              >>> Org mode document that contains the C and R code as src blocks and a
              >>> subtree with the vignette (Using geneRxCluster) that analyzes data,
              >>> produces graphics, etc. tangle-ing the src blocks and exporting the
              >>> vignette creates the package.
              >>>
              >>
              >> I'd love to see that org document, but I do not seem to be able to find
              >> it.  Is it available anywhere?
              >
              > Andreas,
              >
              > It is now, or at least the relevant subset of it is at:
              >
              > https://github.com/chasberry/geneRx/blob/master/Rpackage.org
              >
              > It is best viewed in Org mode as there are internal links that github
              > does not honor (so you get `404 This is not...' messages if you click
              > on them).
              >
              > And it is best to use those internal links as the file was not
              > designed to be read from top to bottom - it has lots of snippets I
              > used to develop and check code along the way that are not in the
              > actual package and I have left them in place.
              >
              
              That is awesome!  Really appreciated!  I will take a good look.
              
              Thanks,
              Andreas
              
              ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
          2015-06-18 12:34     ` Rainer M Krug
          2015-06-18 20:20       ` Charles C. Berry
        @ 2015-06-19 22:31       ` Andreas Leha
          2015-06-20 15:05         ` Rainer M Krug
          1 sibling, 1 reply; 13+ messages in thread
        From: Andreas Leha @ 2015-06-19 22:31 UTC (permalink / raw)
          To: emacs-orgmode
        
        Hi Rainer,
        
        Rainer M Krug <Rainer@krugs.de> writes:
        > "Charles C. Berry" <ccberry@ucsd.edu> writes:
        >
        >> On Wed, 17 Jun 2015, William Denton wrote:
        >>
        >>> On 17 June 2015, Xebar Saram wrote:
        >>>
        >>>> I do alot of modeling work that involves using huge datasets and run
        >>>> process intensive R processes (such as complex mixed models, Gamms etc). in
        >>>> R studio all works well yet when i use the orgmode eval on R code blocks it
        >>>> works well for small simple process but 90% of the time when dealing with
        >>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
        >>>> sometimes i can C-c or C-g it and other times i need to physically kill
        >>>> emacs.
        >>>
        >>> I've been having the same problem for a while, but wasn't able to
        >>> isolate it any more than large data sets, lack of memory, and heavy
        >>> CPU usage. Sometimes everything hangs and I need to power cycle the
        >>> computer. :(
        >>>
        >>
        >> And you (both) have `ess-eval-visibly' set to nil, right?
        >>
        >> I do statistical genomics, which can be compute intensive. Sometimes
        >> processes need to run for a while, and I get impatient having to wait.
        >>
        >> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
        >> org-mode.
        >>
        >> Basically, ravel will export Org mode to a format that knitr (and the
        >> like) can run - turning src blocks into `code chunks'. That allows me
        >> to set the cache=TRUE chunk option, etc. I run knitr on the exported
        >> document to initialize objects for long running computations or to
        >> produce a finished report.
        >>
        >> When I start a session, I run knitr in the R session, then all the
        >> cached objects are loaded in and ready to use.
        >>
        >> If I write a src block I know will take a long time to export, I
        >> export from org mode to update the knitr document and re-knit it to
        >> refresh the cache.
        >
        > I have a similar workflow, only that I use a package like
        > approach, i.e. I tangle function definitions in a folder ./R, data into
        > ./data (which makes it possible to share org defined variables with R
        > running outside org) and scripts, i.e. the things which do a analysis,
        > import data, ... i.e. which might take long, into a folder ./scripts/. I
        > then add the usual R package infrastructure files (DESCRIPTION,
        > NAMESPACE, ...).
        > Then I have one file tangled into ./scripts/init.R:
        >
        > #+begin_src R :tangle ./scripts/init.R  
        > library(devtools)
        > load_all()
        > #+end_src
        >
        >
        > and one for the analysis:
        >
        > #+begin_src R :tangle ./scripts/myAnalysis.R  
        > ## Do some really time intensive and horribly complicated and important
        > ## stuff here
        > save(
        >     fileNames,
        >     bw,
        >     cols,
        >     labels,
        >     fit,
        >     dens,
        >     gof,
        >     gofPerProf,
        >     file = "./cache/results.myAnalysis.rds"
        > )
        > #+end_src
        >
        >
        > Now after tangling, I have my code easily available in a new R session:
        >
        > 1) start R in the directory in which the DESCRIPTION file is, 
        > 2) run source("./scripts/init.R")
        >
        > and I have all my functions and data available.
        >
        > To run a analysis, I do
        >
        > 3) source("./scripts/myAnalysis.R")
        >
        > and the results are saved in a file fn
        >
        > To analyse the data further, I can then simply use
        >
        > #+begin_src R :tangle ./scripts/myAnalysis.R
        > fitSing <- attach("./cache/results.myAnalysis.rds")
        > #+end_src
        >
        >
        > so they won't interfere with my environment in R.
        >
        > I can finally remove the attached environment by doing
        >
        > #+begin_src R :tangle ./scripts/myAnalysis.R  
        > detach(
        >     name = attr(fitSing, "name"),
        >     character.only = TRUE
        > )
        > #+end_src
        >
        > Through these caching and compartmentalizing, I can easily do some
        > things outside org and some inside, and easily combine all the data.
        >
        > Further advantage: I can actually create the package and send it to
        > somebody for testing and review and it should run out of the box, as in
        > the DESCRIPTION file all dependencies are defined.
        >
        > I am using this approach at the moment for a paper and which will also
        > result in a paper. By executing all the scripts, one will be able to do
        > import the raw data, do the analysis and create all graphs used in the
        > paper.
        >
        > Hope this gives you another idea how one can handle long running
        > analysis in R in org,
        >
        > Cheers,
        >
        > Rainer
        >
        
        That is a cool workflow.  I especially like the fact that you end up
        with an R package.
        
        So, I'll try my again.   Is there there any chance to see working
        example of this?  I'd love to see that.
        
        Thanks,
        Andreas
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
            2015-06-19 22:31       ` Andreas Leha
          @ 2015-06-20 15:05         ` Rainer M Krug
            2015-06-20 21:20           ` Andreas Leha
            0 siblings, 1 reply; 13+ messages in thread
          From: Rainer M Krug @ 2015-06-20 15:05 UTC (permalink / raw)
            To: Andreas Leha; +Cc: emacs-orgmode
          
          [-- Attachment #1: Type: text/plain, Size: 5638 bytes --]
          
          Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
          
          > Hi Rainer,
          
          Hi Andreas,
          
          >
          > Rainer M Krug <Rainer@krugs.de> writes:
          >> "Charles C. Berry" <ccberry@ucsd.edu> writes:
          >>
          >>> On Wed, 17 Jun 2015, William Denton wrote:
          >>>
          >>>> On 17 June 2015, Xebar Saram wrote:
          >>>>
          >>>>> I do alot of modeling work that involves using huge datasets and run
          >>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
          >>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
          >>>>> works well for small simple process but 90% of the time when dealing with
          >>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
          >>>>> sometimes i can C-c or C-g it and other times i need to physically kill
          >>>>> emacs.
          >>>>
          >>>> I've been having the same problem for a while, but wasn't able to
          >>>> isolate it any more than large data sets, lack of memory, and heavy
          >>>> CPU usage. Sometimes everything hangs and I need to power cycle the
          >>>> computer. :(
          >>>>
          >>>
          >>> And you (both) have `ess-eval-visibly' set to nil, right?
          >>>
          >>> I do statistical genomics, which can be compute intensive. Sometimes
          >>> processes need to run for a while, and I get impatient having to wait.
          >>>
          >>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
          >>> org-mode.
          >>>
          >>> Basically, ravel will export Org mode to a format that knitr (and the
          >>> like) can run - turning src blocks into `code chunks'. That allows me
          >>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
          >>> document to initialize objects for long running computations or to
          >>> produce a finished report.
          >>>
          >>> When I start a session, I run knitr in the R session, then all the
          >>> cached objects are loaded in and ready to use.
          >>>
          >>> If I write a src block I know will take a long time to export, I
          >>> export from org mode to update the knitr document and re-knit it to
          >>> refresh the cache.
          >>
          >> I have a similar workflow, only that I use a package like
          >> approach, i.e. I tangle function definitions in a folder ./R, data into
          >> ./data (which makes it possible to share org defined variables with R
          >> running outside org) and scripts, i.e. the things which do a analysis,
          >> import data, ... i.e. which might take long, into a folder ./scripts/. I
          >> then add the usual R package infrastructure files (DESCRIPTION,
          >> NAMESPACE, ...).
          >> Then I have one file tangled into ./scripts/init.R:
          >>
          >> #+begin_src R :tangle ./scripts/init.R  
          >> library(devtools)
          >> load_all()
          >> #+end_src
          >>
          >>
          >> and one for the analysis:
          >>
          >> #+begin_src R :tangle ./scripts/myAnalysis.R  
          >> ## Do some really time intensive and horribly complicated and important
          >> ## stuff here
          >> save(
          >>     fileNames,
          >>     bw,
          >>     cols,
          >>     labels,
          >>     fit,
          >>     dens,
          >>     gof,
          >>     gofPerProf,
          >>     file = "./cache/results.myAnalysis.rds"
          >> )
          >> #+end_src
          >>
          >>
          >> Now after tangling, I have my code easily available in a new R session:
          >>
          >> 1) start R in the directory in which the DESCRIPTION file is, 
          >> 2) run source("./scripts/init.R")
          >>
          >> and I have all my functions and data available.
          >>
          >> To run a analysis, I do
          >>
          >> 3) source("./scripts/myAnalysis.R")
          >>
          >> and the results are saved in a file fn
          >>
          >> To analyse the data further, I can then simply use
          >>
          >> #+begin_src R :tangle ./scripts/myAnalysis.R
          >> fitSing <- attach("./cache/results.myAnalysis.rds")
          >> #+end_src
          >>
          >>
          >> so they won't interfere with my environment in R.
          >>
          >> I can finally remove the attached environment by doing
          >>
          >> #+begin_src R :tangle ./scripts/myAnalysis.R  
          >> detach(
          >>     name = attr(fitSing, "name"),
          >>     character.only = TRUE
          >> )
          >> #+end_src
          >>
          >> Through these caching and compartmentalizing, I can easily do some
          >> things outside org and some inside, and easily combine all the data.
          >>
          >> Further advantage: I can actually create the package and send it to
          >> somebody for testing and review and it should run out of the box, as in
          >> the DESCRIPTION file all dependencies are defined.
          >>
          >> I am using this approach at the moment for a paper and which will also
          >> result in a paper. By executing all the scripts, one will be able to do
          >> import the raw data, do the analysis and create all graphs used in the
          >> paper.
          >>
          >> Hope this gives you another idea how one can handle long running
          >> analysis in R in org,
          >>
          >> Cheers,
          >>
          >> Rainer
          >>
          >
          > That is a cool workflow.  I especially like the fact that you end up
          > with an R package.
          
          Thanks. Yes - the idea of having a package at the end was one main
          reason why I am using this approach.
          
          
          >
          > So, I'll try my again.   Is there there any chance to see working
          > example of this?  I'd love to see that.
          
          Let's say I am working on it. I am working on a project which is using
          this workflow and when it is finished, the package will be available as
          an electronic appendix to the paper.
          
          But I will see if I can condense an example and blog it - I'll let you
          kow when it is done.
          
          Cheers,
          
          Rainer
          
          
          
          >
          > Thanks,
          > Andreas
          >
          >
          
          -- 
          Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
          
          Centre of Excellence for Invasion Biology
          Stellenbosch University
          South Africa
          
          Tel :       +33 - (0)9 53 10 27 44
          Cell:       +33 - (0)6 85 62 59 98
          Fax :       +33 - (0)9 58 10 27 44
          
          Fax (D):    +49 - (0)3 21 21 25 22 44
          
          email:      Rainer@krugs.de
          
          Skype:      RMkrug
          
          PGP: 0x0F52F982
          
          [-- Attachment #2: signature.asc --]
          [-- Type: application/pgp-signature, Size: 480 bytes --]
          
          ^ permalink raw reply	[flat|nested] 13+ messages in thread

        • • * Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
              2015-06-20 15:05         ` Rainer M Krug
            @ 2015-06-20 21:20           ` Andreas Leha
              0 siblings, 0 replies; 13+ messages in thread
            From: Andreas Leha @ 2015-06-20 21:20 UTC (permalink / raw)
              To: emacs-orgmode
            
            Rainer M Krug <Rainer@krugs.de> writes:
            > Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
            >
            >> Hi Rainer,
            >
            > Hi Andreas,
            >
            >>
            >> Rainer M Krug <Rainer@krugs.de> writes:
            >>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
            >>>
            >>>> On Wed, 17 Jun 2015, William Denton wrote:
            >>>>
            >>>>> On 17 June 2015, Xebar Saram wrote:
            >>>>>
            >>>>>> I do alot of modeling work that involves using huge datasets and run
            >>>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
            >>>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
            >>>>>> works well for small simple process but 90% of the time when dealing with
            >>>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
            >>>>>> sometimes i can C-c or C-g it and other times i need to physically kill
            >>>>>> emacs.
            >>>>>
            >>>>> I've been having the same problem for a while, but wasn't able to
            >>>>> isolate it any more than large data sets, lack of memory, and heavy
            >>>>> CPU usage. Sometimes everything hangs and I need to power cycle the
            >>>>> computer. :(
            >>>>>
            >>>>
            >>>> And you (both) have `ess-eval-visibly' set to nil, right?
            >>>>
            >>>> I do statistical genomics, which can be compute intensive. Sometimes
            >>>> processes need to run for a while, and I get impatient having to wait.
            >>>>
            >>>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
            >>>> org-mode.
            >>>>
            >>>> Basically, ravel will export Org mode to a format that knitr (and the
            >>>> like) can run - turning src blocks into `code chunks'. That allows me
            >>>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
            >>>> document to initialize objects for long running computations or to
            >>>> produce a finished report.
            >>>>
            >>>> When I start a session, I run knitr in the R session, then all the
            >>>> cached objects are loaded in and ready to use.
            >>>>
            >>>> If I write a src block I know will take a long time to export, I
            >>>> export from org mode to update the knitr document and re-knit it to
            >>>> refresh the cache.
            >>>
            >>> I have a similar workflow, only that I use a package like
            >>> approach, i.e. I tangle function definitions in a folder ./R, data into
            >>> ./data (which makes it possible to share org defined variables with R
            >>> running outside org) and scripts, i.e. the things which do a analysis,
            >>> import data, ... i.e. which might take long, into a folder ./scripts/. I
            >>> then add the usual R package infrastructure files (DESCRIPTION,
            >>> NAMESPACE, ...).
            >>> Then I have one file tangled into ./scripts/init.R:
            >>>
            >>> #+begin_src R :tangle ./scripts/init.R  
            >>> library(devtools)
            >>> load_all()
            >>> #+end_src
            >>>
            >>>
            >>> and one for the analysis:
            >>>
            >>> #+begin_src R :tangle ./scripts/myAnalysis.R  
            >>> ## Do some really time intensive and horribly complicated and important
            >>> ## stuff here
            >>> save(
            >>>     fileNames,
            >>>     bw,
            >>>     cols,
            >>>     labels,
            >>>     fit,
            >>>     dens,
            >>>     gof,
            >>>     gofPerProf,
            >>>     file = "./cache/results.myAnalysis.rds"
            >>> )
            >>> #+end_src
            >>>
            >>>
            >>> Now after tangling, I have my code easily available in a new R session:
            >>>
            >>> 1) start R in the directory in which the DESCRIPTION file is, 
            >>> 2) run source("./scripts/init.R")
            >>>
            >>> and I have all my functions and data available.
            >>>
            >>> To run a analysis, I do
            >>>
            >>> 3) source("./scripts/myAnalysis.R")
            >>>
            >>> and the results are saved in a file fn
            >>>
            >>> To analyse the data further, I can then simply use
            >>>
            >>> #+begin_src R :tangle ./scripts/myAnalysis.R
            >>> fitSing <- attach("./cache/results.myAnalysis.rds")
            >>> #+end_src
            >>>
            >>>
            >>> so they won't interfere with my environment in R.
            >>>
            >>> I can finally remove the attached environment by doing
            >>>
            >>> #+begin_src R :tangle ./scripts/myAnalysis.R  
            >>> detach(
            >>>     name = attr(fitSing, "name"),
            >>>     character.only = TRUE
            >>> )
            >>> #+end_src
            >>>
            >>> Through these caching and compartmentalizing, I can easily do some
            >>> things outside org and some inside, and easily combine all the data.
            >>>
            >>> Further advantage: I can actually create the package and send it to
            >>> somebody for testing and review and it should run out of the box, as in
            >>> the DESCRIPTION file all dependencies are defined.
            >>>
            >>> I am using this approach at the moment for a paper and which will also
            >>> result in a paper. By executing all the scripts, one will be able to do
            >>> import the raw data, do the analysis and create all graphs used in the
            >>> paper.
            >>>
            >>> Hope this gives you another idea how one can handle long running
            >>> analysis in R in org,
            >>>
            >>> Cheers,
            >>>
            >>> Rainer
            >>>
            >>
            >> That is a cool workflow.  I especially like the fact that you end up
            >> with an R package.
            >
            > Thanks. Yes - the idea of having a package at the end was one main
            > reason why I am using this approach.
            >
            >
            >>
            >> So, I'll try my again.   Is there there any chance to see working
            >> example of this?  I'd love to see that.
            >
            > Let's say I am working on it. I am working on a project which is using
            > this workflow and when it is finished, the package will be available as
            > an electronic appendix to the paper.
            >
            > But I will see if I can condense an example and blog it - I'll let you
            > kow when it is done.
            >
            
            Thanks!  Either way, I am really looking forward to this.
            
            Regards,
            Andreas
            
            ^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2015-06-20 21:20 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-17 19:09 Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Xebar Saram
2015-06-17 19:19 ` William Denton
2015-06-18  3:17   ` Charles C. Berry
2015-06-18 11:39     ` Xebar Saram
2015-06-18 18:45       ` Charles C. Berry
2015-06-18 12:34     ` Rainer M Krug
2015-06-18 20:20       ` Charles C. Berry
2015-06-18 21:20         ` Andreas Leha
2015-06-19 22:13           ` Charles C. Berry
2015-06-19 22:25             ` Andreas Leha
2015-06-19 22:31       ` Andreas Leha
2015-06-20 15:05         ` Rainer M Krug
2015-06-20 21:20           ` Andreas Leha

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).