Re: Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Nick Dokos <nicholas.dokos@hp.com>]

Thomas S. Dye <tsd@tsdye.com> wrote:

> Hi John,
> 
> Have you tried wrapping your R read in system.time()?  If you are right
> about :results silent eating up lots of time, then this should fix the
> problem.  system.time yields just a bit of output, so shouldn't slow
> things down if writing out the data is indeed the problem as you
> suspect. 
> 

I missed this in the thread so I thought I'd add one more thing to try:

o run iostat (or equivalent) to check how your disks are doing.
  on linux, iostat is part of a package usually called sysstat.

Nick

> 
> 
> John Hendy <jw.hendy@gmail.com> writes:
> 
> > On Wed, Oct 31, 2012 at 3:12 PM, <cberry@tajo.ucsd.edu> wrote:
> >
> >> John Hendy <jw.hendy@gmail.com> writes:
> >>
> >> > On Wed, Oct 31, 2012 at 11:41 AM,  <span dir="ltr"><mailto:
> >> cberry@tajo.ucsd.edu></span> wrote:
> >> > John Hendy <mailto:jw.hendy@gmail.com> writes:
> >> >
> >> >> I edited the subject to be more concise/clear.I let orgmode chug away
> >> >> on reading in some ~10-30mb csv files for nearly 30min.
> >> >
> >> > [rest deleted]
> >> >
> >> > You need an ECM.I did my best to provide one, other than the file, which
> >> I offered to provide if others requested that I upload it somewhere. Since
> >> you have done so, so have I:
> >> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> >>
> >> > Let me know if that doesn&#39;t work. I put it on Google docs and
> >> sometimes have issues with the sharing settings...
> >>
> >> Not an ECM in my book, but ...
> >>
> >>
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
> >
> >
> >> On my 4 year old MacBook:
> >>
> >> ,----
> >> |
> >> | #+PROPERTY: session *R*
> >> |
> >> | #+name: bigcsv
> >> | #+begin_src R
> >> | bigcsv <- Sys.glob("~/Downloads/*.csv")
> >> | #+end_src
> >> |
> >> | #+RESULTS: bigcsv
> >> | : /Users/cberry/Downloads/test-file.csv
> >> |
> >> | #+name: readbig
> >> | #+begin_src R :results output
> >> |   system.time(
> >> |     tmp <- read.csv(bigcsv)
> >> |     )
> >> |
> >> | #+end_src
> >> |
> >> | #+RESULTS: readbig
> >> | :    user  system elapsed
> >> | :   5.679   0.306   6.002
> >> |
> >> `----
> >>
> >> About the same as running from ESS.
> >>
> >>
> > Not sure what to say. Looking for ways to troubleshoot or confirm. Since
> > you can't confirm, any suggestions on where I should look for my issue? I
> > can't explain it! All I know is that org chugs and chugs and the direct
> > execution in ESS session is lightning fast.
> >
> >
> >
> > FYI,
> >>
> >> > dim(tmp)
> >> [1] 964805      9
> >>
> >>
> >>
> > Was that just to show the size, or is something wrong with my file?
> >
> >
> >> Chuck
> >>
> >>
> >
> > Thanks for attempting to replicate!
> > John
> >
> >
> >> [ rest deleted ]
> >>
> >>
> >>
> >>
> > On Wed, Oct 31, 2012 at 3:12 PM, <cberry@tajo.ucsd.edu> wrote:
> >
> >     John Hendy <jw.hendy@gmail.com> writes:
> >     
> >     > On Wed, Oct 31, 2012 at 11:41 AM,  <span
> >     dir="ltr"><mailto:cberry@tajo.ucsd.edu></span> wrote:
> >     > John Hendy <mailto:jw.hendy@gmail.com> writes:
> >     >
> >     >> I edited the subject to be more concise/clear.I let orgmode
> >     chug away
> >     >> on reading in some ~10-30mb csv files for nearly 30min.
> >     >
> >     > [rest deleted]
> >     >
> >     > You need an ECM.I did my best to provide one, other than the
> >     file, which I offered to provide if others requested that I upload
> >     it somewhere. Since you have done so, so have I:
> >     > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> >     
> >     > Let me know if that doesn&#39;t work. I put it on Google docs
> >     and sometimes have issues with the sharing settings...
> >     
> >     Not an ECM in my book, but ...
> >     
> >     
> >
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
> >  
> >
> >     On my 4 year old MacBook:
> >     
> >     ,----
> >     |
> >     | #+PROPERTY: session *R*
> >     |
> >     | #+name: bigcsv
> >     | #+begin_src R
> >     | bigcsv <- Sys.glob("~/Downloads/*.csv")
> >     | #+end_src
> >     |
> >     | #+RESULTS: bigcsv
> >     | : /Users/cberry/Downloads/test-file.csv
> >     |
> >     | #+name: readbig
> >     | #+begin_src R :results output
> >     |   system.time(
> >     |     tmp <- read.csv(bigcsv)
> >     |     )
> >     |
> >     | #+end_src
> >     |
> >     | #+RESULTS: readbig
> >     | :    user  system elapsed
> >     | :   5.679   0.306   6.002
> >     |
> >     `----
> >     
> >     About the same as running from ESS.
> >     
> >     
> >
> > Not sure what to say. Looking for ways to troubleshoot or confirm.
> > Since you can't confirm, any suggestions on where I should look for my
> > issue? I can't explain it! All I know is that org chugs and chugs and
> > the direct execution in ESS session is lightning fast.
> >
> >
> >
> >     FYI,
> >     
> >     > dim(tmp)
> >     [1] 964805      9
> >     
> >     
> >     
> >
> > Was that just to show the size, or is something wrong with my file?
> >  
> >
> >     Chuck
> >     
> >     
> >
> >
> > Thanks for attempting to replicate!
> > John
> >  
> >
> >     [ rest deleted ]
> >     
> >     
> >     
> >     
> >
> 
> -- 
> Thomas S. Dye
> http://www.tsdye.com
>