From: John Hendy <jw.hendy@gmail.com>
To: nicholas.dokos@hp.com
Cc: emacs-orgmode@gnu.org, cberry@tajo.ucsd.edu
Subject: Re: Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Date: Thu, 1 Nov 2012 09:53:51 -0500 [thread overview]
Message-ID: <CA+M2ft_TZphAy=VuyhCuhGbVFvYuaUFpyLQFZZ931LmEAaVe4w@mail.gmail.com> (raw)
In-Reply-To: <3477.1351723988@alphaville>
[-- Attachment #1: Type: text/plain, Size: 5992 bytes --]
On Wed, Oct 31, 2012 at 5:53 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:
> John Hendy <jw.hendy@gmail.com> wrote:
>
> > On Wed, Oct 31, 2012 at 3:12 PM, <cberry@tajo.ucsd.edu> wrote:
> >
> > John Hendy <jw.hendy@gmail.com> writes:
> >
> > > On Wed, Oct 31, 2012 at 11:41 AM, <span dir="ltr"><mailto:
> cberry@tajo.ucsd.edu></span> wrote:
> > > John Hendy <mailto:jw.hendy@gmail.com> writes:
> > >
> > >> I edited the subject to be more concise/clear.I let orgmode chug
> away
> > >> on reading in some ~10-30mb csv files for nearly 30min.
> > >
> > > [rest deleted]
> > >
> > > You need an ECM.I did my best to provide one, other than the file,
> which I offered to provide
> > if others requested that I upload it somewhere. Since you have done
> so, so have I:
> > > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> >
> > > Let me know if that doesn't work. I put it on Google docs and
> sometimes have issues with
> > the sharing settings...
> >
> > Not an ECM in my book, but ...
> >
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
>
> Smaller set of data I'd guess :-) But it does not seem to be the
> size of the data that matters.
>
> >
> >
> > On my 4 year old MacBook:
> >
> > ,----
> > |
> > | #+PROPERTY: session *R*
> > |
> > | #+name: bigcsv
> > | #+begin_src R
> > | bigcsv <- Sys.glob("~/Downloads/*.csv")
> > | #+end_src
> > |
> > | #+RESULTS: bigcsv
> > | : /Users/cberry/Downloads/test-file.csv
> > |
> > | #+name: readbig
> > | #+begin_src R :results output
> > | system.time(
> > | tmp <- read.csv(bigcsv)
> > | )
> > |
> > | #+end_src
> > |
> > | #+RESULTS: readbig
> > | : user system elapsed
> > | : 5.679 0.306 6.002
> > |
> > `----
> >
> > About the same as running from ESS.
> >
> > Not sure what to say. Looking for ways to troubleshoot or confirm. Since
> you can't confirm, any
> > suggestions on where I should look for my issue? I can't explain it! All
> I know is that org chugs
> > and chugs and the direct execution in ESS session is lightning fast.
> >
>
> A few things to try in no particular order:
>
>
This was extremely helpful. Thanks for the suggestions.
Here's my attempt at an ECM, though I'm going to keep using the big file
since that's what's actually doing it an I've already uploaded it :)
- Using emacs config here: http://pastebin.com/raw.php?i=iTbRtCE9
- Using this org-mode file:
#+begin_src org
* headline
#+begin_src R :session r :results silent
# file here:
https://docs.google.com/uc?export=download&confirm=no_antivirus&id=0BzQupOSnvw08WHdabHh5VVczRGM
data <- read.csv("path/to/file.csv")
#+end_src
#+end_src org
- Execute block with C-c C-c after downloading and changing path
o run top (or whatever equivalent is available on your OS) and see
> whether the CPU (or one of the CPUs) gets pegged at 100% utilization
> and stays there. If yes, that's an indication of an infinite loop
> somewhere.
>
>
- quit any other instances of emacs/R
- start `top` in terminal
- execute block
- Use '<' '>' to sort back and forth between cpu and ram
Observations
- R is at 80-100% cpu for about 5sec
- Then emacs shifts to fairly constant ~100% cpu usage
- After about a minute, the minibuffer expands to ~1/3 of the window height
and fills with the csv data
- Finished after ~5min total time
- So, R took about 5sec, emacs took another 5min to finish
> o run vmstat (or equivalent) and see if any of the counters are out of
> whack.
> That requires some experience though.
>
>
I'll skip for now; no experience with that.
> o use elp-instrument-package to instrument org and run the test, getting
> a profile. I'm not sure whether the results will be useful, since you
> are going to interrupt the test when you run out of patience, but it
> cannot hurt and it might tell you something useful.
>
> o run your ECM on a different computer/OS/emacs installation. Being able
> to compare things side by side is often very useful.
>
> o Halve your file and run the test on each half (but that's probably not
> the problem given Chuck's results).
>
> o Reinstall org from scratch - you might have some corruption in one of
> the compiled files that's causing it to go into an infinite loop.
>
>
- `cd ~/.elisp`
- `sudo rm -r org.git`
- `git clone http://git://orgmode.org/org-mode.git org.git`
- cd org.git && make clean && make && make doc
- Quit previous emacs instance; reopen
- Remove (require 'org-install) per prompt; restart again
- Repeat `top` experiment
Results:
- Didn't even see R flash on the screen this time; emacs just jumped to 100%
- After 1min 10sec, the minibuffer filled with data
- At that point I quit, as I think it will be a repeat of the above
> o Turn on debug-on-quit, start your test, wait a bit and then interrupt
> it. Check the backtrace. Do it again and check whether the backtrace
> looks the same. That's often an indication of an infinite loop
> (inferring an infinite loop from a two element sample is statistically
> suspect of course, but surprisingly effective nevertheless). The point
> here is that the infinite loop is in emacs and the backtrace tells you
> something about the parties involved.
>
>
- =M-x customize-variable RET debug-on-quit RET=
- Toggled to on; saved for current session
- Waited about a min (till the minibuffer filled), then did C-g
Don't have experience with debugging. It brings me to a *Backtrace* buffer,
which is empty except for the line "Debugger entered--Lisp error: (quit)"
Thanks for the suggestions and help. That was quite above and beyond. Much
appreciated.
Best regards,
John
These are obviously not independent and the results of one experiment will
> have to guide you in what you try next.
>
> Good luck,
> Nick
>
[-- Attachment #2: Type: text/html, Size: 9116 bytes --]
next prev parent reply other threads:[~2012-11-01 14:54 UTC|newest]
Thread overview: 217+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-31 6:12 Bug? R: Org babel block execution *drastically* slower than in ESS session directly John Hendy
2012-10-31 16:41 ` cberry
2012-10-31 17:18 ` John Hendy
2012-10-31 20:12 ` cberry
2012-10-31 20:23 ` John Hendy
2012-10-31 20:56 ` Thomas S. Dye
2012-10-31 23:00 ` Nick Dokos
2012-10-31 22:53 ` Nick Dokos
2012-11-01 14:53 ` John Hendy [this message]
2012-11-01 15:38 ` Nick Dokos
2012-11-01 18:17 ` John Hendy
2012-11-01 18:18 ` John Hendy
2012-11-01 18:48 ` Nick Dokos
2012-11-14 3:27 ` Aaron Ecay
2012-11-14 4:52 ` John Hendy
2012-11-14 6:32 ` Aaron Ecay
2012-11-14 8:28 ` Thomas S. Dye
2012-11-14 8:35 ` Andreas Leha
2012-11-16 15:45 ` Eric Schulte
2012-11-16 17:47 ` Andreas Leha
2012-11-16 19:00 ` John Hendy
2012-11-16 20:25 ` Achim Gratz
2012-11-17 17:12 ` Eric Schulte
2012-11-18 0:41 ` Aaron Ecay
2012-11-18 0:57 ` Eric Schulte
2012-11-18 1:41 ` Aaron Ecay
2012-11-19 1:18 ` Eric Schulte
2012-11-19 5:11 ` Eric Schulte
-- strict thread matches above, loose matches on Subject: below --
2013-03-19 21:40 python sessions Gary Oberbrunner
2013-03-20 0:04 ` John Hendy
2013-03-20 3:07 ` Nick Dokos
2013-03-20 6:44 ` Andreas Röhler
2013-03-20 12:54 ` Ista Zahn
2013-03-20 14:08 ` Nick Dokos
2013-03-20 14:28 ` Gary Oberbrunner
2013-03-20 17:21 ` Andreas Röhler
2013-03-20 7:08 ` Andreas Röhler
2013-03-20 15:38 ` John Hendy
2013-03-20 15:42 ` John Hendy
2013-03-20 16:06 ` Nick Dokos
[not found] ` <CAFChFygsUeBejEY6m2bSU0C5cjLYz0nUADH=s+BYxajfZ2sKKQ@mail.gmail.com>
2013-03-20 16:12 ` Fwd: " Gary Oberbrunner
2013-03-20 17:08 ` Andreas Röhler
2013-03-20 17:25 ` Nick Dokos
2013-03-20 17:54 ` John Hendy
2013-03-20 18:15 ` Nick Dokos
2013-03-20 18:32 ` Ista Zahn
2013-03-20 18:39 ` Nick Dokos
2013-03-20 18:43 ` Andreas Röhler
2013-03-20 19:42 ` Ista Zahn
2013-03-20 19:53 ` Andreas Röhler
2013-03-20 20:12 ` Ista Zahn
2013-03-20 20:25 ` Gary Oberbrunner
2013-03-20 22:06 ` Ista Zahn
2013-03-20 18:42 ` Andreas Röhler
2013-03-20 19:01 ` Andreas Röhler
[not found] ` <CAFChFygPrhuHHyMo2Vk2_OdksWVB=oACA7jnOXN+itUNqh-nqw@mail.gmail.com>
[not found] ` <CA+M2ft9bO7DU61vVeYrhduVvkg1V6PMuiyvsOy5tNWuXcSUEpw@mail.gmail.com>
[not found] ` <CAFChFyhbeXE4HNLKA6Ubii+MtQwD7=4khAypa+--coZrQiJekA@mail.gmail.com>
2013-03-21 1:22 ` Gary Oberbrunner
2013-03-21 1:54 ` John Hendy
2013-03-21 1:56 ` Gary Oberbrunner
2013-03-21 7:42 ` Andreas Röhler
2013-03-21 7:43 ` Bastien
2013-03-21 8:13 ` Andreas Röhler
2013-03-23 22:07 ` Eric Schulte
2013-03-24 1:29 ` John Hendy
2013-03-24 22:59 ` Eric Schulte
2013-03-25 2:38 ` Nick Dokos
2013-03-25 2:59 ` John Hendy
2013-03-25 6:34 ` Andreas Röhler
2013-03-25 15:40 ` Eric Schulte
2013-03-25 15:55 ` John Hendy
2013-03-25 16:07 ` Eric Schulte
2013-03-25 16:41 ` Nick Dokos
2013-03-25 16:01 ` Ista Zahn
2013-03-25 16:23 ` John Hendy
2013-03-25 16:43 ` Eric Schulte
2013-03-25 17:27 ` Andreas Röhler
2013-03-25 17:41 ` John Hendy
2013-03-25 18:16 ` Ista Zahn
2013-03-25 19:30 ` Ivan Andrus
2013-03-24 7:47 ` Andreas Röhler
2013-03-24 14:47 ` John Hendy
2013-03-24 15:30 ` Andreas Röhler
2013-03-24 16:58 ` Nick Dokos
2013-03-24 16:59 ` John Hendy
2013-03-24 18:41 ` Nick Dokos
2013-03-25 20:46 ` Andreas Röhler
2013-03-25 21:37 ` Eric Schulte
2013-03-26 6:29 ` Andreas Röhler
2013-03-26 12:32 ` Eric Schulte
2013-03-26 21:41 ` Gary Oberbrunner
2013-03-26 21:47 ` John Hendy
2013-03-27 6:19 ` Andreas Röhler
2013-03-30 7:19 ` Andreas Röhler
2012-08-26 20:11 Alternate format for datetree c b
2012-08-27 1:16 ` John Hendy
2012-08-27 4:04 ` Nick Dokos
2012-08-27 6:08 ` John Hendy
2012-08-28 8:49 ` Ian Barton
2012-08-28 12:49 ` Nick Dokos
2012-08-28 12:50 ` Nick Dokos
2012-08-29 7:49 ` Ian Barton
[not found] ` <lists@manor-farm.org>
2012-08-29 14:17 ` Nick Dokos
2012-08-29 14:25 ` John Hendy
2012-08-29 20:01 ` Ian Barton
2012-09-06 7:46 ` Ian Barton
2012-09-06 5:42 ` c b
2012-09-06 6:26 ` Nick Dokos
2012-09-06 7:00 ` Jambunathan K
2012-09-06 15:33 ` John Hendy
2012-09-06 16:08 ` Jonathan Leech-Pepin
2011-11-02 2:22 Pass LaTeX exporter option prior to \documentclass John Hendy
2011-11-02 2:43 ` suvayu ali
2011-11-02 10:30 ` suvayu ali
2011-11-03 17:48 ` John Hendy
2011-11-03 17:55 ` suvayu ali
2011-11-03 18:07 ` John Hendy
2011-11-04 17:29 ` Nick Dokos
2011-11-04 19:29 ` suvayu ali
2011-10-19 14:54 Prompt for time when clocking in? Nathan Neff
2011-10-19 15:12 ` John Hendy
2011-10-19 15:35 ` Nick Dokos
2011-10-20 16:15 ` Nathan Neff
2011-10-20 17:09 ` Nick Dokos
2011-10-20 17:44 ` Nathan Neff
2011-10-19 16:37 ` Bernt Hansen
2011-08-05 22:15 Handling errors in command line exporting of agenda? John Hendy
2011-08-05 23:34 ` suvayu ali
2011-08-05 23:35 ` suvayu ali
2011-08-08 18:12 ` John Hendy
2011-08-08 18:28 ` suvayu ali
2011-08-08 18:53 ` John Hendy
2011-08-08 18:59 ` suvayu ali
2011-08-08 19:48 ` John Hendy
2011-08-08 20:22 ` suvayu ali
2011-08-08 20:31 ` John Hendy
2011-08-08 20:33 ` Nick Dokos
2011-08-08 20:37 ` Nick Dokos
2011-08-08 20:54 ` John Hendy
2011-08-08 21:09 ` Nick Dokos
2011-08-08 21:27 ` John Hendy
2011-08-08 21:30 ` suvayu ali
2011-08-08 22:17 ` Nick Dokos
2011-08-08 21:46 ` suvayu ali
2011-06-15 10:29 BUG: Newest org-mode interferes with desktop save Rainer M Krug
2011-06-15 16:49 ` Eric Schulte
2011-06-15 17:35 ` Nick Dokos
2011-06-15 18:23 ` Rainer M Krug
2011-06-15 18:21 ` Rainer M Krug
2011-06-15 18:31 ` Eric Schulte
2011-06-15 18:59 ` Rainer M Krug
2011-06-15 22:40 ` Nick Dokos
2011-06-16 7:15 ` Rainer M Krug
2011-06-16 7:29 ` Jambunathan K
2011-06-16 8:22 ` Rainer M Krug
2011-05-24 15:54 Passing font size to exported LaTeX table John Hendy
2011-05-24 16:44 ` Sebastien Vauban
2011-05-24 18:57 ` John Hendy
2011-05-24 19:25 ` Nick Dokos
2011-05-24 19:31 ` John Hendy
2011-05-24 19:58 ` Thomas S. Dye
2011-05-24 20:42 ` Nick Dokos
2011-05-24 21:12 ` Nick Dokos
2011-05-24 21:19 ` Nick Dokos
2011-05-25 7:22 ` Thomas S. Dye
2011-05-27 0:58 ` suvayu ali
2011-05-27 6:46 ` Thomas S. Dye
2011-05-27 7:07 ` suvayu ali
2011-05-27 16:17 ` Thomas S. Dye
2011-05-27 16:50 ` Suvayu Ali
2011-05-27 17:37 ` Thomas S. Dye
2011-05-27 18:30 ` Suvayu Ali
2011-05-27 21:25 ` Thomas S. Dye
2011-05-28 7:17 ` Sebastien Vauban
2011-05-28 10:14 ` Suvayu Ali
2011-05-28 15:21 ` Nick Dokos
2011-05-28 18:10 ` Suvayu Ali
2011-06-01 18:50 ` Nick Dokos
2011-06-01 19:08 ` Suvayu Ali
2011-06-10 22:31 ` Suvayu Ali
2011-06-13 1:38 ` suvayu ali
2011-06-13 1:52 ` Nick Dokos
2011-06-13 2:40 ` Suvayu Ali
2011-06-13 3:29 ` Nick Dokos
2011-06-13 3:45 ` Thomas S. Dye
2011-06-13 4:27 ` Nick Dokos
2011-06-13 4:56 ` Suvayu Ali
2011-06-13 5:29 ` Nick Dokos
2011-06-13 6:42 ` Suvayu Ali
2011-05-24 20:06 ` Nick Dokos
2011-05-24 19:42 ` Sebastien Vauban
2010-08-21 19:30 Gnuplot unevenly spaced non-numeric data plot? John Hendy
2010-08-21 20:35 ` Eric S Fraga
2010-08-21 20:41 ` Nick Dokos
2010-08-21 21:11 ` John Hendy
2010-08-21 22:41 ` John Hendy
2010-08-21 23:17 ` Nick Dokos
2010-08-23 14:22 ` John Hendy
2010-08-23 15:16 ` Nick Dokos
[not found] ` <AANLkTi=WTLBL1Giq0GTrjCo-A1s=iP4u1Qxn57cH-xnB@mail.gmail.com>
[not found] ` <AANLkTimSVm=-2o39CQ9wSMT276Dee4Tuj9jLKJH-c+cd@mail.gmail.com>
2010-08-24 2:44 ` [For Worg?] " John Hendy
2010-08-24 5:25 ` suvayu ali
2010-08-24 5:56 ` Nick Dokos
2010-08-25 0:32 ` Eric Schulte
2010-08-25 1:06 ` John Hendy
2010-08-25 1:35 ` Eric Schulte
2010-08-25 18:38 ` John Hendy
2010-08-25 18:52 ` Eric Schulte
[not found] ` <jw.hendy@gmail.com>
2012-03-12 21:13 ` Beamer specific setupfile? John Hendy
2012-03-12 21:48 ` Nick Dokos
2012-03-12 22:09 ` John Hendy
2012-03-12 22:30 ` Nick Dokos
2012-03-12 22:44 ` Nick Dokos
2012-03-13 5:02 ` John Hendy
2012-03-13 5:46 ` Nick Dokos
2012-04-25 8:16 ` Eric Fraga
2012-04-25 8:59 ` Eric Fraga
2012-03-12 23:01 ` Nick Dokos
2012-03-12 21:51 ` Nick Dokos
2012-03-12 22:05 ` Nick Dokos
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CA+M2ft_TZphAy=VuyhCuhGbVFvYuaUFpyLQFZZ931LmEAaVe4w@mail.gmail.com' \
--to=jw.hendy@gmail.com \
--cc=cberry@tajo.ucsd.edu \
--cc=emacs-orgmode@gnu.org \
--cc=nicholas.dokos@hp.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).