From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xebar Saram Subject: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Date: Wed, 17 Jun 2015 22:09:35 +0300 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7b15ad7983d3a50518bb6e64 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49224) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5IiP-0005zO-Fj for emacs-orgmode@gnu.org; Wed, 17 Jun 2015 15:09:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z5IiO-0007nc-Fp for emacs-orgmode@gnu.org; Wed, 17 Jun 2015 15:09:37 -0400 Received: from mail-pd0-x229.google.com ([2607:f8b0:400e:c02::229]:33256) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5IiO-0007mb-7x for emacs-orgmode@gnu.org; Wed, 17 Jun 2015 15:09:36 -0400 Received: by pdjn11 with SMTP id n11so46960271pdj.0 for ; Wed, 17 Jun 2015 12:09:35 -0700 (PDT) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: org mode --047d7b15ad7983d3a50518bb6e64 Content-Type: text/plain; charset=UTF-8 Hi all im not sure if this is absolutely the correct forum to raise this and if not would be also happy to get input on where to persue this issue. i recently dived into using orgmode, ESS, Babel etc to run code and im really love it. The problem i have (and im not sure if its a org,Emacs or ess issue) is that emacs sometimes (and mostly when dealing with R processes involving HUGE databases) will just hang/freeze. I do alot of modeling work that involves using huge datasets and run process intensive R processes (such as complex mixed models, Gamms etc). in R studio all works well yet when i use the orgmode eval on R code blocks it works well for small simple process but 90% of the time when dealing with complex models and bug data (up to 256GB) it will just freeze emacs/ess. sometimes i can C-c or C-g it and other times i need to physically kill emacs. here is an example of such process that hangs **** lmer run the lmer part regressing stage 2 pred Vs mean pm #+BEGIN_SRC R :session Rorg :results none m2.smooth = lme(pred.m2 ~ meanPM25,random = list(aodid= ~1 + meanPM25),control=lmeControl(opt = "optim"), data= mod2 ) #correlate to see everything from mod2 and the mpm works mod2[, pred.t31 := predict(m2.smooth)] mod2[, resid := residuals(m2.smooth)] print(summary(lm(pred.m2~pred.t31,data=mod2))$r.squared) #+END_SRC i usually issue org-babel-execute-subtree to eval several subsections under a main header. again i dont know if its an org mode isse perse but would love to hear from people that have experience using R/org with big data/RAM and maybe point me to where to raise these issues best Z --047d7b15ad7983d3a50518bb6e64 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all

im not sure if this is absolutel= y the correct forum to raise this and if not would be also happy to get inp= ut on where to persue this issue.

i recently dived= into using orgmode, ESS, Babel etc to run code and im really love it. The = problem i have (and im not sure if its a org,Emacs or ess issue) is that em= acs sometimes (and mostly when dealing with R processes involving HUGE data= bases) will just hang/freeze.

I do alot of modelin= g work that involves using huge datasets and run process intensive R proces= ses (such as complex mixed models, Gamms etc). in R studio all works well y= et when i use the orgmode eval on R code blocks it works well for small sim= ple process but 90% of the time when dealing with complex models and bug da= ta (up to 256GB) it will just freeze emacs/ess. sometimes i can C-c or C-g = it and other times i need to physically kill emacs.

here is an example of such process that hangs


**** lmer=C2=A0
run the lmer part regressing st= age 2 pred Vs mean pm

#+BEGIN_SRC R =C2=A0:session= Rorg =C2=A0:results none
m2.smooth =3D lme(pred.m2 ~ meanPM25,ra= ndom =3D list(aodid=3D ~1 + meanPM25),control=3DlmeControl(opt =3D "op= tim"), data=3D mod2 )
#correlate to see everything from mod2= and the mpm works
mod2[, pred.t31 :=3D predict(m2.smooth)]
=
mod2[, resid =C2=A0:=3D residuals(m2.smooth)]
print(summary(= lm(pred.m2~pred.t31,data=3Dmod2))$r.squared)
#+END_SRC

i usually issue org-babel-execute-subtree to eval seve= ral subsections under a main header.

again i dont = know if its an org mode isse perse but would love to hear from people that = have experience using R/org with big data/RAM and maybe point me to where t= o raise these issues

best

Z --047d7b15ad7983d3a50518bb6e64--