minor bug in babel with silent output and remote R session

minor bug in babel with silent output and remote R session

[Thomas Alexander Gerds]

Using the silent option together with a remote R session block (started
via ssh.el and ess-remote), like this:

#+BEGIN_SRC R  :results silent :exports results  :session *ssh gauss* :cache yes 
a=1
1
#+END_SRC

produces:

,----
| > > > [1] 1
| > Warning message:
| In file.rename(tfile, transfer.file) :
|   cannot rename file '/tmp/RtmpQwlyCf/file7c9b78867f6c' to '/tmp/babel-4977UIT/R-4977ucf', reason 'No such file or directory'
| > 
`----

and emacs freezes. No big deal because C-g gets me out of it, but
slightly annoying.  

with `:results output' instead of `:results: silent' everything works
fine.


best,
Thomas

Re: minor bug in babel with silent output and remote R session

[Eric Schulte]

Thomas Alexander Gerds <tag@biostat.ku.dk> writes:

> Using the silent option together with a remote R session block (started
> via ssh.el and ess-remote), like this:
>
> #+BEGIN_SRC R  :results silent :exports results  :session *ssh gauss* :cache yes 
> a=1
> 1
> #+END_SRC
>
> produces:
>
> ,----
> | > > > [1] 1
> | > Warning message:
> | In file.rename(tfile, transfer.file) :
> |   cannot rename file '/tmp/RtmpQwlyCf/file7c9b78867f6c' to
> | /tmp/babel-4977UIT/R-4977ucf', reason 'No such file or directory'
> | > 
> `----
>
> and emacs freezes. No big deal because C-g gets me out of it, but
> slightly annoying.  
>
> with `:results output' instead of `:results: silent' everything works
> fine.
>

Are you sure this problem is related to the ":results silent" header
argument?  I would expect this problem to arise *any* time results are
requested from a remove R session.  This is because R and Emacs use the
file system to hand results back and forth, and when the R session
refers to a remote file system, this communication fails.

One possible solution would be to use the :dir header argument to
specify to the code block the machine on which the execution is taking
place.

Best,

>
>
> best,
> Thomas
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: minor bug in babel with silent output and remote R session

[Thomas Alexander Gerds]

yes, I am quite sure. here is the org code
,----
| 
| #+BEGIN_SRC R :results silent :exports results  :session *ssh gauss* :cache yes 
| a=1
| 1
| #+END_SRC
| 
| #+BEGIN_SRC R :results output :exports results  :session *ssh gauss* :cache yes 
| a=1
| 1
| #+END_SRC
| 
| #+RESULTS[<2013-03-13 17:26:16> 4d5d8eeab67e30060345cd66f44466bd168af55a]:
| : [1] 1
`----

and here the contents of the *ssh gauss* buffer

,----
| 
| Last login: Wed Mar 13 08:41:53 2013 from 10.128.132.66
| IFSV default server Policy.
| NFS monteret home under /home/ifsv
| $ R
| WARNING: ignoring environment value of R_HOME
| 
| R version 2.15.2 (2012-10-26) -- "Trick or Treat"
| Copyright (C) 2012 The R Foundation for Statistical Computing
| ISBN 3-900051-07-0
| Platform: x86_64-unknown-linux-gnu (64-bit)
| 
| R is free software and comes with ABSOLUTELY NO WARRANTY.
| You are welcome to redistribute it under certain conditions.
| Type 'license()' or 'licence()' for distribution details.
| 
|   Natural language support but running in an English locale
| 
| R is a collaborative project with many contributors.
| Type 'contributors()' for more information and
| 'citation()' on how to cite R or R packages in publications.
| 
| Type 'demo()' for some demos, 'help()' for on-line help, or
| 'help.start()' for an HTML browser interface to help.
| Type 'q()' to quit R.
| 
| options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
| [Previously saved workspace restored]
| 
| > > > [1] 1
| > Warning message:
| In file.rename(tfile, transfer.file) :
|   cannot rename file '/tmp/RtmpLkxWKf/file8bd75fb3ce' to '/tmp/babel-8270IwX/R-8270vVe', reason 'No such file or directory'
| > a=1
| 1
| 'org_babel_R_eoe'
| a=1
| > 1
| [1] 1
| > 'org_babel_R_eoe'
| [1] "org_babel_R_eoe"
| > 
`----

cheers Thomas


Eric Schulte <schulte.eric@gmail.com> writes:

> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>
>> Using the silent option together with a remote R session block
>> (started via ssh.el and ess-remote), like this:
>> #+BEGIN_SRC R :results silent :exports results :session *ssh gauss*
>> :cache yes a=1 1 #+END_SRC
>> produces:
>> ,----
>> | > > > [1] 1 Warning message:
>> | In file.rename(tfile, transfer.file) : cannot rename file
>> | /tmp/RtmpQwlyCf/file7c9b78867f6c' to
>> | /tmp/babel-4977UIT/R-4977ucf', reason 'No such file or directory'
>> | >
>> `----
>> and emacs freezes. No big deal because C-g gets me out of it, but
>> slightly annoying.
>> with `:results output' instead of `:results: silent' everything
>> works fine.
>>
>
> Are you sure this problem is related to the ":results silent" header
> argument?  I would expect this problem to arise *any* time results are
> requested from a remove R session.  This is because R and Emacs use
> the file system to hand results back and forth, and when the R session
> refers to a remote file system, this communication fails.
>
> One possible solution would be to use the :dir header argument to
> specify to the code block the machine on which the execution is taking
> place.
>
> Best,
>
>>
>> best, Thomas
>>

Re: minor bug in babel with silent output and remote R session

[Bastien]

Hi Thomas,

Thomas Alexander Gerds <tag@biostat.ku.dk> writes:

> Using the silent option together with a remote R session block (started
> via ssh.el and ess-remote), like this:
>
> #+BEGIN_SRC R  :results silent :exports results  :session *ssh gauss* :cache yes 
> a=1
> 1
> #+END_SRC
>
> produces:
>
> ,----
> | > > > [1] 1
> | > Warning message:
> | In file.rename(tfile, transfer.file) :
> |   cannot rename file '/tmp/RtmpQwlyCf/file7c9b78867f6c' to '/tmp/babel-4977UIT/R-4977ucf', reason 'No such file or directory'
> | > 
> `----
>
> and emacs freezes. No big deal because C-g gets me out of it, but
> slightly annoying.  
>
> with `:results output' instead of `:results: silent' everything works
> fine.

Please let us know if the documentation* needs some clarification here,
or if this is a bug -- maybe someone will have time to look at it.

Thanks,

* Better to check against the latest documentation from our master
  branch, of course.

-- 
 Bastien

Re: minor bug in babel with silent output and remote R session

[Achim Gratz]

Bastien writes:
>> #+BEGIN_SRC R  :results silent :exports results  :session *ssh gauss* :cache yes 
[…]
>> with `:results output' instead of `:results: silent' everything works
>> fine.
>
> Please let us know if the documentation* needs some clarification here,
> or if this is a bug -- maybe someone will have time to look at it.

Looking at the session argument it looks like this is supposed to be a
remote R session.  I'd think the session results would end up at the
remote side and need to be copied from there.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: minor bug in babel with silent output and remote R session

[Thomas Alexander Gerds]

Hi Bastien

I think that I can describe the problem a bit better now. It is not
related to the silent option but occurs whenever ":results value".

Emacs freezes due to the following line in
org-babel-comint-eval-invisibly-and-wait-for-file 

(while (not (file-exists-p file)) (sit-for (or period 0.25)))

it seems that R cannot transfer the file and hence this is an endless
loop.

it may be possible to fix this using tramp, e.g. by setting the
default-directory in the buffer which runs the remote-session, or by
adding a more thorough check of whether the R-session is remote.

to this end let me note that there are at least two ways to start a
remote R session in emacs:

1) M-x shell
2) M-x ssh via ssh.el (not part of emacs)

Cheers 
Thomas


Bastien <bzg@gnu.org> writes:

> Hi Thomas,
>
> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>
>> Using the silent option together with a remote R session block
>> (started via ssh.el and ess-remote), like this:
>> #+BEGIN_SRC R :results silent :exports results :session *ssh gauss*
>> :cache yes a=1 1 #+END_SRC
>> produces:
>> ,----
>> | > > > [1] 1 Warning message:
>> | In file.rename(tfile, transfer.file) : cannot rename file
>> | /tmp/RtmpQwlyCf/file7c9b78867f6c' to
>> | /tmp/babel-4977UIT/R-4977ucf', reason 'No such file or directory'
>> | >
>> `----
>> and emacs freezes. No big deal because C-g gets me out of it, but
>> slightly annoying.
>> with `:results output' instead of `:results: silent' everything
>> works fine.
>
> Please let us know if the documentation* needs some clarification
> here, or if this is a bug -- maybe someone will have time to look at
> it.
>
> Thanks,
>
> * Better to check against the latest documentation from our master
> branch, of course.
--
Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen, Denmark
Office: CSS-15.2.07 (Gamle Kommunehospital)
tel: 35327914 (sec: 35327901) 

Re: minor bug in babel with silent output and remote R session

[Bastien]

Hi Thomas,

thanks for the follow-up.

Thomas Alexander Gerds <tag@biostat.ku.dk> writes:

> I think that I can describe the problem a bit better now. It is not
> related to the silent option but occurs whenever ":results value".
>
> Emacs freezes due to the following line in
> org-babel-comint-eval-invisibly-and-wait-for-file 
>
> (while (not (file-exists-p file)) (sit-for (or period 0.25)))
>
> it seems that R cannot transfer the file and hence this is an endless
> loop.

I'm not knowledgeable enough in this area to provide a fix, maybe
someone else will.

Best,

-- 
 Bastien

Re: minor bug in babel with silent output and remote R session

[Eric Schulte]

Bastien <bzg@gnu.org> writes:

> Hi Thomas,
>
> thanks for the follow-up.
>
> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>
>> I think that I can describe the problem a bit better now. It is not
>> related to the silent option but occurs whenever ":results value".
>>
>> Emacs freezes due to the following line in
>> org-babel-comint-eval-invisibly-and-wait-for-file 
>>
>> (while (not (file-exists-p file)) (sit-for (or period 0.25)))
>>
>> it seems that R cannot transfer the file and hence this is an endless
>> loop.
>
> I'm not knowledgeable enough in this area to provide a fix, maybe
> someone else will.
>

Could this be a problem with whatever tool (I'm assuming ESS) you are
using to maintain the R session and generate the R file?

Perhaps babel needs to modify the R code used to create the file (held
in the `org-babel-R-write-object-command' variable).  Could you take a
shot at providing another version of this variable?  I don't really use
R myself.

Thanks,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: minor bug in babel with silent output and remote R session

[Thomas Alexander Gerds]

yes, I am using ESS. ess-remote allows me to evaluate R-code from the local
emacs-session on a remote machine connected to via ssh.

there are two problems:

1) the remote machine cannot write to org-babel-temp-file because the
   tmp-directory exists on the local machine. here we could add

   if(!file.exists(dirname(transfer.file))){dir.create(dirname(transfer.file))}
   
in the middle of the variable org-babel-R-write-object-command
this would achieve that the file is at least  generated on the remote host.

2) however, still the transfer file does not exist on the local
machine. there are several possiblities:

a) tell org-babel-comint-eval-invisibly-and-wait-for-file
 that the file is remote and then test if
 (concat "/" username "@" host ":" file) exists instead of file.

b) use tramp to transfer the file from the remote to the local
machine. the function ssh does define ssh-host and ssh-username,
however, calling ess-remote removes these variables again.

c) tell org-babel-comint-eval-invisibly-and-wait-for-file not to wait
 for file if it is remote

my conclusion: it would be nice to have this functionality, but perhaps
it is not worth the efforts and it would be sufficient to avoid the
endless loop when waiting for a file which never will generated.

cheers thomas


 
Eric Schulte <schulte.eric@gmail.com> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Hi Thomas,
>> thanks for the follow-up.
>> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>>
>>> I think that I can describe the problem a bit better now. It is not
>>> related to the silent option but occurs whenever ":results value".
>>> Emacs freezes due to the following line in
>>> org-babel-comint-eval-invisibly-and-wait-for-file
>>> (while (not (file-exists-p file)) (sit-for (or period 0.25)))
>>> it seems that R cannot transfer the file and hence this is an
>>> endless loop.
>> I'm not knowledgeable enough in this area to provide a fix, maybe
>> someone else will.
>>
>
> Could this be a problem with whatever tool (I'm assuming ESS) you are
> using to maintain the R session and generate the R file?
>
> Perhaps babel needs to modify the R code used to create the file (held
> in the `org-babel-R-write-object-command' variable).  Could you take a
> shot at providing another version of this variable?  I don't really
> use R myself.
>
> Thanks,
--
Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen, Denmark
Office: CSS-15.2.07 (Gamle Kommunehospital)
tel: 35327914 (sec: 35327901) 

Re: minor bug in babel with silent output and remote R session

[Eric Schulte]

It is a shame that this can't be handled gracefully either through ESS
or R code.

I agree it would be nice to raise a warning rather than hang waiting for
a file which won't ever exist.  So, how can we tell from the Babel
source if the R session is remote?

Thanks,

Thomas Alexander Gerds <tag@biostat.ku.dk> writes:

> yes, I am using ESS. ess-remote allows me to evaluate R-code from the local
> emacs-session on a remote machine connected to via ssh.
>
> there are two problems:
>
> 1) the remote machine cannot write to org-babel-temp-file because the
>    tmp-directory exists on the local machine. here we could add
>
>    if(!file.exists(dirname(transfer.file))){dir.create(dirname(transfer.file))}
>    
> in the middle of the variable org-babel-R-write-object-command
> this would achieve that the file is at least  generated on the remote host.
>
> 2) however, still the transfer file does not exist on the local
> machine. there are several possiblities:
>
> a) tell org-babel-comint-eval-invisibly-and-wait-for-file
>  that the file is remote and then test if
>  (concat "/" username "@" host ":" file) exists instead of file.
>
> b) use tramp to transfer the file from the remote to the local
> machine. the function ssh does define ssh-host and ssh-username,
> however, calling ess-remote removes these variables again.
>
> c) tell org-babel-comint-eval-invisibly-and-wait-for-file not to wait
>  for file if it is remote
>
> my conclusion: it would be nice to have this functionality, but perhaps
> it is not worth the efforts and it would be sufficient to avoid the
> endless loop when waiting for a file which never will generated.
>
> cheers thomas
>
>
>  
> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Bastien <bzg@gnu.org> writes:
>>
>>> Hi Thomas,
>>> thanks for the follow-up.
>>> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>>>
>>>> I think that I can describe the problem a bit better now. It is not
>>>> related to the silent option but occurs whenever ":results value".
>>>> Emacs freezes due to the following line in
>>>> org-babel-comint-eval-invisibly-and-wait-for-file
>>>> (while (not (file-exists-p file)) (sit-for (or period 0.25)))
>>>> it seems that R cannot transfer the file and hence this is an
>>>> endless loop.
>>> I'm not knowledgeable enough in this area to provide a fix, maybe
>>> someone else will.
>>>
>>
>> Could this be a problem with whatever tool (I'm assuming ESS) you are
>> using to maintain the R session and generate the R file?
>>
>> Perhaps babel needs to modify the R code used to create the file (held
>> in the `org-babel-R-write-object-command' variable).  Could you take a
>> shot at providing another version of this variable?  I don't really
>> use R myself.
>>
>> Thanks,
> --
> Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
> University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen, Denmark
> Office: CSS-15.2.07 (Gamle Kommunehospital)
> tel: 35327914 (sec: 35327901) 

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: minor bug in babel with silent output and remote R session

[Andreas Leha]

Hi all,


Thomas Alexander Gerds <tag@biostat.ku.dk> writes:

> Hi Bastien
>
> I think that I can describe the problem a bit better now. It is not
> related to the silent option but occurs whenever ":results value".
>
> Emacs freezes due to the following line in
> org-babel-comint-eval-invisibly-and-wait-for-file 
>
> (while (not (file-exists-p file)) (sit-for (or period 0.25)))
>

sorry for hijacking this thread.  With a quite useless message...

I experience freezes at that exact position also on local sessions
sometimes.  I have never reported, because I can't send the scripts
where this happens and slight modifications to the causing source block
usually 'fix' this.  So I failed in creating a reproducible example that
I can send so far.

The main purpose of this message is the hope, that someone else has
experienced and solved that problem before (maybe changing some
ESS-setup?).

I will try more to get a reproducible example and report if I get one.

Best,
Andreas

Re: minor bug in babel with silent output and remote R session

[Thomas Alexander Gerds]

hmm. I agree that this should be handled by ESS and I have not given up yet.

as indicated in my previous mail, I dont know how to test if an
R-session is remote because the command ess-remote deletes all local
variables. a hack would be to let the R-process evaluate

file.exists((file-name-directory org-babel-temp-file))

thanks

Eric Schulte <schulte.eric@gmail.com> writes:

> It is a shame that this can't be handled gracefully either through ESS
> or R code.
>
> I agree it would be nice to raise a warning rather than hang waiting
> for a file which won't ever exist.  So, how can we tell from the Babel
> source if the R session is remote?
>
> Thanks,
>
> Thomas Alexander Gerds <tag@biostat.ku.dk> writes:
>
>> yes, I am using ESS. ess-remote allows me to evaluate R-code from
>> the local emacs-session on a remote machine connected to via ssh.
>> there are two problems:
>> 1) the remote machine cannot write to org-babel-temp-file because
>> the tmp-directory exists on the local machine. here we could add
>> if(!file.exists(dirname(transfer.file))){dir.create(dirname(transfer.file))}
>>    
>> in the middle of the variable org-babel-R-write-object-command this
>> would achieve that the file is at least generated on the remote
>> host.
>> 2) however, still the transfer file does not exist on the local
>> machine. there are several possiblities:
>> a) tell org-babel-comint-eval-invisibly-and-wait-for-file that the
>> file is remote and then test if (concat "/" username "@" host ":"
>> file) exists instead of file.
>> b) use tramp to transfer the file from the remote to the local
>> machine. the function ssh does define ssh-host and ssh-username,
>> however, calling ess-remote removes these variables again.
>> c) tell org-babel-comint-eval-invisibly-and-wait-for-file not to
>> wait for file if it is remote
>> my conclusion: it would be nice to have this functionality, but
>> perhaps it is not worth the efforts and it would be sufficient to
>> avoid the endless loop when waiting for a file which never will
>> generated.
>> cheers thomas
>>
>>
>>  
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> Bastien <bzg@gnu.org> writes:
>>>
>>>> Hi Thomas, thanks for the follow-up.  Thomas Alexander Gerds
>>>> <tag@biostat.ku.dk> writes:
>>>>
>>>>> I think that I can describe the problem a bit better now. It is
>>>>> not related to the silent option but occurs whenever ":results
>>>>> value".  Emacs freezes due to the following line in
>>>>> org-babel-comint-eval-invisibly-and-wait-for-file (while (not
>>>>> (file-exists-p file)) (sit-for (or period 0.25))) it seems that R
>>>>> cannot transfer the file and hence this is an endless loop.
>>>> I'm not knowledgeable enough in this area to provide a fix, maybe
>>>> someone else will.
>>>>
>>> Could this be a problem with whatever tool (I'm assuming ESS) you
>>> are using to maintain the R session and generate the R file?
>>> Perhaps babel needs to modify the R code used to create the file
>>> (held in the `org-babel-R-write-object-command' variable).  Could
>>> you take a shot at providing another version of this variable?  I
>>> don't really use R myself.
>>> Thanks,
>> -- Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
>> University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen,
>> Denmark Office: CSS-15.2.07 (Gamle Kommunehospital) tel: 35327914
>> (sec: 35327901)
--
Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen, Denmark
Office: CSS-15.2.07 (Gamle Kommunehospital)
tel: 35327914 (sec: 35327901)