Using tramp to run R source code blocks on remote server

Using tramp to run R source code blocks on remote server

[Vikas Rawal]

I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as cesp).

I am facing several problems.

1.  The following sample block, when used with ":results value" does not give any results.

#+NAME: level1
#+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:/home/vikas/
c(1:5)
#+end_src

I get the following message:

tramp-flush-directory-property: Wrong type argument: arrayp, nil

I get the expected output if I use ":results output”, but I don’t want that.

2. Is there a way to be able to use screen/byobu with remote processing? I have tried various options of using shell, term, or ansi-term. Each of them gives some problem or the other. Does anyone have experience with using any of these?

Would appreciate help

Vikas

Re: Using tramp to run R source code blocks on remote server

[Charles C. Berry]

On Thu, 21 Apr 2016, Vikas Rawal wrote:

> I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as cesp).
>
> I am facing several problems.
>
> 1.  The following sample block, when used with ":results value" does not give any results.
>
> #+NAME: level1
> #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:/home/vikas/
> c(1:5)
> #+end_src
>
> I get the following message:
>
> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>

I think this is an issue with having different version of ESS on your 
local and remote systems.

In particular, newer and older versions of `.ess.eval' have different 
arguments. You can check this by running `args(.ess.eval)' in the session 
of your remote server and in a session on your local system.

In any case, I'd suggest asking on ess-help@r-project.org for help in 
fixing this.

HTH,

Chuck

Re: Using tramp to run R source code blocks on remote server

[Charles C. Berry]

On Thu, 21 Apr 2016, Charles C. Berry wrote:

> On Thu, 21 Apr 2016, Vikas Rawal wrote:
>
>> I am trying to run R source code blocks on a remote server (defined in my 
>> ~/.ssh/config as cesp).
>> 
>> I am facing several problems.
>> 
>> 1.  The following sample block, when used with ":results value" does not 
>> give any results.
>> 
>> #+NAME: level1
>> #+begin_src R :results value :exports results :colnames yes :hline yes 
>> :session cesp :dir /cesp:/home/vikas/
>> c(1:5)
>> #+end_src
>> 
>> I get the following message:
>> 
>> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>> 
>
> I think this is an issue with having different version of ESS on your local 
> and remote systems.
>

OOPS!

The `different versions' issue affected my setup. But after 
correcting it, I still get the error you cited above.

Not sure exactly where the fault lies.

Chuck

Re: Using tramp to run R source code blocks on remote server

[Charles C. Berry]

On Thu, 21 Apr 2016, Charles C. Berry wrote:

> On Thu, 21 Apr 2016, Charles C. Berry wrote:
>
>> On Thu, 21 Apr 2016, Vikas Rawal wrote:
>> 
>>> I am trying to run R source code blocks on a remote server (defined in my 
>>> ~/.ssh/config as cesp).
>>> 
>>> I am facing several problems.
>>> 
>>> 1.  The following sample block, when used with ":results value" does not 
>>> give any results.
>>> 
>>> #+NAME: level1
>>> #+begin_src R :results value :exports results :colnames yes :hline yes 
>>> :session cesp :dir /cesp:/home/vikas/
>>> c(1:5)
>>> #+end_src
>>> 
>>> I get the following message:
>>> 
>>> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>>> 
>> 
>> I think this is an issue with having different version of ESS on your local 
>> and remote systems.
>> 
>
> OOPS!
>
> The `different versions' issue affected my setup. But after correcting it, I 
> still get the error you cited above.
>
> Not sure exactly where the fault lies.
>


Maybe in `org-babel-comint-eval-invisibly-and-wait-for-file', but I am not 
sure what/how.

I tried to instrument the function (C-u C-M-x on the function source) and 
run it. And it ran without error when I stepped thru it. Then I removed 
instrumentation with C-M-x and reran - again no error.

I tried removing the *.elc and restarting emacs, but that seems to make no 
difference - the error is still there till I instrument the code.

Until this is sorted out, you might try the instrument/uninstrument gambit 
as a workaround.

HTH,

Chuck

Re: Using tramp to run R source code blocks on remote server

[Cook, Malcolm]

I believe some of the inter-process communication magic depends upon i/o through a file that exists in /tmp on the remote host

If the remote server shares a file system with localhost (does it?), you could contrive for that temp file to live there (~/.emacs.d/ESS.IPC.HACK) instead.

I've been stung by similar in the past.

What version of ESS and org are you using? (meta-x org-version and meta-x ess-version) - this has been a bit of a moving target.  I've most recently had tramp/ess/org work FINE.  That is when I am editing a file on a remote host via tramp.  The R process starts up on the remote host (as does a shell for "#+SRC sh" blocks).   This _should_ work for you....

~Malcolm

 > -----Original Message-----
 > From: Emacs-orgmode [mailto:emacs-orgmode-
 > bounces+mec=stowers.org@gnu.org] On Behalf Of Charles C. Berry
 > Sent: Thursday, April 21, 2016 5:05 PM
 > To: Vikas Rawal <vikaslists@agrarianresearch.org>
 > Cc: org-mode mailing list <emacs-orgmode@gnu.org>
 > Subject: Re: [O] Using tramp to run R source code blocks on remote server
 > 
 > On Thu, 21 Apr 2016, Charles C. Berry wrote:
 > 
 > > On Thu, 21 Apr 2016, Charles C. Berry wrote:
 > >
 > >> On Thu, 21 Apr 2016, Vikas Rawal wrote:
 > >>
 > >>> I am trying to run R source code blocks on a remote server (defined in my
 > >>> ~/.ssh/config as cesp).
 > >>>
 > >>> I am facing several problems.
 > >>>
 > >>> 1.  The following sample block, when used with ":results value" does not
 > >>> give any results.
 > >>>
 > >>> #+NAME: level1
 > >>> #+begin_src R :results value :exports results :colnames yes :hline yes
 > >>> :session cesp :dir /cesp:/home/vikas/
 > >>> c(1:5)
 > >>> #+end_src
 > >>>
 > >>> I get the following message:
 > >>>
 > >>> tramp-flush-directory-property: Wrong type argument: arrayp, nil
 > >>>
 > >>
 > >> I think this is an issue with having different version of ESS on your local
 > >> and remote systems.
 > >>
 > >
 > > OOPS!
 > >
 > > The `different versions' issue affected my setup. But after correcting it, I
 > > still get the error you cited above.
 > >
 > > Not sure exactly where the fault lies.
 > >
 > 
 > 
 > Maybe in `org-babel-comint-eval-invisibly-and-wait-for-file', but I am not
 > sure what/how.
 > 
 > I tried to instrument the function (C-u C-M-x on the function source) and
 > run it. And it ran without error when I stepped thru it. Then I removed
 > instrumentation with C-M-x and reran - again no error.
 > 
 > I tried removing the *.elc and restarting emacs, but that seems to make no
 > difference - the error is still there till I instrument the code.
 > 
 > Until this is sorted out, you might try the instrument/uninstrument gambit
 > as a workaround.
 > 
 > HTH,
 > 
 > Chuck
 > 

Re: Using tramp to run R source code blocks on remote server

[Vikas Rawal]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]


> 
>> I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as cesp).
>> 
>> I am facing several problems.
>> 
>> 1.  The following sample block, when used with ":results value" does not give any results.
>> 
>> #+NAME: level1
>> #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:/home/vikas/
>> c(1:5)
>> #+end_src
>> 
>> I get the following message:
>> 
>> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>> 
> 
> I think this is an issue with having different version of ESS on your local and remote systems.
> 
> In particular, newer and older versions of `.ess.eval' have different arguments. You can check this by running `args(.ess.eval)' in the session of your remote server and in a session on your local system.
> 
> In any case, I'd suggest asking on ess-help@r-project.org <mailto:ess-help@r-project.org> for help in fixing this.
> 


Thanks for this. I did not think that ess on the remote system would be needed at all. I just want emacs+ess on my system to use R on the remote system.

Having to maintain version compatibility between every machine, and across multiple users, is an impossible situation.

But let me see what folks at ess-help have to say.

Thanks very much,

Vikas




[-- Attachment #2: Type: text/html, Size: 6917 bytes --]

Re: Using tramp to run R source code blocks on remote server

[Nick Dokos]

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

>         I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as
>         cesp).
>        
>         I am facing several problems.
>        
>         1.  The following sample block, when used with ":results value" does not give any results.
>        
>         #+NAME: level1
>         #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:
>         /home/vikas/
>         c(1:5)
>         #+end_src
>        
>         I get the following message:
>        
>         tramp-flush-directory-property: Wrong type argument: arrayp, nil
>

You might also want to ping Michael Albinus on the tramp mailing list about
this.

--
Nick

>     I think this is an issue with having different version of ESS on your local and remote systems.
>    
>     In particular, newer and older versions of `.ess.eval' have different arguments. You can check this
>     by running `args(.ess.eval)' in the session of your remote server and in a session on your local
>     system.
>    
>     In any case, I'd suggest asking on ess-help@r-project.org for help in fixing this.
>
> Thanks for this. I did not think that ess on the remote system would be needed at all. I just want
> emacs+ess on my system to use R on the remote system.
>
> Having to maintain version compatibility between every machine, and across multiple users, is an
> impossible situation.
>
> But let me see what folks at ess-help have to say.
>
> Thanks very much,
>
> Vikas

Re: Using tramp to run R source code blocks on remote server

[Charles C. Berry]

On Fri, 22 Apr 2016, Nick Dokos wrote:

> Vikas Rawal <vikaslists@agrarianresearch.org> writes:
>
>>         I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as
>>         cesp).
>>
>>         I am facing several problems.
>>
>>         1.  The following sample block, when used with ":results value" does not give any results.
>>
>>         #+NAME: level1
>>         #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:
>>         /home/vikas/
>>         c(1:5)
>>         #+end_src
>>
>>         I get the following message:
>>
>>         tramp-flush-directory-property: Wrong type argument: arrayp, nil
>>
>
> You might also want to ping Michael Albinus on the tramp mailing list about
> this.
>


And contrary to what I said earlier this *is* related to compilation. Not 
sure if this is a tramp issue for ob-comint.el issue.

Details:

If I run org-reload uncompiled, C-u C-c C-x !, code like

#+BEGIN_SRC R :session :dir /user@remote:/userdir
c(3,2,6)
#+END_SRC

runs fine and produces the expected result.

But reloading compiled, C-c C-x !, the code produces the

: tramp-flush-directory-property: Wrong type argument: arrayp, nil

message.

HTH,

Chuck

Re: Using tramp to run R source code blocks on remote server

[Cook, Malcolm]

> On Fri, 22 Apr 2016, Nick Dokos wrote:
 > 
 > > Vikas Rawal <vikaslists@agrarianresearch.org> writes:
 > >
 > >>         I am trying to run R source code blocks on a remote server (defined in
 > my ~/.ssh/config as
 > >>         cesp).
 > >>
 > >>         I am facing several problems.
 > >>
 > >>         1.  The following sample block, when used with ":results value" does
 > not give any results.
 > >>
 > >>         #+NAME: level1
 > >>         #+begin_src R :results value :exports results :colnames yes :hline yes
 > :session cesp :dir /cesp:
 > >>         /home/vikas/
 > >>         c(1:5)
 > >>         #+end_src
 > >>
 > >>         I get the following message:
 > >>
 > >>         tramp-flush-directory-property: Wrong type argument: arrayp, nil
 > >>
 > >
 > > You might also want to ping Michael Albinus on the tramp mailing list about
 > > this.
 > >
 > 
 > 
 > And contrary to what I said earlier this *is* related to compilation. Not
 > sure if this is a tramp issue for ob-comint.el issue.

I do not trust org-reload.  If I want a fresh org environment, I quit and restart emacs.

Also, when I am playing with effect of (re)compiling org, I don't compile it with org already loaded.  I compile it in a fresh emacs.

I don't trust results gained other ways.

YMMV

 > 
 > Details:
 > 
 > If I run org-reload uncompiled, C-u C-c C-x !, code like
 > 
 > #+BEGIN_SRC R :session :dir /user@remote:/userdir
 > c(3,2,6)
 > #+END_SRC
 > 
 > runs fine and produces the expected result.
 > 
 > But reloading compiled, C-c C-x !, the code produces the
 > 
 > : tramp-flush-directory-property: Wrong type argument: arrayp, nil
 > 
 > message.
 > 
 > HTH,
 > 
 > Chuck

Re: Using tramp to run R source code blocks on remote server

[Tim Howes]

On 4/21/16 3:53 AM, Vikas Rawal wrote:
> 
> 2. Is there a way to be able to use screen/byobu with remote processing?

This thread suggests using dtach:
http://grokbase.com/t/r/ess-help/1536b42tjy/ess-remote-r

[PATCH] ob-comint.el bug WAS: Re: Using tramp to run R source code blocks on remote server

[Charles C. Berry]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

On Thu, 21 Apr 2016, Vikas Rawal wrote:

> I am trying to run R source code blocks on a remote server (defined in my ~/.ssh/config as cesp).
>
> I am facing several problems.
>
> 1.  The following sample block, when used with ":results value" does not give any results.
>
> #+NAME: level1
> #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:/home/vikas/
> c(1:5)
> #+end_src
>
> I get the following message:
>
> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>

This was a near trivial byte-compiler issue.

`tramp-flush-directory-property' is in tramp-cache.el not tramp.el.

I'll push it tomorrow.

Chuck

[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 1171 bytes --]

From b108391e0ba56e0a272aae1d1fb2eed4f1c4a03c Mon Sep 17 00:00:00 2001
From: Charles Berry <ccberry@ucsd.edu>
Date: Fri, 22 Apr 2016 21:04:16 -0700
Subject: [PATCH] ob-comint.el: Fix byte-compiler/declare-function bug

* ob-comint.el: (tramp-flush-directory-property declaration): The
  file is now "tramp-cache".

Bug reported by

From: Vikas Rawal
Subject: Using tramp to run R source code blocks on remote server
Archived-At: <http://permalink.gmane.org/gmane.emacs.orgmode/106557>
---
 lisp/ob-comint.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index a5a7888..833d492 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -35,7 +35,7 @@
 (require 'comint)
 (eval-when-compile (require 'cl))
 (declare-function with-parsed-tramp-file-name "tramp" (filename var &rest body))
-(declare-function tramp-flush-directory-property "tramp" (vec directory))
+(declare-function tramp-flush-directory-property "tramp-cache" (vec directory))
 
 (defun org-babel-comint-buffer-livep (buffer)
   "Check if BUFFER is a comint buffer with a live process."
-- 
2.6.4 (Apple Git-63)

Re: Using tramp to run R source code blocks on remote server

[Benda Xu]

Hi Vikas,

I cannot address your problem directly.  I have a workaround.

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

> I am trying to run R source code blocks on a remote server (defined in
> my ~/.ssh/config as cesp).
>
> I am facing several problems.
>
> 1.  The following sample block, when used with ":results value" does
> not give any results.
>
> #+NAME: level1
> #+begin_src R :results value :exports results :colnames yes :hline yes :session cesp :dir /cesp:/home/vikas/
> c(1:5)
> #+end_src
>
> I get the following message:
>
> tramp-flush-directory-property: Wrong type argument: arrayp, nil
>
> I get the expected output if I use ":results output”, but I don’t want
> that.

I have gone through similar pains as you did.

> 2. Is there a way to be able to use screen/byobu with remote
> processing? I have tried various options of using shell, term, or
> ansi-term. Each of them gives some problem or the other. Does anyone
> have experience with using any of these?

This might be possible with the future async executions.

For the moment, I have settled down with the following solution:

  1. Tangle each source blocks as a script to the remote host.

  2. Tangle a Makefile to manage the execution, e.g. how the scripts are
     executed with which arguments.

  3. Login the remote host and open a tmux session and run "make" in the
     tangled directory.

With this workflow, all the programs are managed by org-mode, and they
are executed asychronously.  At the same time, I lose the flexibility to
run the scripts in a org spawned session, which is a trade-off.

Cheers,
Benda