Erroneous "No such file or directory" with babel and remote dir

Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Dear List,

I have updated from 7.8.something to 7.9.1 and evaluating the following
source block:

,----------------------------------------------------------------------------------
| #+header: :cache no :eval query
| #+name: sacct-output
| #+begin_src sh  :exports both :dir /root@xxxxxxxxxx:
| module load shared slurm
| sacct -d -S 2011-11-01 | grep JOB_TERMINATED | cut -d ' ' -f 4,23 | grep -v ' 0$'
| #+end_src
`----------------------------------------------------------------------------------

used to work, but now produces the error:

,---------------------------------------------------------------------------------
| bin/bash: /scp:root@xxxxxxxxxx:/tmp/sh-script-2829ZfG: No such file or directory
`---------------------------------------------------------------------------------

Have I overlooked some change?

Cheers,

Loris

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

> Dear List,
>
> I have updated from 7.8.something to 7.9.1 and evaluating the following
> source block:
>
> ,----------------------------------------------------------------------------------
> | #+header: :cache no :eval query
> | #+name: sacct-output
> | #+begin_src sh  :exports both :dir /root@xxxxxxxxxx:
> | module load shared slurm
> | sacct -d -S 2011-11-01 | grep JOB_TERMINATED | cut -d ' ' -f 4,23 | grep -v ' 0$'
> | #+end_src
> `----------------------------------------------------------------------------------
>
> used to work, but now produces the error:
>
> ,---------------------------------------------------------------------------------
> | bin/bash: /scp:root@xxxxxxxxxx:/tmp/sh-script-2829ZfG: No such file or directory
> `---------------------------------------------------------------------------------

I should mention here that the file does in fact exist and does the
right thing when executed on the remote host.

> Have I overlooked some change?
>
> Cheers,
>
> Loris
>
>
>
>
>

-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett@fu-berlin.de

Re: Erroneous "No such file or directory" with babel and remote dir

[Bastien]

Hi Loris,

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

> I have updated from 7.8.something to 7.9.1 and evaluating the following
> source block:
>
> ,----------------------------------------------------------------------------------
> | #+header: :cache no :eval query
> | #+name: sacct-output
> | #+begin_src sh  :exports both :dir /root@xxxxxxxxxx:
> | module load shared slurm
> | sacct -d -S 2011-11-01 | grep JOB_TERMINATED | cut -d ' ' -f 4,23 | grep -v ' 0$'
> | #+end_src
> `----------------------------------------------------------------------------------
>
> used to work, but now produces the error:

Can you try to bisect and find the bad commit (or a set of 
suspicious ones)?

Thanks,

-- 
 Bastien

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Bastien <bzg@altern.org> writes:

> Hi Loris,
>
> "Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>
>> I have updated from 7.8.something to 7.9.1 and evaluating the following
>> source block:
>>
>> ,----------------------------------------------------------------------------------
>> | #+header: :cache no :eval query
>> | #+name: sacct-output
>> | #+begin_src sh  :exports both :dir /root@xxxxxxxxxx:
>> | module load shared slurm
>> | sacct -d -S 2011-11-01 | grep JOB_TERMINATED | cut -d ' ' -f 4,23 | grep -v ' 0$'
>> | #+end_src
>> `----------------------------------------------------------------------------------
>>
>> used to work, but now produces the error:
>
> Can you try to bisect and find the bad commit (or a set of 
> suspicious ones)?
>
> Thanks,

How do I avoid the mixed installation problem when testing with a clone
of the org repository?  My version is 

,----------------------------------------------------------------
| Org-mode version 7.9.1 (release_7.9.1-git @ mixed installation!
| /usr/local/share/emacs/site-lisp/ and
| /home/loris/git/org-mode.git/lisp/)
`----------------------------------------------------------------

Loris
-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Memnon Anon]

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

> How do I avoid the mixed installation problem when testing with a clone
> of the org repository? 

,----[ http://orgmode.org/worg/org-faq.html#mixed-install ]
| Among the most common reasons is Orgmode gets loaded before the
| load-path variable is updated to include the installation directory of
| the latest Orgmode. To avoid issues like this, it is recommended that
| the load path is updated very early on in your init file. 
`----
 
hth 
Memnon

Re: Erroneous "No such file or directory" with babel and remote dir

[Achim Gratz]

Loris Bennett writes:
> How do I avoid the mixed installation problem when testing with a clone
> of the org repository?  My version is 
>
> ,----------------------------------------------------------------
> | Org-mode version 7.9.1 (release_7.9.1-git @ mixed installation!
> | /usr/local/share/emacs/site-lisp/ and
> | /home/loris/git/org-mode.git/lisp/)
> `----------------------------------------------------------------

Assuming that you've set up the load-path correctly, this should correct
itself when you do a

make autoloads

in the Git working tree.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Achim Gratz <Stromeko@nexgo.de> writes:

> Loris Bennett writes:
>> How do I avoid the mixed installation problem when testing with a clone
>> of the org repository?  My version is 
>>
>> ,----------------------------------------------------------------
>> | Org-mode version 7.9.1 (release_7.9.1-git @ mixed installation!
>> | /usr/local/share/emacs/site-lisp/ and
>> | /home/loris/git/org-mode.git/lisp/)
>> `----------------------------------------------------------------
>
> Assuming that you've set up the load-path correctly, this should correct
> itself when you do a
>
> make autoloads
>
> in the Git working tree.
>
>
> Regards,
> Achim.

Thanks, Achim, that worked.  But what exactly does it do?

Cheers,

Loris

-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Bastien]

Hi Loris,

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

> Thanks, Achim, that worked.  But what exactly does it do?

let me try -- I'm confident Achim will correct me if I'm wrong
or incomplete.

`make autoloads' creates two files in your lisp/ directory:

- org-install.el
- org-version.el

The first one declares the functions and macros that should be
autoloaded with this version of Org.  Those functions/macros are
always accessible, even when the library that contain them is not
explicitely loaded.  Calling one of those functions/macros will
load the library.  See the Emacs manual for further details.

The second one defines the correct version for your Org distrib
and it is further checked by (org-version).

If it does not exist, systems that have Git will try to infer the
version number from Git (checking against the latest tag, which is
always of the form "release_X.X[.X]".

If org-version.el has not been created and Git is not available,
you will end up with a "N/A" version number.

Note that org-version.el and org-install.el are both included in
the .tar.gz/.zip distribution files.

HTH,

-- 
 Bastien

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Bastien <bzg@altern.org> writes:

> Hi Loris,
>
> "Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>
>> I have updated from 7.8.something to 7.9.1 and evaluating the following
>> source block:
>>
>> ,----------------------------------------------------------------------------------
>> | #+header: :cache no :eval query
>> | #+name: sacct-output
>> | #+begin_src sh  :exports both :dir /root@xxxxxxxxxx:
>> | module load shared slurm
>> | sacct -d -S 2011-11-01 | grep JOB_TERMINATED | cut -d ' ' -f 4,23 | grep -v ' 0$'
>> | #+end_src
>> `----------------------------------------------------------------------------------
>>
>> used to work, but now produces the error:
>
> Can you try to bisect and find the bad commit (or a set of 
> suspicious ones)?
>
> Thanks,

552b0edb254a104e441e28f3a942dc6005e97f87 is the first bad commit
commit 552b0edb254a104e441e28f3a942dc6005e97f87
Author: Bastien Guerry <bzg@altern.org>
Date:   Sat Mar 17 15:44:41 2012 +0100

    Manually revert to 78ec8e21 (Release 7.8.04)
    
    Let's take a fresh start.  Sorry Mama.

:040000 040000 163e6eefeae7ce8287a9b95376a182725463b6e4 00f1fc024be6f61d8f7b520be509d46555418adb M	EXPERIMENTAL
:100644 100644 1753dd8d44eab0fc512284172e8902302959bbab 1022cdc9ff14c9d7f3004c4181507acf9f7414a2 M	Makefile
:100644 100644 ad822658a8954b7d02b79f2cccd9c9b721b3e889 cf67c9933ccbc13676b6b9ded82bb62c3ab67220 M	README_DIST
:040000 040000 35c93a0ab0b246483543b14ba3cb218dae9172e1 46f1a162838f9234bf0f34ee202ad88690af8e5e M	contrib
:040000 040000 e680a29de46430d26ee79a24abd7ec4f6cec28ba c34f8f62ceced35f6519935bbe0404b95a6d5b4d M	doc
:040000 040000 5989cfb368b5bee95e1fcceb42a88c381d72bdee 012e82d1831a339b52148716cd9a1cad7d3159f9 M	etc
:040000 040000 7ef0c8f2ecdd18ef48acfc0e13815ac6e8b01d64 d2895eead691b78d01435a0c9ba45c161e462f39 M	lisp
:040000 040000 99d7331b0f659fe1b1e6323cb5a33c66152e7d67 e2e86614d689cef98523b2d6f6c8f3b1dea5ddff M	testing

Loris
-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Achim Gratz]

Loris Bennett writes:
>> Can you try to bisect and find the bad commit (or a set of 
>> suspicious ones)?
>
> 552b0edb254a104e441e28f3a942dc6005e97f87 is the first bad commit
> commit 552b0edb254a104e441e28f3a942dc6005e97f87
> Author: Bastien Guerry <bzg@altern.org>
> Date:   Sat Mar 17 15:44:41 2012 +0100

This is not the commit you're looking for...

Please "skip" these commits after starting the bisect by doing

git bisect skip 7e903acccd..df82832fb7

If the offending change is inside that range we can still deal with it
later.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Erroneous "No such file or directory" with babel and remote dir

[Nick Dokos]

Achim Gratz <Stromeko@nexgo.de> wrote:

> Loris Bennett writes:
> >> Can you try to bisect and find the bad commit (or a set of 
> >> suspicious ones)?
> >
> > 552b0edb254a104e441e28f3a942dc6005e97f87 is the first bad commit
> > commit 552b0edb254a104e441e28f3a942dc6005e97f87
> > Author: Bastien Guerry <bzg@altern.org>
> > Date:   Sat Mar 17 15:44:41 2012 +0100
> 
> This is not the commit you're looking for...
> 
> Please "skip" these commits after starting the bisect by doing
> 
> git bisect skip 7e903acccd..df82832fb7
> 
> If the offending change is inside that range we can still deal with it
> later.
> 
> 

I sent mail to Eric with some partial findings about this, but I guess I
should have sent it to the list as well. Here is what I sent to Eric
(slightly edited by adding a few commas):

--8<---------------cut here---------------start------------->8---
Eric,

I went back to the above problem (see
http://thread.gmane.org/gmane.emacs.orgmode/57152/focus=60469 and the
associated thread) and I think I have taken it a bit further.

5cb80c7e5b9bca introduced the following bit of code at the end of org-babel-sh-evaluate:

,----
|      ('otherwise				; external shell script
| -     (org-babel-eval org-babel-sh-command (org-babel-trim body))))))
| +     (if (cdr (assoc :shebang params))
| +	 (let ((script-file (org-babel-temp-file "sh-script-"))
| +	       (shebang (cdr (assoc :shebang params)))
| +	       (padline (not (string= "no" (cdr (assoc :padline params))))))
| +	   (with-temp-file script-file
| +	     (when shebang (insert (concat shebang "\n")))
| +	     (when padline (insert "\n"))
| +	     (insert body))
| +	   (set-file-modes script-file #o755)
| +	   (org-babel-eval script-file ""))
| +       (org-babel-eval org-babel-sh-command (org-babel-trim body)))))))
`----

So before, there was no if and we went straight into the org-babel-eval.
Now, the ``if'' seems to succeed all the time: the (cdr ..) returns "" with
Loris's example and (org-babel-eval script-file "") returns nil.

If I modify the condition so that it takes the false path and executes
the (org-babel-eval org-babel-sh-command (org-babel-trim body)) form,
it works.

I have no idea if this is correct - but it does seem to me that in this
particular case, it's taking the wrong branch: in particular, if no
shebang is specified, it should go the way it used to before this
change. Of course, there is also the problem of specifying a
shebang *and* a remote dir which is not handled correctly by this
modification.

For the record, the change I made was:

-     (if (cdr (assoc :shebang params))
+     (if (not (string= (cdr (assoc :shebang params)) ""))

It's at the very least incomplete and it may be wrong, but I thought
I'd let you know.
--8<---------------cut here---------------end--------------->8---

HTH,
Nick

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Achim Gratz <Stromeko@nexgo.de> writes:

> Loris Bennett writes:
>>> Can you try to bisect and find the bad commit (or a set of 
>>> suspicious ones)?
>>
>> 552b0edb254a104e441e28f3a942dc6005e97f87 is the first bad commit
>> commit 552b0edb254a104e441e28f3a942dc6005e97f87
>> Author: Bastien Guerry <bzg@altern.org>
>> Date:   Sat Mar 17 15:44:41 2012 +0100
>
> This is not the commit you're looking for...
>
> Please "skip" these commits after starting the bisect by doing
>
> git bisect skip 7e903acccd..df82832fb7
>
> If the offending change is inside that range we can still deal with it
> later.
>
>
> Regards,
> Achim.

OK.  I don't really get what is going on here, as this is my first
attempt at bisecting.  The bad commit does seem to be among those
skipped:

$ git bisect bad
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
095e5786761d4edd51dfdfc9a69cd22b942096fa
b0e24a2b5e3ce872e9eb43462f03edf616438f57
cdd9e2b91a7b9257e4f94e0aacc8e470ac1c951a
73cbeabe1b457da1364547d7f2d89db889bec926
9382faf5927f84a2c496ae677d99b64517abeeb2
05847efd4ace13fda174c2f1d8dc354eb217ea35
c63275ca8ea46e0e96ab7ed70efe6bb937c39b29
82019bc7c7a786303e82cfa5891193acb3754606
b2913c9227504ffd31d616482d258f5b10f821fb
49dd6de08384f913f014ac06c24da4ff08ccdb6a
dab84afa7592e842ae866881c532cd60944bb0da
552b0edb254a104e441e28f3a942dc6005e97f87
b32d2f737d567222bdbdf742329b9a34d0dd069a
a1a01224d4c1f76761e3562c54020c04bfd3ac9b
ecd0562c5f6e69ce0e4546d9f64d8e89eda8eef1
5ed0384e51449e6462eed3a6ff4ecba6a6f52790
6bcba149195c2c281eefc3065dabe2ec1fdea687
4e5c24ea81c6758241a828e28ea44b75d36115e3
856f5c27fb11a1f034c80fc29c39f211434fc650
06b94ca884733eaefb0ec868d641da78ed536ceb
df82832fb7525e5d57f1e968334e1057329e95f2
6c911234ac66d62d99cb0c996dc10c938b2b3679
We cannot bisect more!


-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett@fu-berlin.de

Re: Erroneous "No such file or directory" with babel and remote dir

[Achim Gratz]

Loris Bennett writes:
> OK.  I don't really get what is going on here, as this is my first
> attempt at bisecting.  The bad commit does seem to be among those
> skipped:

I've done a diff around those commits and there seems nothing related to
your problem.  So I'm afraid you've marked another commit as "bad" that
rather should have been "skip"ped.  The commits you "skip" have
problems, but unrelated to the bug you are hunting — it's important to
mark them correctly.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Achim Gratz <Stromeko@nexgo.de> writes:

> Loris Bennett writes:
>> OK.  I don't really get what is going on here, as this is my first
>> attempt at bisecting.  The bad commit does seem to be among those
>> skipped:
>
> I've done a diff around those commits and there seems nothing related to
> your problem.  So I'm afraid you've marked another commit as "bad" that
> rather should have been "skip"ped.  The commits you "skip" have
> problems, but unrelated to the bug you are hunting — it's important to
> mark them correctly.
>
>
> Regards,
> Achim.

OK, I must have goofed.  What I did after starting the bisection was

- run 'make autoload'
- open test file in emacs with minimal .emacs
- test
- end emacs
- mark bisection good or bad

I then repeated this for the next commit.

Is something missing here, or did I just incorrectly mark a commit?

Cheers,

Loris

-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Achim Gratz]

Loris Bennett writes:
> OK, I must have goofed.  What I did after starting the bisection was
>
> - run 'make autoload'
> - open test file in emacs with minimal .emacs
> - test
> - end emacs
> - mark bisection good or bad
>
> I then repeated this for the next commit.

This would be the correct thing to do, yes.

> Is something missing here, or did I just incorrectly mark a commit?

I really don't know, I'm just guessing… :-)

It may be more informative if you traced the invocation of the remote
execution in both a failing and a working version in the debugger and
look for any differences as the arguments are cobbled together.  Once
you find where an how these diverge it would be much easier to find the
code that is responsible for it (if it wasn't glaringly obvious from the
trace already).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Achim Gratz <Stromeko@nexgo.de> writes:

> Loris Bennett writes:
>> OK, I must have goofed.  What I did after starting the bisection was
>>
>> - run 'make autoload'
>> - open test file in emacs with minimal .emacs
>> - test
>> - end emacs
>> - mark bisection good or bad
>>
>> I then repeated this for the next commit.
>
> This would be the correct thing to do, yes.
>
>> Is something missing here, or did I just incorrectly mark a commit?
>
> I really don't know, I'm just guessing… :-)
>
> It may be more informative if you traced the invocation of the remote
> execution in both a failing and a working version in the debugger and
> look for any differences as the arguments are cobbled together.  Once
> you find where an how these diverge it would be much easier to find the
> code that is responsible for it (if it wasn't glaringly obvious from the
> trace already).
>
>
> Regards,
> Achim.

So just as a reminder:

Back in the days of release_7.01h, the following used to work:

,------------------------------------------
| #+begin_src sh :dir /loris@othercomputer:
| hostname
| #+end_src
`------------------------------------------

Currently it produces the error:

,---------------------------------------------------
| apply: Wrong type argument: number-or-marker-p, ""
`---------------------------------------------------

I have had another go at bisecting this problem and came up with this:

,-------------------------------------------------------------------------------------------------------------
| 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
| commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
| Author: Dan Davison <davison@stats.ox.ac.uk>
| Date:   Mon Aug 30 09:34:05 2010 -0700
| 
|     babel: Fix temporary file processing in the remote execution case.
|     
|         * ob.el (org-babel-temp-file): Don't use babel temporary
|         directory in remote case; use make-temp-file with remote file
|         name so that temp file is guaranteed not to exist previously
|         on remote machine.
|         (org-babel-tramp-localname): New function to return local name
|         portion of possibly remote file specification
|     
|         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
|         changes in `org-babel-temp-file'; pass local file name to
|         remote R process.
|         (org-babel-R-evaluate-session) Respond to
|         changes in `org-babel-temp-file'; pass local file name to
|         remote R process.
| 
| :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
`-------------------------------------------------------------------------------------------------------------

Does that help anyone any further?

Cheers,

Loris

PS: I am amazed there aren't any more peasants like myself with torches
and pitchforks beating on the gates of Castle Org about this - remote
execution is/was such great feature!

-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Nick Dokos]

Loris Bennett <loris.bennett@fu-berlin.de> wrote:

> Achim Gratz <Stromeko@nexgo.de> writes:
> 
> > Loris Bennett writes:
> >> OK, I must have goofed.  What I did after starting the bisection was
> >>
> >> - run 'make autoload'
> >> - open test file in emacs with minimal .emacs
> >> - test
> >> - end emacs
> >> - mark bisection good or bad
> >>
> >> I then repeated this for the next commit.
> >
> > This would be the correct thing to do, yes.
> >
> >> Is something missing here, or did I just incorrectly mark a commit?
> >
> > I really don't know, I'm just guessing… :-)
> >
> > It may be more informative if you traced the invocation of the remote
> > execution in both a failing and a working version in the debugger and
> > look for any differences as the arguments are cobbled together.  Once
> > you find where an how these diverge it would be much easier to find the
> > code that is responsible for it (if it wasn't glaringly obvious from the
> > trace already).
> >
> >
> > Regards,
> > Achim.
> 
> So just as a reminder:
> 
> Back in the days of release_7.01h, the following used to work:
> 
> ,------------------------------------------
> | #+begin_src sh :dir /loris@othercomputer:
> | hostname
> | #+end_src
> `------------------------------------------
> 
> Currently it produces the error:
> 
> ,---------------------------------------------------
> | apply: Wrong type argument: number-or-marker-p, ""
> `---------------------------------------------------
> 
> I have had another go at bisecting this problem and came up with this:
> 
> ,-------------------------------------------------------------------------------------------------------------
> | 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
> | commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
> | Author: Dan Davison <davison@stats.ox.ac.uk>
> | Date:   Mon Aug 30 09:34:05 2010 -0700
> | 
> |     babel: Fix temporary file processing in the remote execution case.
> |     
> |         * ob.el (org-babel-temp-file): Don't use babel temporary
> |         directory in remote case; use make-temp-file with remote file
> |         name so that temp file is guaranteed not to exist previously
> |         on remote machine.
> |         (org-babel-tramp-localname): New function to return local name
> |         portion of possibly remote file specification
> |     
> |         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
> |         changes in `org-babel-temp-file'; pass local file name to
> |         remote R process.
> |         (org-babel-R-evaluate-session) Respond to
> |         changes in `org-babel-temp-file'; pass local file name to
> |         remote R process.
> | 
> | :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
> `-------------------------------------------------------------------------------------------------------------
> 
> Does that help anyone any further?
> 

I don't think so: afaict, both 7.01h (which did not contain this patch)
and 7.02 (which did) do the right thing (at least with your simple
script above). Something else broke it.

[I've been wrong about these things before so before Achim asks, I
did

  git tag --contains 9c878a8290c071fbe5e97bc33c300ef2f07d6153

and used its output as a basis of the above statement. I think (hope?)
it's correct.]

> Loris
> 
> PS: I am amazed there aren't any more peasants like myself with torches
> and pitchforks beating on the gates of Castle Org about this - remote
> execution is/was such great feature!
> 

Agreed, but Dan Davison (the principal contributor of the remoting code)
is gone from Castle Org (as you put it), Eric S. is busy, and nobody
else has stepped up to the plate. I've made fitful attempts to debug it,
but have had no success: I've never looked at the code in detail and I
don't have much time either.

So how about becoming a resident of Castle Org rather than beating on
the gates? There's glory galore if you fix it - and you get to scratch
your itch too :-)

Nick

Re: Erroneous "No such file or directory" with babel and remote dir

[Nick Dokos]

Nick Dokos <nicholas.dokos@hp.com> wrote:

> > Back in the days of release_7.01h, the following used to work:
> > 
> > ,------------------------------------------
> > | #+begin_src sh :dir /loris@othercomputer:
> > | hostname
> > | #+end_src
> > `------------------------------------------
> > 
> > Currently it produces the error:
> > 
> > ,---------------------------------------------------
> > | apply: Wrong type argument: number-or-marker-p, ""
> > `---------------------------------------------------
> > 

BTW, I get no error with the (more or less) current version of
org I'm using:

Org-mode version 7.9.2 (release_7.9.2-582-g6d099e @ /home/nick/elisp/org-mode/lisp/)

I just get the wrong answer: instead of evaluating on the remote, babel
evaluates it locallly.

> > I have had another go at bisecting this problem and came up with this:
> > 
> > ,-------------------------------------------------------------------------------------------------------------
> > | 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
> > | commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
> > | Author: Dan Davison <davison@stats.ox.ac.uk>
> > | Date:   Mon Aug 30 09:34:05 2010 -0700
> > | 
> > |     babel: Fix temporary file processing in the remote execution case.
> > |     
> > |         * ob.el (org-babel-temp-file): Don't use babel temporary
> > |         directory in remote case; use make-temp-file with remote file
> > |         name so that temp file is guaranteed not to exist previously
> > |         on remote machine.
> > |         (org-babel-tramp-localname): New function to return local name
> > |         portion of possibly remote file specification
> > |     
> > |         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
> > |         changes in `org-babel-temp-file'; pass local file name to
> > |         remote R process.
> > |         (org-babel-R-evaluate-session) Respond to
> > |         changes in `org-babel-temp-file'; pass local file name to
> > |         remote R process.
> > | 
> > | :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
> > `-------------------------------------------------------------------------------------------------------------
> > 
> > Does that help anyone any further?
> > 
> 
> I don't think so: afaict, both 7.01h (which did not contain this patch)
> and 7.02 (which did) do the right thing (at least with your simple
> script above). Something else broke it.
> 

I checked successive versions from 7.01h onwards and found breakage
between release_7.8 and release_7.9. So I bisected and came up
with this as the bad commit:

,----
| commit 5cb80c7e5b9bcae180b799d2a49c78d529e029f0
| Author: Eric Schulte <eric.schulte@gmx.com>
| Date:   Mon Mar 12 13:23:53 2012 -0400
| 
|     apply :shebang and :padline to shell script execution
|     
|     * lisp/ob-sh.el (org-babel-execute:sh): Pass all params to subroutine.
|       (org-babel-sh-evaluate): Apply :shebang and :padline to shell script
|       execution.
`----

Reverting it caused a merge conflict that I didn't have the patience to
resolve.  But I made branches, one with this commit as its tip and one
with its predecessor:

    git checkout -b foo 5cb80c7
    git checkout -b bar de09874

Testing on foo gives me an error, testing on bar gives the correct
result.  So I'm pretty sure this commit introduced the problem. Eric has
applied a partial fix since then which gets rid of the error but still
gives the wrong answer (not Eric's fault: he followed my suggestion, but
I was treating the symptom, not the disease).

But I think this commit is the place that deserves more scrutiny.

HTH,
Nick

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Nick Dokos <nicholas.dokos@hp.com> wrote:
>
>> > Back in the days of release_7.01h, the following used to work:
>> > 
>> > ,------------------------------------------
>> > | #+begin_src sh :dir /loris@othercomputer:
>> > | hostname
>> > | #+end_src
>> > `------------------------------------------
>> > 
>> > Currently it produces the error:
>> > 
>> > ,---------------------------------------------------
>> > | apply: Wrong type argument: number-or-marker-p, ""
>> > `---------------------------------------------------
>> > 
>
> BTW, I get no error with the (more or less) current version of
> org I'm using:
>
> Org-mode version 7.9.2 (release_7.9.2-582-g6d099e @ /home/nick/elisp/org-mode/lisp/)
>
> I just get the wrong answer: instead of evaluating on the remote, babel
> evaluates it locallly.

I also get this using 7.9.2-78-gf48a8b-elpa and Emacs 24.2.50.1.

However, I then tested this on Emacs 23.1.1 with the following minimal
.emacs:

,-------------------------------------------------
| (add-to-list 'load-path "~/elisp/org-mode/lisp")
| (require 'org-install)
| 
| (org-babel-do-load-languages
|  'org-babel-load-languages
|  '((emacs-lisp . nil)
|    (R . t)
|    (sh . t)))
`-------------------------------------------------

and got:

,--------------------------------------------------------------------------
| OVERVIEW
| release_7.02
| Org-mode version 7.02 (release_7.02)
| Quit
| byte-code: Beginning of buffer [2 times]
| Loading tramp...done
| executing Sh code block...
| Wrote /tmp/tramp.6477nIt
| Tramp: Opening connection for loris@othercomputer using scp...
| Tramp: Waiting 60s for local shell to come up...
| Tramp: Sending command `ssh node003 -l loris  -q -e none && exit || exit'
| Tramp: Waiting for prompts from remote shell
| Tramp: Found remote shell prompt on `node003'
| Wrote /scp:loris@othercomputer:/tmp/tramp.64770Sz
| Wrote /tmp/tramp.6477N7U.6477AxO
| Wrote /tmp/scor6477a-m
| apply: Wrong type argument: number-or-marker-p, ""
`--------------------------------------------------------------------------

So there seem to be two problems, namely

1. Wrong type argument: number-or-marker-p, "", which only occurs in my
   minimal set-up post release_7.01h

2. No error, but local execution in the current version of Org.

So before I can get started looking at 2, I need to sort out 1

The way I am testing is 

,------------------------------------------------------------------
| git checkout release_7.02
| cd $HOME/elisp/org-mode && make clean
| cd $HOME/elisp/org-mode && make
| emacs -q -l /home/loris/.emacs_minimal /home/loris/tmp/test.org &
`------------------------------------------------------------------

Is there something wrong with this?

Cheers,

Loris


>> > I have had another go at bisecting this problem and came up with this:
>> > 
>> > ,-------------------------------------------------------------------------------------------------------------
>> > | 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
>> > | commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
>> > | Author: Dan Davison <davison@stats.ox.ac.uk>
>> > | Date:   Mon Aug 30 09:34:05 2010 -0700
>> > | 
>> > |     babel: Fix temporary file processing in the remote execution case.
>> > |     
>> > |         * ob.el (org-babel-temp-file): Don't use babel temporary
>> > |         directory in remote case; use make-temp-file with remote file
>> > |         name so that temp file is guaranteed not to exist previously
>> > |         on remote machine.
>> > |         (org-babel-tramp-localname): New function to return local name
>> > |         portion of possibly remote file specification
>> > |     
>> > |         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
>> > |         changes in `org-babel-temp-file'; pass local file name to
>> > |         remote R process.
>> > |         (org-babel-R-evaluate-session) Respond to
>> > |         changes in `org-babel-temp-file'; pass local file name to
>> > |         remote R process.
>> > | 
>> > | :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
>> > `-------------------------------------------------------------------------------------------------------------
>> > 
>> > Does that help anyone any further?
>> > 
>> 
>> I don't think so: afaict, both 7.01h (which did not contain this patch)
>> and 7.02 (which did) do the right thing (at least with your simple
>> script above). Something else broke it.
>> 
>
> I checked successive versions from 7.01h onwards and found breakage
> between release_7.8 and release_7.9. So I bisected and came up
> with this as the bad commit:
>
> ,----
> | commit 5cb80c7e5b9bcae180b799d2a49c78d529e029f0
> | Author: Eric Schulte <eric.schulte@gmx.com>
> | Date:   Mon Mar 12 13:23:53 2012 -0400
> | 
> |     apply :shebang and :padline to shell script execution
> |     
> |     * lisp/ob-sh.el (org-babel-execute:sh): Pass all params to subroutine.
> |       (org-babel-sh-evaluate): Apply :shebang and :padline to shell script
> |       execution.
> `----
>
> Reverting it caused a merge conflict that I didn't have the patience to
> resolve.  But I made branches, one with this commit as its tip and one
> with its predecessor:
>
>     git checkout -b foo 5cb80c7
>     git checkout -b bar de09874
>
> Testing on foo gives me an error, testing on bar gives the correct
> result.  So I'm pretty sure this commit introduced the problem. Eric has
> applied a partial fix since then which gets rid of the error but still
> gives the wrong answer (not Eric's fault: he followed my suggestion, but
> I was treating the symptom, not the disease).
>
> But I think this commit is the place that deserves more scrutiny.
>
> HTH,
> Nick

-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Nick Dokos]

Nick Dokos <nicholas.dokos@hp.com> wrote:


> I checked successive versions from 7.01h onwards and found breakage
> between release_7.8 and release_7.9. So I bisected and came up
> with this as the bad commit:
> 
> ,----
> | commit 5cb80c7e5b9bcae180b799d2a49c78d529e029f0
> | Author: Eric Schulte <eric.schulte@gmx.com>
> | Date:   Mon Mar 12 13:23:53 2012 -0400
> | 
> |     apply :shebang and :padline to shell script execution
> |     
> |     * lisp/ob-sh.el (org-babel-execute:sh): Pass all params to subroutine.
> |       (org-babel-sh-evaluate): Apply :shebang and :padline to shell script
> |       execution.
> `----
> 
> Reverting it caused a merge conflict that I didn't have the patience to
> resolve.  But I made branches, one with this commit as its tip and one
> with its predecessor:
> 
>     git checkout -b foo 5cb80c7
>     git checkout -b bar de09874
> 
> Testing on foo gives me an error, testing on bar gives the correct
> result.  So I'm pretty sure this commit introduced the problem. Eric has
> applied a partial fix since then which gets rid of the error but still
> gives the wrong answer (not Eric's fault: he followed my suggestion, but
> I was treating the symptom, not the disease).
> 
> But I think this commit is the place that deserves more scrutiny.
> 

The plot thickens. If I create a branch with 5cb80c7 as the tip
and then immediately apply the later patch that fixes the error, then
things are working as they should.

Here is the procedure in case somebody wants to try to replicate:


o First, save the later commit as a patch to apply:

  git show 86e515d > patch.to.apply

o Then go back in time:

  git checkout -b foo 5cb80c7

That gets you back to 7.8.03 and when I try the code block with

  emacs -q -l /path/to/minimal.emacs.that.requires.ob-sh /path/to/loris.org

I get an error:

/bin/bash: /scpc:nick@lefou.usa.hp.com:/tmp/sh-script-6207TKx: No such file or directory


o I then apply the patch:

  git apply patch.to.apply

and retry: I get the remote host name as I should.

Which tells me that the tmp file error is a red herring and the real
breakage occurred after 5cb80c7, probably through a commit that touched
ob-sh.el (although that's far from guaranteed). Here's that list:

$ git log --oneline -- lisp/ob-sh.el
86e515d fix remote execution w/empty shebang header arg
70dd119 Massive code clean-up.
966447c Don't use `org-flet' in ob-awk.el and ob-sh.el
8eb5843 Add punctuation at the end of the first line of docstrings.  Code cleanup.
63b5f8f replace flet/labels with org-flet/org-labels
ecd0562 Fix the master branch.
6e306f6 Fix copyright years in maint.
de42649 Manually revert maint to e85080.
73bb18b Manually revert to the Release 7.8.04 tag.
38c5045 Fix copyright years.
6e534f9 Manually revert back to commit e85080.
5cb80c7 apply :shebang and :padline to shell script execution
...


So unless somebody sees something wrong with the above procedure, we are a bit
closer to finding the culprit, but it's certainly not who I thought it was.

Nick

Re: Erroneous "No such file or directory" with babel and remote dir

[Nick Dokos]

In a previous mail, I wrote:

,----
| Which tells me that the tmp file error is a red herring and the real
| breakage occurred after 5cb80c7, probably through a commit that touched
| ob-sh.el (although that's far from guaranteed). Here's that list:
| 
| $ git log --oneline -- lisp/ob-sh.el
| 86e515d fix remote execution w/empty shebang header arg
| 70dd119 Massive code clean-up.
| 966447c Don't use `org-flet' in ob-awk.el and ob-sh.el
| 8eb5843 Add punctuation at the end of the first line of docstrings.  Code cleanup.
| 63b5f8f replace flet/labels with org-flet/org-labels
| ecd0562 Fix the master branch.
| 6e306f6 Fix copyright years in maint.
| de42649 Manually revert maint to e85080.
| 73bb18b Manually revert to the Release 7.8.04 tag.
| 38c5045 Fix copyright years.
| 6e534f9 Manually revert back to commit e85080.
| 5cb80c7 apply :shebang and :padline to shell script execution
| ...
`----

Continuing along these lines, I think I've figured it out and it's not
pretty.

The executive summary is that the last commit in the list introduced a
small bug that was fixed by the first commit in the list. But neither of
those has much to do with the remote-dir breakage (except that the bug
made things harder to bisect). The principal culprits are two sets of
commits for code transformations that were supposed to do nothing
functionally: they were just supposed to get away from using flet/labels
(which are deprecated and will be obsolete by emacs version 24.3.)

The end result is that one change fixes the remote dir problem in
Loris's example. I'm not sure that it solves every such problem though:
I haven't audited all the code.

The change is a one-liner: in org-babel-shell-command-on-region,
replace the line

		(call-process-region start end shell-file-name t
by
		(funcall call-process-region start end shell-file-name t

BTW, I'm using Org-mode version 7.9.2
(release_7.9.2-582-g6d099e.dirty @ /home/nick/elisp/org-mode/lisp/).

The rest of the email explains why the change is needed (in excruciating
detail: grab a beer or maybe a cup of coffee before starting on it.)

I hope this is (mostly) correct but corrections would be more than
welcome. The problem seems multi-faceted and therefore needs as many
eyes on it as possible.

Nick

P.S. I'm not sure whether to thank Loris or to curse him for pushing me
on this path, but there is no question that he is responsible for
finding the bug, providing the reproducer and then beating on the gates
with pitchfork and torch :-)

===========================================================================

* Root cause

The root cause of the problem was a set of code transformations that were
supposed to leave the functionality intact. The code transformations
were driven by the need to replace the flet/labels constructs which were
declared obsolete (as of 24.3 - they are still available, but cause warnings
to be issued).

The problem was hard to find because there were four commits (at widely varying
times) that contributed to various manifestations of the problem and
it was difficult to bisect. In chronological order, they were

- commit 5cb80c7 apply :shebang and :padline to shell script execution

  This did not cause the problem, but it introduced a bug that causes
  errors when executing a source block with a remote :dir spec. That
  confused the issue (at least it confused me: I fingered this as the
  culprit in a bisect, but it was only guilty of the bug fixed by
  commit 86e515d - see below -, not of the remote-dir problem).

- commit 63b5f8f replace flet/labels with org-flet/org-labels

  Section [[flet --> org-flet]] describes this.
  
- commit e85479a Don't use `org-flet' in some functions

  and several others that slowly got rid of org-flet in favor of let,
  and org-labels (somehow - I didn't check this carefully), the latter
  reverted and reapplied, presumably because problems were found and
  fixed in several iterations. I've only skimmed the surface here: I
  think this has the potential to be a minefield of problems waiting to
  explode - see the [[org-flet --> let]] section below.

- commit 86e515d fix remote execution w/empty shebang header arg

  This finally fixed the little bug that was introduced by 5cb80c7.

The first and the last of these commits are irrelevant to the remote-dir
problem, except that the bug gets in the way of testing: any version
later than 5cb80c7 exhibits the bug and that bug hides whether the
remote-dir problem is present or not.

The general procedure I followed was to make a branch with some commit
as its tip and then manually apply the patch of commit 86e515d. Only
then could I test for the remote-dir problem. In the following, when
I say commit X, I mean commit X *plus* the manually applied patch
from 86e515d.


** flet --> org-flet 
# <<flet --> org-flet>>

The first set of code transformations (implemented as commit
63b5f8f2e85b3059a2d30041db6939347a7a2d7d) dealt with the situation by
doing a mass substitution: flet --> org-flet and labels --> org-labels
(and in at least one case, flet --> org-labels to deal with a
recursive definition - I presume that was a preexisting bug that was
fixed by this substitution) and adding compatibility aliases in
org-compat.el to use the cl-flet/cl-labels macros from cl.el in emacs
versions >= 24.1.50.

I found out that already this broke the remote dir
functionality. Since this is a large but straightforward commit, I
split it up into separate patches for each affected file and applied
each patch in order: org-compat.el implements the org-flet/org-labels
functionality, but since the other patches were not applied yet, I was
still using flet/labels (obsolete but still present).  I then applied
the patch to org-macs.el, org.el and ob.el, testing after each one. At
first, things were working but applying the patch to ob.el broke
remote-dir. I then determined that most of the transformation in ob.el
were benign (at least as far as *this* problem is concerned), except
one: when I reverted that single one from org-flet back to flet,
remote-dir started working again.

The "bad" transformation was in this code segment in
org-babel-execute-src-block around line 544 in ob.el (but before you
go looking for it, be warned: you will not find it if you have a
reasonably current version of org - read the next section):

    
            ...
    	(unwind-protect
    	    (flet ((call-process-region (&rest args)
    		    (apply 'org-babel-tramp-handle-call-process-region args)))
    	      (org-flet ((lang-check (f)
    		       (let ((f (intern (concat "org-babel-execute:" f))))
    			 (when (fboundp f) f))))
            ...

  
If I left the first flet alone, remote-dir worked; if I change it to
org-flet it does not.

Here's the procedure I used to prove this to my satisfaction, just in
case anybody wants to try to duplicate my results:

     - create a branch with 63b5f8f as its tip and switch to it:

       git checkout -b foo 63b5f8f

     - make a patch for the "little" bug and apply it:

       git show 86e515d > patch.to.apply
       git apply patch.to.apply

     - Put Loris's example in a file loris.org (with appropriate modifications -
       user id and remote machine name):

       #+BEGIN_SRC sh :dir /user@host:
       hostname
       #+END_SRC

     - Add a (require 'ob-sh) to your minimal .emacs

     - Start emacs with

       emacs -q -l /path/to/minimal/.emacs /path/to/loris.org

       and C-c C-c in the code block. If it behaves as I expect,
       it should print the local hostname, not the remote one.

     - Now edit lisp/ob.el and change that single org-flet back to an flet
       (remember you are in a branch, so changes here won't affect the files
       in other branches) and test again: it should now print the remote
       hostname.

     - Before you can switch to some other branch, you'll need to either
       commit these changes or throw them away - I'll leave that up to you.

So the moral of the story is that the code transformations have *not*
left functionality unchanged. Something went awry but to be honest, I
don't know what. I didn't spend much time on it because of what I
found out next.


** org-flet --> let
# <<org-flet --> let>>
The second set of code transformations is more difficult to describe
because it's not just a (more or less) straight substitution: it tries
to completely eliminate the need for an flet-like construct and
replace it with a simple let.

The trouble here is that in many, but by no means all lisps, a symbol
can have *two* bindings - emacs lisp is one such lisp). The two are a
function binding and a value binding. Which binding is used depends on
the context: (f a b c) is usually rendered as "call the function f with
arguments a, b and c" but it really says "look up the function binding
of the symbol f and apply the resulting function to the values obtained
by looking up the value bindings of the symbols a, b and c". setq is the
usual way to set the value binding of a symbol and defun is the usual
way to set the function binding of a symbol. So you could do the
following (although it's probably a bad idea):

(defun f (x)
   (* x x))

to set the function binding of the symbol f and

(setq f 3)

to set the value binding of the symbol f. Then you could do

(f f)

calling the function in the function binding of f with argument the
value binding of the symbol f. Since the former is the squaring
function and the latter is the value 3, the result is 9. That may
be confusing but it is legal.

Now the *values* bound to symbols can be locally overridden with `let'
and the functions bound to symbols can be locally overridden with
`flet'. But now that `flet' is going away, how do you override the
function binding using `let'? Without `flet' (or something similar),
you just cannot: trying to use `let' to change the function binding
of a symbol is impossible. So how exactly is this second set of
transformations supposed to work?

It uses `let' to locally override the *value* binding of a symbol
with a *function*:

(defun f (x y z))
   (let ((g (lambda (x) (* x x))))
       ....)

would bind the squaring function to the symbol g - but it is 
the value binding of g, not its function binding. When the time
comes to call that function, you cannot just say

(defun f (x y z)
   (let ((g (lambda (x) (* x x))))
       (+ (g x) (g y) (g z))))

There is no function binding for the symbol g so you cannot use it in
the function position. That's where funcall (or apply in a different
context) can help. The functtion  can be written like this

(defun f (x y z)
   (let ((g (lambda (x) (* x x))))
       (+ (funcall g x)) (funcall g y) (funcall g z)))

That's the price one has to pay in order to eliminate flet and replace
it with let.

As you can see, this is a rather intrusive transformation. And it's even
worse: an flet in some function would define a function that could be
passed down an arbitrarily long call chain and then called at the lowest
level. If you use let, you can still pass the symbol down the chain but
when it is time to call the function, you need to use the funcall trick.
And even more obscurely, you don't have to pass it down explicitly: dynamic
binding will take care of finding the definition of the symbol an arbitrary
distance up the call chain. But if you have let-bound a function to a symbol,
you have to funcall it explicitly. And if the symbol *has* a function value
already, then calling it directly will also work but it will call a completely
different function!

That's an easy thing to overlook and that's exactly what happened here
I believe:

In org-babel-execute-src-block, the code looks like this:

        ...
	(unwind-protect
	    (let ((call-process-region
		   (lambda (&rest args)
		     (apply 'org-babel-tramp-handle-call-process-region args))))
	      (let ((lang-check (lambda (f)
				  (let ((f (intern (concat "org-babel-execute:" f))))
				    (when (fboundp f) f)))))
		(setq cmd
		      (or (funcall lang-check lang)
			  (funcall lang-check (symbol-name
					       (cdr (assoc lang org-src-lang-modes))))
			  (error "No org-babel-execute function for %s!" lang))))

                ...			  

		(setq result
		      ((lambda (result)
			 (if (and (eq (cdr (assoc :result-type params)) 'value)
				  (or (member "vector" result-params)
				      (member "table" result-params))
				  (not (listp result)))
			     (list (list result)) result))
		       (funcall cmd body params)))
        ...

Here we get the src block language (sh in Loris's example) and construct
a symbol based on it (org-babel-execute:sh in this case). Because of the
let, lang-check cannot be called as a function: it needs to be
funcall'ed. It binds cmd to the symbol org-babel-execute-sh and later
funcalls it. But note that call-process-region is also let-bound to a
function. So we have the call chain

org-babel-execute-src-block --> [through (funcall cmd body params)]
  org-babel-execute:sh --> 
    org-babel-sh-evaluate -->
      org-babel-eval -->
        org-babel-shell-command-on-region -->
	  call-process-region

And here we have a symbol that has a function binding as well as a value
binding (also a function, but a *different* function). The symbol is used
in the function position of the function application so the function
binding (i.e the usual call-process-region function) is used. It runs
hostname locally - Loris loses.

The correct thing to do (in this particular case at least) is to funcall
the symbol at the lowest level. That would call the let-bound function
(org-babel-tramp-handle-call-process-region) which would call hostname
remotely (making Loris happy).

Is it always correct to do that? I don't know but I suspect not: it
would probably be safer to have an org symbol
(org-call-process-region-function maybe) that's value-set to the
standard call-process-region function, but can then be let-bound and
dynamically passed all over the place.

And is this the only problem? Probably not: every flet->let
transformation would have to be scrutinized.

Re: Erroneous "No such file or directory" with babel and remote dir

[Achim Gratz]

Nick Dokos writes:
> The first set of code transformations (implemented as commit
> 63b5f8f2e85b3059a2d30041db6939347a7a2d7d) dealt with the situation by
> doing a mass substitution: flet --> org-flet and labels --> org-labels
> (and in at least one case, flet --> org-labels to deal with a
> recursive definition - I presume that was a preexisting bug that was
> fixed by this substitution) and adding compatibility aliases in
> org-compat.el to use the cl-flet/cl-labels macros from cl.el in emacs
> versions >= 24.1.50.

As has slowly transpired in the meantime, there is no substitution for
the deprecated flet.  The new cl-flet does lexical instead of dynamical
binding of the function slot and the other alternatives have other
subtle differences that don't really seem to be documented in a single
place.

> So the moral of the story is that the code transformations have *not*
> left functionality unchanged. Something went awry but to be honest, I
> don't know what. I didn't spend much time on it because of what I
> found out next.
[...]
> And is this the only problem? Probably not: every flet->let
> transformation would have to be scrutinized.

Thanks for this extensive explanation.  I don't know if that might
convince Stefan Monnier to un-deprecate letf, you'd not be the only one
to be rattling his cage on this issue.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Loris Bennett <loris.bennett@fu-berlin.de> wrote:
>
>> Achim Gratz <Stromeko@nexgo.de> writes:
>> 
>> > Loris Bennett writes:
>> >> OK, I must have goofed.  What I did after starting the bisection was
>> >>
>> >> - run 'make autoload'
>> >> - open test file in emacs with minimal .emacs
>> >> - test
>> >> - end emacs
>> >> - mark bisection good or bad
>> >>
>> >> I then repeated this for the next commit.
>> >
>> > This would be the correct thing to do, yes.
>> >
>> >> Is something missing here, or did I just incorrectly mark a commit?
>> >
>> > I really don't know, I'm just guessing… :-)
>> >
>> > It may be more informative if you traced the invocation of the remote
>> > execution in both a failing and a working version in the debugger and
>> > look for any differences as the arguments are cobbled together.  Once
>> > you find where an how these diverge it would be much easier to find the
>> > code that is responsible for it (if it wasn't glaringly obvious from the
>> > trace already).
>> >
>> >
>> > Regards,
>> > Achim.
>> 
>> So just as a reminder:
>> 
>> Back in the days of release_7.01h, the following used to work:
>> 
>> ,------------------------------------------
>> | #+begin_src sh :dir /loris@othercomputer:
>> | hostname
>> | #+end_src
>> `------------------------------------------
>> 
>> Currently it produces the error:
>> 
>> ,---------------------------------------------------
>> | apply: Wrong type argument: number-or-marker-p, ""
>> `---------------------------------------------------
>> 
>> I have had another go at bisecting this problem and came up with this:
>> 
>> ,-------------------------------------------------------------------------------------------------------------
>> | 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
>> | commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
>> | Author: Dan Davison <davison@stats.ox.ac.uk>
>> | Date:   Mon Aug 30 09:34:05 2010 -0700
>> | 
>> |     babel: Fix temporary file processing in the remote execution case.
>> |     
>> |         * ob.el (org-babel-temp-file): Don't use babel temporary
>> |         directory in remote case; use make-temp-file with remote file
>> |         name so that temp file is guaranteed not to exist previously
>> |         on remote machine.
>> |         (org-babel-tramp-localname): New function to return local name
>> |         portion of possibly remote file specification
>> |     
>> |         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
>> |         changes in `org-babel-temp-file'; pass local file name to
>> |         remote R process.
>> |         (org-babel-R-evaluate-session) Respond to
>> |         changes in `org-babel-temp-file'; pass local file name to
>> |         remote R process.
>> | 
>> | :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
>> `-------------------------------------------------------------------------------------------------------------
>> 
>> Does that help anyone any further?
>> 
>
> I don't think so: afaict, both 7.01h (which did not contain this patch)
> and 7.02 (which did) do the right thing (at least with your simple
> script above). Something else broke it.
>
> [I've been wrong about these things before so before Achim asks, I
> did
>
>   git tag --contains 9c878a8290c071fbe5e97bc33c300ef2f07d6153
>
> and used its output as a basis of the above statement. I think (hope?)
> it's correct.]
>
>> Loris
>> 
>> PS: I am amazed there aren't any more peasants like myself with torches
>> and pitchforks beating on the gates of Castle Org about this - remote
>> execution is/was such great feature!
>> 
>
> Agreed, but Dan Davison (the principal contributor of the remoting code)
> is gone from Castle Org (as you put it), Eric S. is busy, and nobody
> else has stepped up to the plate. I've made fitful attempts to debug it,
> but have had no success: I've never looked at the code in detail and I
> don't have much time either.
>
> So how about becoming a resident of Castle Org rather than beating on
> the gates? There's glory galore if you fix it - and you get to scratch
> your itch too :-)

OK, I don't have that much time either but, as my Lisp skills are pretty
rudimentary, I've often thought it might be a good idea to take on some
sort of lisp project, so I'll try to give it a go.

Cheers,

Loris

-- 
no sig is good sig

Re: Erroneous "No such file or directory" with babel and remote dir

[Loris Bennett]

Achim Gratz <Stromeko@nexgo.de> writes:

> Loris Bennett writes:
>> OK, I must have goofed.  What I did after starting the bisection was
>>
>> - run 'make autoload'
>> - open test file in emacs with minimal .emacs
>> - test
>> - end emacs
>> - mark bisection good or bad
>>
>> I then repeated this for the next commit.
>
> This would be the correct thing to do, yes.
>
>> Is something missing here, or did I just incorrectly mark a commit?
>
> I really don't know, I'm just guessing… :-)
>
> It may be more informative if you traced the invocation of the remote
> execution in both a failing and a working version in the debugger and
> look for any differences as the arguments are cobbled together.  Once
> you find where an how these diverge it would be much easier to find the
> code that is responsible for it (if it wasn't glaringly obvious from the
> trace already).
>
>
> Regards,
> Achim.

[Sorry if this appears twice - Gnus got the better of me]

So just as a reminder:

Back in the days of release_7.01h, the following used to work:

,------------------------------------------
| #+begin_src sh :dir /loris@othercomputer:
| hostname
| #+end_src
`------------------------------------------

Currently it produces the error:

,---------------------------------------------------
| apply: Wrong type argument: number-or-marker-p, ""
`---------------------------------------------------

I have had another go at bisecting this problem and came up with this:

,-------------------------------------------------------------------------------------------------------------
| 9c878a8290c071fbe5e97bc33c300ef2f07d6153 is the first bad commit
| commit 9c878a8290c071fbe5e97bc33c300ef2f07d6153
| Author: Dan Davison <davison@stats.ox.ac.uk>
| Date:   Mon Aug 30 09:34:05 2010 -0700
| 
|     babel: Fix temporary file processing in the remote execution case.
|     
|         * ob.el (org-babel-temp-file): Don't use babel temporary
|         directory in remote case; use make-temp-file with remote file
|         name so that temp file is guaranteed not to exist previously
|         on remote machine.
|         (org-babel-tramp-localname): New function to return local name
|         portion of possibly remote file specification
|     
|         * ob-R.el (org-babel-R-evaluate-external-process): Respond to
|         changes in `org-babel-temp-file'; pass local file name to
|         remote R process.
|         (org-babel-R-evaluate-session) Respond to
|         changes in `org-babel-temp-file'; pass local file name to
|         remote R process.
| 
| :040000 040000 b31e072cf6b2951e95b7956d907303e7a04a8cfd 5f794ada52cceb0614fe7962a399f7e549759003 M      lisp
`-------------------------------------------------------------------------------------------------------------

Does that help anyone any further?

Cheers,

Loris

PS: I am amazed there aren't any more peasants like myself with torches
and pitchforks beating on the gates of Castle Org about this - remote
execution is/was such great feature!
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett@fu-berlin.de