bug in expansion of variables in babel Perl

bug in expansion of variables in babel Perl

[D M German]

Hi Everybody,

I found a bug in the Babel perl code. When a table is used as input, the
values of the table  are not escaped. In fact, they are surrounded by
double quotes " instead of single ones '. This means that special
characters are interpreted: $<string>, and @<variable are considered
variables. See below.


For example:
----------------------------------------------------------------------
#+RESULTS: patito
| alias                                                | uniname                |
|------------------------------------------------------+------------------------|
| Jon  <tixy@xyz.org>                                  | jon


#+name: output(data=patito)
#+begin_src perl :results output
print "Begin\n";
print $$data[0][0], "\n";
print "End\n";
#+end_src                                                                                                                                                                

#+RESULTS: output
: Begin
: Jon  <tixy.org>
: End   

----------------------------------------------------------------------

I see two ways to solve this. The first is simply to replace the output
format of the variable from "%S" to "'%s'" (use quotes '). The other one
is to optionally escape the fields of the table (which is more
complicated, and would require replacing each). The third one is a
combination of both: replace them only if desired, via some header
configuration variable.

diff --git a/lisp/ob-perl.el b/lisp/ob-perl.el
index ccd3826..2f795aa 100644
--- a/lisp/ob-perl.el
+++ b/lisp/ob-perl.el
@@ -75,7 +75,7 @@ The elisp value, VAR, is converted to a string of perl source code
 specifying a var of the same value."
   (if (listp var)
       (concat "[" (mapconcat #'org-babel-perl-var-to-perl var ", ") "]")
-    (format "%S" var)))
+    (format "'%s'" var)))



Debugging perl is very cumbersome in org-mode. It would be nice to have
a feature to export the source to a file. This is because the variable
expansion needs to be done before the code can be used (hence simply cut
and paste does not work, nor shell-command-on-region)

I used the org-babel-perl-command variable to replace perl with a script
that simply wrote to a file. 

It would be nice to be able to write the script created by org a file,
so this can be debugged (it would have the variable definitions). Maybe
this is already a feature and I don't know about it.

As we are into it, I found this declaration to be very useful. 

----------------------------------------------------------------------
(setq org-babel-perl-wrapper-method
  "
use strict;

sub org_columns
{
    my ($table) = @_;
    my $y = $$table[0];
    return scalar(@$y);
}

sub org_rows
{
    my ($table) = @_;
    return scalar(@$table);
}

sub main {
%s
}

my @r = main;
open(o, \">%s\");
print o join(\"\\n\", @r), \"\\n\"")
----------------------------------------------------------------------

It does two things: it uses strict, so undeclared variables create
errors, and it also creates two functions: org_columns and org_rows
that, when used on the variable declared as input, return its number of
columns and rows:

my $rows = org_rows($data);
my $columns = org_columns($data);

the only problem with using strict is that variables would have to be
defined with "my" too: so that would require this patch:

diff --git a/lisp/ob-perl.el b/lisp/ob-perl.el
index ccd3826..82f8086 100644
--- a/lisp/ob-perl.el
+++ b/lisp/ob-perl.el
@@ -62,7 +62,7 @@ This function is called by `org-babel-execute-src-block'."
   "Return list of perl statements assigning the block's variables."
   (mapcar
    (lambda (pair)
-     (format "$%s=%s;"
+     (format "my $%s=%s;"
 	     (car pair)
 	     (org-babel-perl-var-to-perl (cdr pair))))
    (mapcar #'cdr (org-babel-get-header params :var))))


Finally, if interested, i can write a couple of examples for Perl that
could help people who want to use it. 

thanks again,

--
Daniel M. German                  "Great algorithms are
   Francis Sullivan ->             the poetry of computation"
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: bug in expansion of variables in babel Perl

[dmg]

Mm, I also noticed that when :results output is used, there is no way
to insert perl code before or after the executed code.
org-babel-perl-wrapper-method only works for all the methods but
output. It would be nice to have a variable that
does this for any output type.

--dmg

On Sun, Feb 24, 2013 at 1:16 AM, D M German <dmg@uvic.ca> wrote:
>
> Hi Everybody,
>
> I found a bug in the Babel perl code. When a table is used as input, the
> values of the table  are not escaped. In fact, they are surrounded by
> double quotes " instead of single ones '. This means that special
> characters are interpreted: $<string>, and @<variable are considered
> variables. See below.
>
>
--dmg

---
Daniel M. German
http://turingmachine.org

Re: bug in expansion of variables in babel Perl

[D M German]

 dmg> Mm, I also noticed that when :results output is used, there is no way
 dmg> to insert perl code before or after the executed code.
 dmg> org-babel-perl-wrapper-method only works for all the methods but
 dmg> output. It would be nice to have a variable that
 dmg> does this for any output type.

I implemented a proof-of-concept. The idea is to have a variable called
org-babel-perl-preface that is inserted before the code. I also like to
be able to use my in all variables, so I can use strict, if I choose
to. See the patch at the bottom. 


here is a test example:


======================================================================
Input table

#+RESULTS: patito
| id                 | title       | year | index | notes | attr |
|--------------------+-------------+------+-------+-------+------|
| Taxi Driver (1944) | Taxi Driver | 1944 |       |       |      |
| Taxi Driver (1954) | Taxi Driver | 1954 |       |       |      |
| Taxi Driver (1973) | Taxi Driver | 1973 |       |       |      |
| Taxi Driver (1976) | Taxi Driver | 1976 |       |       |      |
| Taxi Driver (1978) | Taxi Driver | 1978 |       |       |      |
| Taxi Driver (1981) | Taxi Driver | 1981 |       |       |      |
| Taxi Driver (1990) | Taxi Driver | 1990 |       |       |      |
| Taxi Driver (2004) | Taxi Driver | 2004 |       |       |      |


Simple row output: the last statement is returned as a list, each in a line.

#+name: output2(data=patito)
#+begin_src perl :results raw
org_rows($data), org_columns($data);
#+end_src

#+RESULTS: output2
8
6

More complex example. By defining org_rows and org_columns in the
preface, it makes it easier to manipulate them. org-babel implements
tables as a reference to an array of references to arrays.

#+name: rip(data=patito)
#+begin_src perl :results output
my $rows = org_rows($data);
my $columns = org_columns($data);

for (my $j=0;$j<$rows; $j++) {
    for (my $i=0;$i<$columns; $i++) {
	print "$i:$j ";
	print $$data[$j][$i];
        print "   ;"
    }
    print "|\n";
}
print "Row $rows\n";
print "Columns $columns\n";
#+end_src

#+RESULTS: rip
#+begin_example
0:0 Taxi Driver (1944)   ;1:0 Taxi Driver   ;2:0 1944   ;3:0    ;4:0    ;5:0    ;|
0:1 Taxi Driver (1954)   ;1:1 Taxi Driver   ;2:1 1954   ;3:1    ;4:1    ;5:1    ;|
0:2 Taxi Driver (1973)   ;1:2 Taxi Driver   ;2:2 1973   ;3:2    ;4:2    ;5:2    ;|
0:3 Taxi Driver (1976)   ;1:3 Taxi Driver   ;2:3 1976   ;3:3    ;4:3    ;5:3    ;|
0:4 Taxi Driver (1978)   ;1:4 Taxi Driver   ;2:4 1978   ;3:4    ;4:4    ;5:4    ;|
0:5 Taxi Driver (1981)   ;1:5 Taxi Driver   ;2:5 1981   ;3:5    ;4:5    ;5:5    ;|
0:6 Taxi Driver (1990)   ;1:6 Taxi Driver   ;2:6 1990   ;3:6    ;4:6    ;5:6    ;|
0:7 Taxi Driver (2004)   ;1:7 Taxi Driver   ;2:7 2004   ;3:7    ;4:7    ;5:7    ;|
Row 8
Columns 6
#+end_example

======================================================================

diff --git a/lisp/ob-perl.el b/lisp/ob-perl.el
index ccd3826..65e6b88 100644
--- a/lisp/ob-perl.el
+++ b/lisp/ob-perl.el
@@ -62,7 +62,7 @@ This function is called by `org-babel-execute-src-block'."
   "Return list of perl statements assigning the block's variables."
   (mapcar
    (lambda (pair)
-     (format "$%s=%s;"
+     (format "my $%s=%s;"
 	     (car pair)
 	     (org-babel-perl-var-to-perl (cdr pair))))
    (mapcar #'cdr (org-babel-get-header params :var))))
@@ -85,13 +85,34 @@ specifying a var of the same value."
 
 (defvar org-babel-perl-wrapper-method
   "
+%s
 sub main {
 %s
 }
-@r = main;
+my @r = main;
 open(o, \">%s\");
 print o join(\"\\n\", @r), \"\\n\"")
 
+(defvar org-babel-perl-preface
+ "
+use strict;
+
+sub org_columns
+{
+    my ($table) = @_;
+    my $y = $$table[0];
+    return scalar(@$y);
+}
+
+sub org_rows
+{
+    my ($table) = @_;
+    return scalar(@$table);
+}
+
+")
+
+
 (defvar org-babel-perl-pp-wrapper-method
   nil)
 
@@ -102,11 +123,11 @@ of the statements in BODY, if RESULT-TYPE equals 'value then
 return the value of the last statement in BODY, as elisp."
   (when session (error "Sessions are not supported for Perl"))
   (case result-type
-    (output (org-babel-eval org-babel-perl-command body))
+    (output (org-babel-eval org-babel-perl-command (format "%s\n%s" org-babel-perl-preface body)))
     (value (let ((tmp-file (org-babel-temp-file "perl-")))
 	     (org-babel-eval
 	      org-babel-perl-command
-	      (format org-babel-perl-wrapper-method body
+	      (format org-babel-perl-wrapper-method org-babel-perl-preface body
 		      (org-babel-process-file-name tmp-file 'noquote)))
 	     (org-babel-eval-read-file tmp-file)))))
 



--
Daniel M. German                  "In questions of science the authority of
                                   a thousand is not worth the humble
   Galileo ->                      reasoning of a single individual."
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: bug in expansion of variables in babel Perl

[Achim Gratz]

D M German writes:
> I see two ways to solve this. The first is simply to replace the output
> format of the variable from "%S" to "'%s'" (use quotes ').

I think that's the right thing to do.  There shouldn't be anything in
the table that needs to be interpolated by Perl while the variable is
defined.

> +    (format "'%s'" var)))

A slightly more perlish way would be to use
       (format "q(%s)" var)))

> Debugging perl is very cumbersome in org-mode. It would be nice to have
> a feature to export the source to a file. This is because the variable
> expansion needs to be done before the code can be used (hence simply cut
> and paste does not work, nor shell-command-on-region)

The other languages have the same problem, maybe there should be a
general option to mirror the commands into a source block in the org
file or to make the buffer with the program to be eval'ed stick around.

Eric, WDYT?  Also, I'm not really sure why we need all the complexity of
shell-command-on-region when it looks like we should be able to
call-process-region ourselves.  Modifying Babel to run (non-session and
perhaps optionally) from files instead of buffers seems to be a more
wide-reaching operation.

> As we are into it, I found this declaration to be very useful. 
[…]

I think this is better done by altering org-babel-perl-command to
include "-Mstrict".  If you put the helper functions into a module in
@INC or tell Perl where to find them, then you can add "-Mmyhelper" as
well.  Here's a wrapper to match:

(defvar org-babel-perl-wrapper-method
  "{
  my @r = eval( q(%s) );
  open my $BO, qq(>%s));
  print $BO join($\\, @r), $\\ ;
}")

For your problem with :results output blocks, I think it would be
possible to wrap them (a bit differently) also, but the "helper module"
above would also solve this problem, so let's see what Eric says since I
don't know if another language has already set a precedent of wrapping
these commands too.

>
> Finally, if interested, i can write a couple of examples for Perl that
> could help people who want to use it. 

Also a few tests would be highly welcome.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: bug in expansion of variables in babel Perl

[Achim Gratz]

D M German writes:
[…]

Please leave the formats alone, if you change the number of parameters
there folks that use their own definitions won't know what hit them.
What you want is to prepend something to the body that Babel gives you,
so let-bind that result and use it.  You could even advise the function
and have it submit to your will without changing Org.

--8<---------------cut here---------------start------------->8---
(defun org-babel-perl-evaluate (session ibody &optional result-type)
  "Pass BODY to the Perl process in SESSION.
If RESULT-TYPE equals 'output then return a list of the outputs
of the statements in BODY, if RESULT-TYPE equals 'value then
return the value of the last statement in BODY, as elisp."
  (when session (error "Sessions are not supported for Perl"))
  (let ((body (concat org-babel-perl-preface ibody)))
    (case result-type
      (output (org-babel-eval org-babel-perl-command body))
      (value (let ((tmp-file (org-babel-temp-file "perl-")))
	       (org-babel-eval
		org-babel-perl-command
		(format org-babel-perl-wrapper-method body
			(org-babel-process-file-name tmp-file 'noquote)))
	       (org-babel-eval-read-file tmp-file))))))
--8<---------------cut here---------------end--------------->8---


BTW, now that I think some more about it: debugging Perl is much easier
than you seem to let on:

(setq org-babel-perl-command "perl -Mstrict -ne print").

This will echo the program sent to Perl in full glory into the output
block.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: bug in expansion of variables in babel Perl

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 3452 bytes --]

Achim Gratz <Stromeko@nexgo.de> writes:

> D M German writes:
>> I see two ways to solve this. The first is simply to replace the output
>> format of the variable from "%S" to "'%s'" (use quotes ').
>
> I think that's the right thing to do.  There shouldn't be anything in
> the table that needs to be interpolated by Perl while the variable is
> defined.
>
>> +    (format "'%s'" var)))
>
> A slightly more perlish way would be to use
>        (format "q(%s)" var)))
>

I just added the variable `org-babel-perl-var-wrap', into ob-perl.el

    ;; emacs-lisp
    (defvar org-babel-perl-var-wrap "q(%s)"
      "Wrapper for variables inserted into Perl code.")

This way we will get what sounds like improved wrapping by default, but
users who really do want to insert interpolated values can customize
this variable.

>
>> Debugging perl is very cumbersome in org-mode. It would be nice to have
>> a feature to export the source to a file. This is because the variable
>> expansion needs to be done before the code can be used (hence simply cut
>> and paste does not work, nor shell-command-on-region)
>
> The other languages have the same problem, maybe there should be a
> general option to mirror the commands into a source block in the org
> file or to make the buffer with the program to be eval'ed stick around.
>
> Eric, WDYT?

Are you familiar with `org-babel-expand-src-block' bound to C-c C-v v?
If I understand the desire correctly, it should be what you're after.
Perhaps an option to raise the expanded source code buffer along with
the error buffer when an error is raised would be a useful addition.

> Also, I'm not really sure why we need all the complexity of
> shell-command-on-region when it looks like we should be able to
> call-process-region ourselves.  Modifying Babel to run (non-session
> and perhaps optionally) from files instead of buffers seems to be a
> more wide-reaching operation.
>

This complexity is related to the need to occasionally run in remote
directories or on remote machines.  If there are ways to reduce this
complexity without losing functionality I'm game.

>
>> As we are into it, I found this declaration to be very useful. 
> […]
>
> I think this is better done by altering org-babel-perl-command to
> include "-Mstrict".  If you put the helper functions into a module in
> @INC or tell Perl where to find them, then you can add "-Mmyhelper" as
> well.  Here's a wrapper to match:
>
> (defvar org-babel-perl-wrapper-method
>   "{
>   my @r = eval( q(%s) );
>   open my $BO, qq(>%s));
>   print $BO join($\\, @r), $\\ ;
> }")
>
> For your problem with :results output blocks, I think it would be
> possible to wrap them (a bit differently) also, but the "helper module"
> above would also solve this problem, so let's see what Eric says since I
> don't know if another language has already set a precedent of wrapping
> these commands too.
>

I'm not currently aware of any language-wide support for printing the
expanded code block along with the results.  I don't think there has
been any desire for this previously.  It shouldn't be hard to write an
emacs lisp block to give the desired result...

Here's an example of such a function.  Please forgive my complete lack
of Perl knowledge (at one time I knew it well). Note that I had to make
a small change to the return value of `org-babel-expand-src-block'.


[-- Attachment #2: it.org --]
[-- Type: text/x-org, Size: 603 bytes --]

#+name: table
| 1 | 2 | 3 |
| 4 | 5 | 6 |

#+name: perl-block
#+begin_src perl :var table=table
  print $table
#+end_src

#+RESULTS: perl-block
: 1

#+begin_src emacs-lisp :var block="perl-block" :results raw
  (save-excursion
    (message "block is %S" block)
    (org-babel-goto-named-src-block block)
    (format "#+begin_example\n%S\n↓\n%s\n#+end_example\n"
            (org-babel-expand-src-block)
            (org-babel-execute-src-block)))
#+end_src

#+RESULTS:
#+begin_example
"$table=[[q(1), q(2), q(3)], [q(4), q(5), q(6)]];
print $table"
↓
1

#+end_example


[-- Attachment #3: Type: text/plain, Size: 254 bytes --]


>
>>
>> Finally, if interested, i can write a couple of examples for Perl that
>> could help people who want to use it. 
>
> Also a few tests would be highly welcome.
>

+1

Cheers,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [PATCH] bug in expansion of variables in babel Perl

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

D M German writes:
> I found a bug in the Babel perl code. When a table is used as input, the
> values of the table  are not escaped.

Here are two patches that fix this and implement (partly) some of your
suggestions.  I don't think Org should pollute the global Perl namespace
by default, so I've left the definition of org-babel-perl-preface to the
user for now.  The second patch has the debugging aid you've been
requesting, if you bind the symbol org-babel--debug-input to anything
the temporary input files won't be deleted after the code has run.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-perl-modify-variable-definition-to-be-compatible-.patch --]
[-- Type: text/x-patch, Size: 3814 bytes --]

From 7a668942b58dc94994b11e6ec0751ec36b07b196 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 24 Feb 2013 13:28:50 +0100
Subject: [PATCH 1/2] ob-perl: modify variable definition to be compatible with
 strict and use non-interpolating quotes

* lisp/ob-perl.el (org-babel-variable-assignments:perl): Add "my" to
  variable declaration so that it becomes compatible with "use
  strict;".
* lisp/ob-perl.el (org-babel-perl-var-to-perl): Use Perl
  non-interpolating quoting on the string that defines the variable to
  suppress spurious interpretation of it as Perl syntax.
* lisp/ob-perl.el (org-babel-perl-wrapper-method): Use a block and
  declare all variables as "my", also use Perl quoting and the output
  record separator instead of a literal LF character.  Do away with
  the subroutine definition and use eval instead.
* lisp/ob-perl.el (org-babel-perl-preface): Content of this variable
  is prepended to body before invocation of perl.
* lisp/ob-perl.el (org-babel-perl-evaluate): Rename input parameter
  body to ibody and let-bind body to concatentation of
  org-babel-perl-preface and ibody.

Following a suggestion by Daniel M. German in
http://thread.gmane.org/gmane.emacs.orgmode/66855.
---
 lisp/ob-perl.el | 37 ++++++++++++++++++++-----------------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/lisp/ob-perl.el b/lisp/ob-perl.el
index ccd3826..53f166e 100644
--- a/lisp/ob-perl.el
+++ b/lisp/ob-perl.el
@@ -62,7 +62,7 @@ (defun org-babel-variable-assignments:perl (params)
   "Return list of perl statements assigning the block's variables."
   (mapcar
    (lambda (pair)
-     (format "$%s=%s;"
+     (format "my $%s=%s;"
 	     (car pair)
 	     (org-babel-perl-var-to-perl (cdr pair))))
    (mapcar #'cdr (org-babel-get-header params :var))))
@@ -75,7 +75,7 @@ (defun org-babel-perl-var-to-perl (var)
 specifying a var of the same value."
   (if (listp var)
       (concat "[" (mapconcat #'org-babel-perl-var-to-perl var ", ") "]")
-    (format "%S" var)))
+    (format "q(%s)" var)))
 
 (defvar org-babel-perl-buffers '(:default . nil))
 
@@ -84,31 +84,34 @@ (defun org-babel-perl-initiate-session (&optional session params)
   nil)
 
 (defvar org-babel-perl-wrapper-method
-  "
-sub main {
+  "{
+  my @r = eval( q(
 %s
-}
-@r = main;
-open(o, \">%s\");
-print o join(\"\\n\", @r), \"\\n\"")
+              ));
+  open my $BO, qq(>%s) or die qq( Perl: Could not open output file.$\\ );
+  print $BO join($\\, @r), $\\ ;
+}")
+
+(defvar org-babel-perl-preface nil)
 
 (defvar org-babel-perl-pp-wrapper-method
   nil)
 
-(defun org-babel-perl-evaluate (session body &optional result-type)
+(defun org-babel-perl-evaluate (session ibody &optional result-type)
   "Pass BODY to the Perl process in SESSION.
 If RESULT-TYPE equals 'output then return a list of the outputs
 of the statements in BODY, if RESULT-TYPE equals 'value then
 return the value of the last statement in BODY, as elisp."
   (when session (error "Sessions are not supported for Perl"))
-  (case result-type
-    (output (org-babel-eval org-babel-perl-command body))
-    (value (let ((tmp-file (org-babel-temp-file "perl-")))
-	     (org-babel-eval
-	      org-babel-perl-command
-	      (format org-babel-perl-wrapper-method body
-		      (org-babel-process-file-name tmp-file 'noquote)))
-	     (org-babel-eval-read-file tmp-file)))))
+  (let ((body (concat org-babel-perl-preface ibody)))
+    (case result-type
+      (output (org-babel-eval org-babel-perl-command body))
+      (value (let ((tmp-file (org-babel-temp-file "perl-")))
+	       (org-babel-eval
+		org-babel-perl-command
+		(format org-babel-perl-wrapper-method body
+			(org-babel-process-file-name tmp-file 'noquote)))
+	       (org-babel-eval-read-file tmp-file))))))
 
 (provide 'ob-perl)
 
-- 
1.8.1.4


[-- Attachment #3: 0002-ob-eval-make-org-babel-shell-command-on-region-inter.patch --]
[-- Type: text/x-patch, Size: 10487 bytes --]

From 6827b07c0e8a03eea11d86ea714c8f10fb05b43d Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 24 Feb 2013 17:15:36 +0100
Subject: [PATCH 2/2] ob-eval: make org-babel--shell-command-on-region internal
 and simplify
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ob-eval.el (org-babel-eval): Use simplified version of
  `org-babel--shell-command-on-region´, we are the only caller of this
  function.
* lisp/ob-eval.el (org-babel--shell-command-on-region): Replace
  `org-babel-shell-command-on-region´ with a much more simplified
  internal version, remove superfluous DOCSTRING and interactive
  clause, strip out all conditionals which were never used.  Prevent
  deletion of temporary input file to aid debugging when the symbol
  `org-babel--debug-input´ is bound and has non-nil value.
---
 lisp/ob-eval.el | 195 +++++++++-----------------------------------------------
 1 file changed, 30 insertions(+), 165 deletions(-)

diff --git a/lisp/ob-eval.el b/lisp/ob-eval.el
index 22d2bcf..681362f 100644
--- a/lisp/ob-eval.el
+++ b/lisp/ob-eval.el
@@ -50,8 +50,8 @@ (defun org-babel-eval (cmd body)
     (with-temp-buffer
       (insert body)
       (setq exit-code
-	    (org-babel-shell-command-on-region
-	     (point-min) (point-max) cmd t 'replace err-buff))
+	    (org-babel--shell-command-on-region
+	     (point-min) (point-max) cmd err-buff))
       (if (or (not (numberp exit-code)) (> exit-code 0))
 	  (progn
 	    (with-current-buffer err-buff
@@ -64,79 +64,15 @@ (defun org-babel-eval-read-file (file)
   (with-temp-buffer (insert-file-contents file)
 		    (buffer-string)))
 
-(defun org-babel-shell-command-on-region (start end command
-						&optional output-buffer replace
-						error-buffer display-error-buffer)
+(defun org-babel--shell-command-on-region (start end command error-buffer)
   "Execute COMMAND in an inferior shell with region as input.
 
-Fixes bugs in the emacs 23.1.1 version of `shell-command-on-region'
-
-Normally display output (if any) in temp buffer `*Shell Command Output*';
-Prefix arg means replace the region with it.  Return the exit code of
-COMMAND.
-
-To specify a coding system for converting non-ASCII characters in
-the input and output to the shell command, use
-\\[universal-coding-system-argument] before this command.  By
-default, the input (from the current buffer) is encoded in the
-same coding system that will be used to save the file,
-`buffer-file-coding-system'.  If the output is going to replace
-the region, then it is decoded from that same coding system.
-
-The noninteractive arguments are START, END, COMMAND,
-OUTPUT-BUFFER, REPLACE, ERROR-BUFFER, and DISPLAY-ERROR-BUFFER.
-Noninteractive callers can specify coding systems by binding
-`coding-system-for-read' and `coding-system-for-write'.
-
-If the command generates output, the output may be displayed
-in the echo area or in a buffer.
-If the output is short enough to display in the echo area
-\(determined by the variable `max-mini-window-height' if
-`resize-mini-windows' is non-nil), it is shown there.  Otherwise
-it is displayed in the buffer `*Shell Command Output*'.  The output
-is available in that buffer in both cases.
-
-If there is output and an error, a message about the error
-appears at the end of the output.
-
-If there is no output, or if output is inserted in the current buffer,
-then `*Shell Command Output*' is deleted.
-
-If the optional fourth argument OUTPUT-BUFFER is non-nil,
-that says to put the output in some other buffer.
-If OUTPUT-BUFFER is a buffer or buffer name, put the output there.
-If OUTPUT-BUFFER is not a buffer and not nil,
-insert output in the current buffer.
-In either case, the output is inserted after point (leaving mark after it).
-
-If REPLACE, the optional fifth argument, is non-nil, that means insert
-the output in place of text from START to END, putting point and mark
-around it.
-
-If optional sixth argument ERROR-BUFFER is non-nil, it is a buffer
-or buffer name to which to direct the command's standard error output.
-If it is nil, error output is mingled with regular output.
-If DISPLAY-ERROR-BUFFER is non-nil, display the error buffer if there
-were any errors.  (This is always t, interactively.)
-In an interactive call, the variable `shell-command-default-error-buffer'
-specifies the value of ERROR-BUFFER."
-  (interactive (let (string)
-		 (unless (mark)
-		   (error "The mark is not set now, so there is no region"))
-		 ;; Do this before calling region-beginning
-		 ;; and region-end, in case subprocess output
-		 ;; relocates them while we are in the minibuffer.
-		 (setq string (read-shell-command "Shell command on region: "))
-		 ;; call-interactively recognizes region-beginning and
-		 ;; region-end specially, leaving them in the history.
-		 (list (region-beginning) (region-end)
-		       string
-		       current-prefix-arg
-		       current-prefix-arg
-		       shell-command-default-error-buffer
-		       t)))
-  (let ((input-file (org-babel-temp-file "input-"))
-	(error-file (if error-buffer (org-babel-temp-file "scor-") nil))
+Stripped down version of shell-command-on-region for internal use
+in Babel only.  This lets us work around errors in the original
+function in various versions of Emacs.
+"
+  (let ((input-file (org-babel-temp-file "ob-input-"))
+	(error-file (if error-buffer (org-babel-temp-file "ob-error-") nil))
 	;; Unfortunately, `executable-find' does not support file name
 	;; handlers.  Therefore, we could use it in the local case
 	;; only.
@@ -154,96 +90,26 @@ (defun org-babel-shell-command-on-region (start end command
     ;; workaround for now.
     (unless (file-remote-p default-directory)
       (delete-file error-file))
-    (if (or replace
-	    (and output-buffer
-		 (not (or (bufferp output-buffer) (stringp output-buffer)))))
-	;; Replace specified region with output from command.
-	(let ((swap (and replace (< start end))))
-	  ;; Don't muck with mark unless REPLACE says we should.
-	  (goto-char start)
-	  (and replace (push-mark (point) 'nomsg))
-	  (write-region start end input-file)
-	  (delete-region start end)
-	  (setq exit-status
-		(process-file shell-file-name input-file
-			      (if error-file
-				  (list output-buffer error-file)
-				t)
-			      nil shell-command-switch command))
-	  ;; It is rude to delete a buffer which the command is not using.
-	  ;; (let ((shell-buffer (get-buffer "*Shell Command Output*")))
-	  ;;   (and shell-buffer (not (eq shell-buffer (current-buffer)))
-	  ;; 	 (kill-buffer shell-buffer)))
-	  ;; Don't muck with mark unless REPLACE says we should.
-	  (and replace swap (exchange-point-and-mark)))
-      ;; No prefix argument: put the output in a temp buffer,
-      ;; replacing its entire contents.
-      (let ((buffer (get-buffer-create
-		     (or output-buffer "*Shell Command Output*"))))
-	(unwind-protect
-	    (if (eq buffer (current-buffer))
-		;; If the input is the same buffer as the output,
-		;; delete everything but the specified region,
-		;; then replace that region with the output.
-		(progn (setq buffer-read-only nil)
-		       (delete-region (max start end) (point-max))
-		       (delete-region (point-min) (min start end))
-		       (write-region (point-min) (point-max) input-file)
-		       (delete-region (point-min) (point-max))
-		       (setq exit-status
-			     (process-file shell-file-name input-file
-					   (if error-file
-					       (list t error-file)
-					     t)
-					   nil shell-command-switch command)))
-	      ;; Clear the output buffer, then run the command with
-	      ;; output there.
-	      (let ((directory default-directory))
-		(with-current-buffer buffer
-		  (setq buffer-read-only nil)
-		  (if (not output-buffer)
-		      (setq default-directory directory))
-		  (erase-buffer)))
-	      (setq exit-status
-		    (process-file shell-file-name nil
-				  (if error-file
-				      (list buffer error-file)
-				    buffer)
-				  nil shell-command-switch command)))
-	  ;; Report the output.
-	  (with-current-buffer buffer
-	    (setq mode-line-process
-		  (cond ((null exit-status)
-			 " - Error")
-			((stringp exit-status)
-			 (format " - Signal [%s]" exit-status))
-			((not (equal 0 exit-status))
-			 (format " - Exit [%d]" exit-status)))))
-	  (if (with-current-buffer buffer (> (point-max) (point-min)))
-	      ;; There's some output, display it
-	      (display-message-or-buffer buffer)
-	    ;; No output; error?
-	    (let ((output
-		   (if (and error-file
-			    (< 0 (nth 7 (file-attributes error-file))))
-		       "some error output"
-		     "no output")))
-	      (cond ((null exit-status)
-		     (message "(Shell command failed with error)"))
-		    ((equal 0 exit-status)
-		     (message "(Shell command succeeded with %s)"
-			      output))
-		    ((stringp exit-status)
-		     (message "(Shell command killed by signal %s)"
-			      exit-status))
-		    (t
-		     (message "(Shell command failed with code %d and %s)"
-			      exit-status output))))
-	    ;; Don't kill: there might be useful info in the undo-log.
-	    ;; (kill-buffer buffer)
-	    ))))
-
-    (when (and input-file (file-exists-p input-file))
+    ;; we always call this with 'replace, remove conditional
+    ;; Replace specified region with output from command.
+    (let ((swap (< start end)))
+      (goto-char start)
+      (push-mark (point) 'nomsg)
+      (write-region start end input-file)
+      (delete-region start end)
+      (setq exit-status
+	    (process-file shell-file-name input-file
+			  (if error-file
+			      (list t error-file)
+			    t)
+			  nil shell-command-switch command))
+      (when swap (exchange-point-and-mark)))
+
+    (when (and input-file (file-exists-p input-file)
+	       ;; bind org-babel--debug-input around the call to keep
+	       ;; the temporary input files available for inspection
+	       (not (when (boundp 'org-babel--debug-input)
+		      org-babel--debug-input)))
       (delete-file input-file))
 
     (when (and error-file (file-exists-p error-file))
@@ -258,8 +124,7 @@ (defun org-babel-shell-command-on-region (start end command
 	      (format-insert-file error-file nil)
 	      ;; Put point after the inserted errors.
 	      (goto-char (- (point-max) pos-from-end)))
-	    (and display-error-buffer
-		 (display-buffer (current-buffer)))))
+	    (current-buffer)))
       (delete-file error-file))
     exit-status))
 
-- 
1.8.1.4


[-- Attachment #4: Type: text/plain, Size: 558 bytes --]


Please reply to the list whether this works for you so when Eric sees
this he can decide if he wants to apply them.

Eric, the input file seems to be missing the final newline, probably
because the original region didn't include it.  This can be a problem
with some programs, is this intended to be left off or should we add
one (generally or just for some languages)?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: bug in expansion of variables in babel Perl

[Achim Gratz]

Eric Schulte writes:
> I just added the variable `org-babel-perl-var-wrap', into ob-perl.el
>
>     ;; emacs-lisp
>     (defvar org-babel-perl-var-wrap "q(%s)"
>       "Wrapper for variables inserted into Perl code.")
>
> This way we will get what sounds like improved wrapping by default, but
> users who really do want to insert interpolated values can customize
> this variable.

It would be impossible to use any user variables for interpolation since
we don't have Perl sessions, so only pre-defined variables from Perl
would ever deliver a value (perhaps).  So all interpolation would do
most of the the time is to replace something that happens to look like a
perl variable by nothing.  Having the variable doesn't hurt, but I'm not
sure we should advertize its existence widely since with a more devious
definition you can do arbitrary code execution.

> This complexity is related to the need to occasionally run in remote
> directories or on remote machines.  If there are ways to reduce this
> complexity without losing functionality I'm game.

Already done by throwing away those parts of the code we never used
anyway.  It looks much more manageable now.

> I'm not currently aware of any language-wide support for printing the
> expanded code block along with the results.  I don't think there has
> been any desire for this previously.  It shouldn't be hard to write an
> emacs lisp block to give the desired result...

Also done.  The file the OP wanted to look at gets written out anyway,
we just need to prevent its deletion.  This is a manual affair for now,
if this is really a big issue we can add an option for this.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: bug in expansion of variables in babel Perl

[Achim Gratz]

Eric Schulte writes:
> Are you familiar with `org-babel-expand-src-block' bound to C-c C-v v?

I wasn't, obviously, and neither was the OP.

> If I understand the desire correctly, it should be what you're after.
> Perhaps an option to raise the expanded source code buffer along with
> the error buffer when an error is raised would be a useful addition.

Yes, so there's no three ways for Perl to get the same result for
comparison.  :-P

Still, the debugging option might be useful when one needs to to see all
sources created during a publish or something like that; when you don't
know if or where the error occured.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: bug in expansion of variables in babel Perl

[D M German]

 Achim> D M German writes:
 Achim> […]

 Achim> Please leave the formats alone, if you change the number of parameters
 Achim> there folks that use their own definitions won't know what hit them.
 Achim> What you want is to prepend something to the body that Babel gives you,
 Achim> so let-bind that result and use it.  You could even advise the function
 Achim> and have it submit to your will without changing Org.

Hi Achim, thanks for the recommendation. As I said before, see my
previous patch as a proof-of-concept and not as something that I think
should be applied. I am fully aware of the potential consequences.

 Achim> BTW, now that I think some more about it: debugging Perl is much easier
 Achim> than you seem to let on:

 Achim> (setq org-babel-perl-command "perl -Mstrict -ne print").

Using strict is complicated by use of variables from org tables and the
way that the code in R is executed. It would require the variable @r and
each of the created variables from tables to be defined as my. I know I
can change the behaviour of org-babel-variable-assignment, but perhaps
simply adding "my" to any variable created in org would be a good thing
to do. I don't think it would harm, anyways.

 Achim> This will echo the program sent to Perl in full glory into the output
 Achim> block.

thanks, this i a great suggestion that I didn't know about.

--dmg


 Achim> Regards,
 Achim> Achim.
 Achim> -- 
 Achim> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

 Achim> Waldorf MIDI Implementation & additional documentation:
 Achim> http://Synth.Stromeko.net/Downloads.html#WaldorfDocs




--
Daniel M. German                  "Beware of bugs in the above code;
                                   I have only proved it
   Donald Knuth  ->                correct, not tried it."
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: [PATCH] bug in expansion of variables in babel Perl

[D M German]

 Achim Gratz twisted the bytes to say:

Hi Achim,

thanks for taking the time to do this.

I applied the patch, one of the hunks didn't apply due to Eric's
changes, but that is not an issue, since they do the same:

----------------------------------------------------------------------
diff a/lisp/ob-perl.el b/lisp/ob-perl.el	(rejected hunks)
@@ -75,7 +75,7 @@ (defun org-babel-perl-var-to-perl (var)
 specifying a var of the same value."
   (if (listp var)
       (concat "[" (mapconcat #'org-babel-perl-var-to-perl var ", ") "]")
-    (format "%S" var)))
+    (format "q(%s)" var)))
 
 (defvar org-babel-perl-buffers '(:default . nil))
----------------------------------------------------------------------

Everything works as intended, except for the return value of the perl
code. Values in the list are concatenated, as one:

#+begin_src perl :results table
(1, 2, 3)
#+end_src

#+RESULTS:
| 123 |

#+begin_src perl :results table
(1, 2, 3)
#+end_src

#+RESULTS:
| 123 |


I think the issue is that, at least in my computer the variable $\
returns empty (the record separator).

#+begin_src perl :results output
print "value of \$\\ [$\]\n";
#+end_src

#+RESULTS:
#+begin_example
value of $\ [] 
#+end_example

--daniel

--
Daniel M. German                  "Work. Finish. Publish. "
                                   Michael Faraday
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: bug in expansion of variables in babel Perl

[D M German]

 Achim> Eric Schulte writes:
 >> Are you familiar with `org-babel-expand-src-block' bound to C-c C-v v?

 Achim> I wasn't, obviously, and neither was the OP.

 >> If I understand the desire correctly, it should be what you're after.
 >> Perhaps an option to raise the expanded source code buffer along with
 >> the error buffer when an error is raised would be a useful addition.

 Achim> Yes, so there's no three ways for Perl to get the same result for
 Achim> comparison.  :-P


Just keep in mind, org-babel-expand-src-block is not exactly what emacs
passes to perl. It does not contain the wrapping code.


--dmg


--
Daniel M. German
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: [PATCH] bug in expansion of variables in babel Perl

[Achim Gratz]

D M German <dmg <at> uvic.ca> writes:
> I think the issue is that, at least in my computer the variable $\
> returns empty (the record separator).

Thinko on my side,  what I wanted was the input record separator "$/" (to avoid
specifying a literal newline for those systems where this is actually
multi-character).


Regards,
Achim.

Re: [PATCH] bug in expansion of variables in babel Perl

[D M German]

 Achim> D M German <dmg <at> uvic.ca> writes:
 >> I think the issue is that, at least in my computer the variable $\
 >> returns empty (the record separator).

 Achim> Thinko on my side, what I wanted was the input record separator "$/"
 Achim> (to avoid
 Achim> specifying a literal newline for those systems where this is actually
 Achim> multi-character).

Hi Achim,

Once I changed it:

(defvar org-babel-perl-wrapper-method
  "{
  my @r = eval( q(
%s
              ));
  open my $BO, qq(>%s) or die qq( Perl: Could not open output file.$\\ );
  print $BO join($/, @r), $/ ;
}")

the result now has \n in between fields (literally):

#+name: t_output_table
#+begin_src perl :results table
print "Test\n";
(1, 2)
#+end_src

#+RESULTS: t_output_table
| 1\n2\n |

what is the expected field separator for Org-babel?

 Achim> Regards,
 Achim> Achim.




--
Daniel M. German                  "Prose is intrinsically linear; 
                                   a good book carries the reader forward
   The Economist ->                on the crest of the words"
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .

 

Re: [PATCH] bug in expansion of variables in babel Perl

[Achim Gratz]

D M German <dmg <at> uvic.ca> writes:
>   print $BO join($/, @r), $/ ;

Sorry, this should really be:

print $BO join(qq($/), @r), qq($/);

Anyway, I think I'll have to rework the wrapper to be an anonymous subroutine so
that the (reasonably expactable) "return @foo;" at the end of the program (or in
the middle if so desired) continues to work.

Also, I just checked that noweb syntax does work for code execution, so the
prefix you wanted to have is easy enough to add via Library of Babel:

#+name: prefix
#+begin_src perl
# Santa's little helpers
#+end_src

#+begin_src perl :noweb yes
<<prefix>>
&help();
#+end_src

In light of this, I think we should not implement the "preface" change as its
effects can be had without obscuring things too much.


Regards,
Achim.

Re: [PATCH] bug in expansion of variables in babel Perl

[Achim Gratz]

Achim Gratz writes:
> Here are two patches that fix this and implement (partly) some of your
> suggestions.  I don't think Org should pollute the global Perl namespace
> by default, so I've left the definition of org-babel-perl-preface to the
> user for now.  The second patch has the debugging aid you've been
> requesting, if you bind the symbol org-babel--debug-input to anything
> the temporary input files won't be deleted after the code has run.

I've pushed these changes with some extended babel support for Perl: the
results are now interpreted by babel, like for most other languages.
Also, the wrapper has been extended to be a bit smarter: if the return
value is an arrayref (or arrayref of arrayrefs), the wrapper
automatically formats the output so that it will be interpreted as a
table (a simple arrayref will be interpreted as a column vector).  In
particular that means you can pass a table through a Perl source block
with exactly no code to write (as the variable that is input will be
returned directly).  In all other cases the return value is simply
printed, so it should be a scalar.  I've kept the wrapper as an
(anonymous) subroutine, so if anybody had latched onto "return $rv;" it
will continue to work.  Also a few tests were added.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada