a minor patch to awk invocation

a minor patch to awk invocation

[Greg Minshall]

hi.  i just wandered down a rathole others could avoid.  the following
program fails in (what was to me) a mysterious way:
----
#+BEGIN_SRC awk :var a=2
BEGIN{print $a;}
#+END_SRC
----

it turns out values for variables to awk need to be strings (rather than
a number, as above).  below is a patch to give what might be a less
mysterious error message.

also: for an awk invocation with ":var a=b", all occurrences of "$a" in
the body of the awk code are changed to "b".  i'm curious why this is
done rather than invoking awk with "-v a=b"?  or, moving the
initialization into a "BEGIN{}" block?  maybe because of the variety of
awk variants loose in the world?

cheers, Greg
----
diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el
index ed98afd..162ddfb 100644
--- a/lisp/ob-awk.el
+++ b/lisp/ob-awk.el
@@ -47,6 +47,8 @@
 (defun org-babel-expand-body:awk (body params)
   "Expand BODY according to PARAMS, return the expanded body."
   (dolist (pair (mapcar #'cdr (org-babel-get-header params :var)))
+    (if (not (stringp (cdr pair)))
+	(error "awk variable values must be strings"))
     (setf body (replace-regexp-in-string
                 (regexp-quote (format "$%s" (car pair))) (cdr pair) body)))
   body)

Re: a minor patch to awk invocation

[Eric Schulte]

Greg Minshall <minshall@acm.org> writes:

> hi.  i just wandered down a rathole others could avoid.  the following
> program fails in (what was to me) a mysterious way:
> ----
> #+BEGIN_SRC awk :var a=2
> BEGIN{print $a;}
> #+END_SRC
> ----
>
> it turns out values for variables to awk need to be strings (rather than
> a number, as above).  below is a patch to give what might be a less
> mysterious error message.
>
> also: for an awk invocation with ":var a=b", all occurrences of "$a" in
> the body of the awk code are changed to "b".  i'm curious why this is
> done rather than invoking awk with "-v a=b"?  or, moving the
> initialization into a "BEGIN{}" block?  maybe because of the variety of
> awk variants loose in the world?
>

Hi Greg,

Thanks for pointing out this shortcoming and for suggesting the
command-line variable assignment option.  I've just pushed up a patch
which changes the behavior of awk code blocks to assign variables on the
command line, so the following now work.

#+begin_src awk :var a=2
BEGIN{ print a; }
#+end_src

#+RESULTS:
: 2

Thanks,

>
> cheers, Greg
> ----
> diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el
> index ed98afd..162ddfb 100644
> --- a/lisp/ob-awk.el
> +++ b/lisp/ob-awk.el
> @@ -47,6 +47,8 @@
>  (defun org-babel-expand-body:awk (body params)
>    "Expand BODY according to PARAMS, return the expanded body."
>    (dolist (pair (mapcar #'cdr (org-babel-get-header params :var)))
> +    (if (not (stringp (cdr pair)))
> +	(error "awk variable values must be strings"))
>      (setf body (replace-regexp-in-string
>                  (regexp-quote (format "$%s" (car pair))) (cdr pair) body)))
>    body)
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: a minor patch to awk invocation

[Greg Minshall]

very nice -- thanks!
----
> I've just pushed up a patch which changes the behavior of awk code
> blocks to assign variables on the command line, so the following now
> work.

> #+begin_src awk :var a=2
> BEGIN{ print a; }
> #+end_src

> #+RESULTS:
> : 2

Re: a minor patch to awk invocation

[Greg Minshall]

though, in the spirit of "no good deed goes unpunished":
----
#+name: foo
| a | b | c |

#+begin_src awk :var a=foo
BEGIN{ print a; }
#+end_src
----
gives an error (and, ':var a="this is a test"' doesn't behave as one
might expect).  i haven't looked at ob-*.el enough to know the patterns
used to wrap up various forms of variables into different languages.

Re: a minor patch to awk invocation

[Eric Schulte]

Greg Minshall <minshall@acm.org> writes:

> though, in the spirit of "no good deed goes unpunished":
> ----
> #+name: foo
> | a | b | c |
> #+begin_src awk :var a=foo
> BEGIN{ print a; }
> #+end_src
> ----
> gives an error (and, ':var a="this is a test"' doesn't behave as one
> might expect).  i haven't looked at ob-*.el enough to know the patterns
> used to wrap up various forms of variables into different languages.

Ah, thanks for pointing this out.  I just pushed up a simple fix along
with a unit test for tabular input to awk blocks.

Best,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: a minor patch to awk invocation

[Greg Minshall]

thanks -- nice!

Re: a minor patch to awk invocation

[Achim Gratz]

Eric Schulte writes:
> Thanks for pointing out this shortcoming and for suggesting the
> command-line variable assignment option.  I've just pushed up a patch
> which changes the behavior of awk code blocks to assign variables on the
> command line, so the following now work.

For whatever reason I now get errors from an awk invocation using
standard input:

--8<---------------cut here---------------start------------->8---
tset: standard error: Inappropriate ioctl for device
      
/bin/stty: standard input: Inappropriate ioctl for device
/usr/bin/stty: standard input: Inappropriate ioctl for device
--8<---------------cut here---------------end--------------->8---

Additionally, C-c C-v C-v doesn't work correctly on many blocks and
gives

--8<---------------cut here---------------start------------->8---
org-babel-expand-src-block: Wrong type argument: number-or-marker-p, nil
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: a minor patch to awk invocation

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> Thanks for pointing out this shortcoming and for suggesting the
>> command-line variable assignment option.  I've just pushed up a patch
>> which changes the behavior of awk code blocks to assign variables on the
>> command line, so the following now work.
>
> For whatever reason I now get errors from an awk invocation using
> standard input:
>
> --8<---------------cut here---------------start------------->8---
> tset: standard error: Inappropriate ioctl for device
>       
> /bin/stty: standard input: Inappropriate ioctl for device
> /usr/bin/stty: standard input: Inappropriate ioctl for device
> --8<---------------cut here---------------end--------------->8---
>
> Additionally, C-c C-v C-v doesn't work correctly on many blocks and
> gives
>
> --8<---------------cut here---------------start------------->8---
> org-babel-expand-src-block: Wrong type argument: number-or-marker-p, nil
> --8<---------------cut here---------------end--------------->8---
>

I can't reproduce these problems.  Could you provide examples, and maybe
a stack traces?

Thanks,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: a minor patch to awk invocation

[Achim Gratz]

Eric Schulte writes:
> I can't reproduce these problems.  Could you provide examples, and maybe
> a stack traces?

The tests have all been done with "make vanilla" and
testing/examples/ob-awk-test.org.  The test failure is:

--8<---------------cut here---------------start------------->8---
(("tset:" "standard" "error:" "Inappropriate" "ioctl" "for" "device")
 ("" "" "" "" "" "" "")
 ("/bin/stty:" "standard" "input:" "Inappropriate" "ioctl" "for" "device")
 ("/usr/bin/stty:" "standard" "input:" "Inappropriate" "ioctl" "for" "device")
 (43 "" "" "" "" "" ""))
Test ob-awk/input-src-block-1 backtrace:
  =(43 (("tset:" "standard" "error:" "Inappropriate" "ioctl" "for" "de
  apply(= (43 (("tset:" "standard" "error:" "Inappropriate" "ioctl" "f
  (setq value-74 (apply fn-72 args-73))
  (unwind-protect (setq value-74 (apply fn-72 args-73)) (setq form-des
  (if (unwind-protect (setq value-74 (apply fn-72 args-73)) (setq form
  (let (form-description-76) (if (unwind-protect (setq value-74 (apply
  (let ((value-74 (quote ert-form-evaluation-aborted-75))) (let (form-
  (let ((fn-72 (function =)) (args-73 (list 43 (org-babel-execute-src-
  (save-restriction (org-babel-next-src-block 2) (let ((fn-72 (functio
  (progn (org-id-goto "9e998b2a-3581-43fe-b26d-07d3c507b86a") (setq to
  (unwind-protect (progn (org-id-goto "9e998b2a-3581-43fe-b26d-07d3c50
  (let ((save-match-data-internal (match-data))) (unwind-protect (prog
  (progn (let ((save-match-data-internal (match-data))) (unwind-protec
  (unwind-protect (progn (let ((save-match-data-internal (match-data))
  (let ((wconfig (current-window-configuration))) (unwind-protect (pro
  (unwind-protect (let ((wconfig (current-window-configuration))) (unw
  (let* ((id-location (org-id-find "9e998b2a-3581-43fe-b26d-07d3c507b8
  (lambda nil (let* ((id-location (org-id-find "9e998b2a-3581-43fe-b26
  byte-code("\306\307!▒q\210\310\216\311 \312\216\313\314\315\316\3
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  byte-code("\306\307!\211▒\211r\310\311!q\210\312 d\313\223)L\210)\3
  ert-run-test([cl-struct-ert-test ob-awk/input-src-block-1 "Test a co
  ert-run-or-rerun-test([cl-struct-ert--stats "ob-awk" [[cl-struct-ert
  ert-run-tests("ob-awk" #[(event-type &rest event-args) "\30\307\"\
  ert-run-tests-batch("ob-awk")
  ert-run-tests-batch-and-exit("ob-awk")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("ob-awk")
  eval((org-test-run-batch-tests org-test-select-re))
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test ob-awk/input-src-block-1 condition:
    (wrong-type-argument number-or-marker-p
                         (("tset:" "standard" "error:" "Inappropriate" "ioctl" "for" "device")
                          ("" "" "" "" "" "" "")
                          ("/bin/stty:" "standard" "input:" "Inappropriate" "ioctl" "for" "device")
                          ("/usr/bin/stty:" "standard" "input:" "Inappropriate" "ioctl" "for" "device")
                          (43 "" "" "" "" "" "")))
   FAILED  2/4  ob-awk/input-src-block-1
--8<---------------cut here---------------end--------------->8---

I've no idea how to get at the command line.  It is clear that awk gets
invoked and even delivers the right result, but somehow it is doing this
differently than before in the case where it uses STDIN.  I can't
reproduce with my own awk program on the command line.

The problem with "C-c C-v C-v" happens when invoking it on the
#+BEGIN_SRC line only (which interestingly enough works without error
once you've done it inside the source block first).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: a minor patch to awk invocation

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> I can't reproduce these problems.  Could you provide examples, and maybe
>> a stack traces?
>
> The tests have all been done with "make vanilla" and
> testing/examples/ob-awk-test.org.  The test failure is:
>
[...]
>
> I've no idea how to get at the command line.  It is clear that awk gets
> invoked and even delivers the right result, but somehow it is doing this
> differently than before in the case where it uses STDIN.  I can't
> reproduce with my own awk program on the command line.
>

Thanks for the additional information.

The only thing I can think of is that the ordering of the arguments on
the command line is somehow responsible for this error.  Here's the
command line executed by the ob-awk/input-src-block-1 awk test,

    $ awk -f /tmp/babel-66681az/awk-6668uW2

where /tmp/babel-66681az/awk-6668uW2 has the following content.

,----
| {
|     print 42+$1
| }
`----

The command is invoked with `call-process-shell-command'.

Are these failures only present *after* these recent changes to ob-awk?
I can't think of how these changes could be related to this STDIN error.

>
> The problem with "C-c C-v C-v" happens when invoking it on the
> #+BEGIN_SRC line only (which interestingly enough works without error
> once you've done it inside the source block first).
>

I would be surprised if this "C-c C-v C-v" is caused by my recent
changes to awk's variable handling.  Can you find the offend commit here
(or at least check if the problem is new)?

Thanks,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: a minor patch to awk invocation

[Achim Gratz]

Eric Schulte writes:
> Are these failures only present *after* these recent changes to ob-awk?
> I can't think of how these changes could be related to this STDIN error.

Yes, they've just started with the second commit (the first one broke
the tests in a different way as you may know).

> I would be surprised if this "C-c C-v C-v" is caused by my recent
> changes to awk's variable handling.  Can you find the offend commit here
> (or at least check if the problem is new)?

I'd be surprised, too.  No time for a deeper dig at the moment, but I'll
keep it in mind.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: a minor patch to awk invocation

[Achim Gratz]

The problem was that

a) I'm using tcsh.

b) Some part of the system-wide csh.cshrc related to Emacs' shell-mode
   was also run for non-interactive shell invocation while only
   appropriate for interactive use (that's where the errors from tset
   and stty came from).

c) Your second patch 6fb033c791 corrected a mistake in test-ob-awk.el
   that had shadowed the test invocation which produces the error and
   thus this particular test had never run before.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves