* org babel support for tcl and awk
@ 2011-05-24 9:31 orgmode
2011-05-24 12:51 ` Eric Schulte
0 siblings, 1 reply; 13+ messages in thread
From: orgmode @ 2011-05-24 9:31 UTC (permalink / raw)
To: emacs-orgmode; +Cc: orgmode
Hi,
I am looking for support for Tcl (and AWK) for org-babel. Both have a
supplied emacs mode and Tcl also has an inferior interpreter mode. I
was trying to do it myself, however I am quite lost in the
instructions. Is there someone with the knowledge and willingness to
provide a support file for org babel for Tcl (and maybe awk)?
thanks.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 9:31 org babel support for tcl and awk orgmode
@ 2011-05-24 12:51 ` Eric Schulte
2011-05-24 17:53 ` Eric S Fraga
2011-05-24 18:57 ` orgmode
0 siblings, 2 replies; 13+ messages in thread
From: Eric Schulte @ 2011-05-24 12:51 UTC (permalink / raw)
To: orgmode@h-rd.org; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 671 bytes --]
Hi,
Are you aware of the ob-template.el file [1], which can be used as a
jumping off point to simplify the addition of new languages? After
globally replacing the term "template" with you language name, the only
function that necessarily needs to be re-written is the main
`org-babel-execute:template' function.
I would recommend starting with only non-session based evaluation, and
then slowly adding functionality. If you run into any specific problems
I am happy to help trouble shoot.
As an example, I've worked up an very simple ob-awk.el file from
ob-template.el, it is attached along with an example org-mode file which
demonstrates its usage.
Best -- Eric
[-- Attachment #2: ob-awk.el --]
[-- Type: application/emacs-lisp, Size: 6815 bytes --]
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: example.org --]
[-- Type: text/x-org, Size: 419 bytes --]
* example use of ob-awk
Header Arguments
- cmd-line :: command line flags to pass to =awk=
- in-file :: the text file on which to run the resulting =awk= script
Currently only string results are returned directly from STDOUT.
Currently only external (i.e., non-session) evaluation is supported.
#+begin_src awk :in-file columns-of-numbers.txt
{print $1}
#+end_src
#+results:
: 0
: 1
: 2
: 3
: 4
: 5
: 6
: 7
: 8
[-- Attachment #4: Type: text/plain, Size: 554 bytes --]
"orgmode@h-rd.org" <orgmode@h-rd.org> writes:
> Hi,
>
> I am looking for support for Tcl (and AWK) for org-babel. Both have a
> supplied emacs mode and Tcl also has an inferior interpreter mode. I
> was trying to do it myself, however I am quite lost in the
> instructions. Is there someone with the knowledge and willingness to
> provide a support file for org babel for Tcl (and maybe awk)?
>
> thanks.
>
>
>
Footnotes:
[1] http://repo.or.cz/w/Worg.git/blob/HEAD:/org-contrib/babel/ob-template.el
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 12:51 ` Eric Schulte
@ 2011-05-24 17:53 ` Eric S Fraga
2011-05-24 19:03 ` Eric Schulte
2011-05-24 18:57 ` orgmode
1 sibling, 1 reply; 13+ messages in thread
From: Eric S Fraga @ 2011-05-24 17:53 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode
Eric Schulte <schulte.eric@gmail.com> writes:
[...]
> As an example, I've worked up an very simple ob-awk.el file from
> ob-template.el, it is attached along with an example org-mode file which
> demonstrates its usage.
Eric,
this is great to see as I use awk quite often. What is involved in
extending this to be able to run an awk script on input from within the
org file (output of another babel block, for instance, as my typical use
of awk is to re-arrange output from another program...)? Or, if you
wish, can you suggest one of the ob-XXX modules that best illustrates
how to do this and I can give it a try?
Thanks,
eric
--
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.298.g7c436)
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 12:51 ` Eric Schulte
2011-05-24 17:53 ` Eric S Fraga
@ 2011-05-24 18:57 ` orgmode
1 sibling, 0 replies; 13+ messages in thread
From: orgmode @ 2011-05-24 18:57 UTC (permalink / raw)
To: emacs-orgmode, Eric Schulte; +Cc: orgmode
Hi Eric,
yes I am aware of op-template and tried to use it. However it was not
clear to me how to proceed and I looked into ob-perl, ob-ruby,
ob-scheme and ob-python. But it seemed to me they use a different
structure than op-template and I was stuck. I also saw that the file
ob-template is documented, however I am not so good in org-emacs-speak
that I can decipher it.
Really thanks for the awk example, I will tru to study it.
thanks.
Quoting Eric Schulte <schulte.eric@gmail.com>:
> Hi,
>
> Are you aware of the ob-template.el file [1], which can be used as a
> jumping off point to simplify the addition of new languages? After
> globally replacing the term "template" with you language name, the only
> function that necessarily needs to be re-written is the main
> `org-babel-execute:template' function.
>
> I would recommend starting with only non-session based evaluation, and
> then slowly adding functionality. If you run into any specific problems
> I am happy to help trouble shoot.
>
> As an example, I've worked up an very simple ob-awk.el file from
> ob-template.el, it is attached along with an example org-mode file which
> demonstrates its usage.
>
> Best -- Eric
>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 17:53 ` Eric S Fraga
@ 2011-05-24 19:03 ` Eric Schulte
2011-05-24 19:55 ` Sebastien Vauban
2011-05-26 13:03 ` Eric Schulte
0 siblings, 2 replies; 13+ messages in thread
From: Eric Schulte @ 2011-05-24 19:03 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> Eric Schulte <schulte.eric@gmail.com> writes:
>
> [...]
>
>> As an example, I've worked up an very simple ob-awk.el file from
>> ob-template.el, it is attached along with an example org-mode file which
>> demonstrates its usage.
>
> Eric,
>
> this is great to see as I use awk quite often. What is involved in
> extending this to be able to run an awk script on input from within the
> org file (output of another babel block, for instance, as my typical use
> of awk is to re-arrange output from another program...)? Or, if you
> wish, can you suggest one of the ob-XXX modules that best illustrates
> how to do this and I can give it a try?
>
I've made a quick change so that any variable named "stdin" is treated
specially, in that, rather than using its value to replace strings of
$stdin in the text of the awk code, the value of the stdin variable is
saved into the file processed by awk. This allows awk to operate over
Org-mode references.
See the attached example file.
If babel code block supported a pipe or an actual stdin header argument,
that would be the ideal way to add this behavior, but currently nothing
of that nature exists.
Please let me know if this misses part of your suggestion, or more
generally what else may be advisable before we add this to the core.
Cheers -- Eric
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: example.org --]
[-- Type: text/x-org, Size: 653 bytes --]
* awk example
#+results: simple-table
| 1 | 2 | 3 |
| 4 | 5 | 6 |
| 7 | 8 | 9 |
#+begin_src awk :var stdin=simple-table
{print $1}
#+end_src
#+results:
: 1
: 4
: 7
#+results: simple-text
: Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
: eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enimad
: minim veniam, quis nostrud exercitation ullamco laboris nisi ut
: aliquip ex ea commodo consequat. Duis aute irure dolor in
: reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
: pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
: culpa qui officia deserunt mollit anim id est laborum.
[-- Attachment #3: ob-awk.el --]
[-- Type: application/emacs-lisp, Size: 4665 bytes --]
[-- Attachment #4: Type: text/plain, Size: 67 bytes --]
>
> Thanks,
> eric
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 19:03 ` Eric Schulte
@ 2011-05-24 19:55 ` Sebastien Vauban
2011-05-24 23:51 ` Eric Schulte
2011-05-26 13:03 ` Eric Schulte
1 sibling, 1 reply; 13+ messages in thread
From: Sebastien Vauban @ 2011-05-24 19:55 UTC (permalink / raw)
To: emacs-orgmode-mXXj517/zsQ
Hi Eric(s),
Eric Schulte wrote:
> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes:
>> this is great to see as I use awk quite often. What is involved in
>> extending this to be able to run an awk script on input from within the org
>> file (output of another babel block, for instance, as my typical use of awk
>> is to re-arrange output from another program...)? Or, if you wish, can you
>> suggest one of the ob-XXX modules that best illustrates how to do this and
>> I can give it a try?
>
> I've made a quick change so that any variable named "stdin" is treated
> specially, in that, rather than using its value to replace strings of $stdin
> in the text of the awk code, the value of the stdin variable is saved into
> the file processed by awk. This allows awk to operate over Org-mode
> references.
>
> If babel code block supported a pipe or an actual stdin header argument,
> that would be the ideal way to add this behavior, but currently nothing of
> that nature exists.
>
> Please let me know if this misses part of your suggestion, or more generally
> what else may be advisable before we add this to the core.
Could this be implemented for sh as well?
AFAI understand, this is exactly the missing piece for me to be able to:
- run consecutive partial blocks of code in my Org buffer (seeing what they
really do on input data),
- export the full list of block as a script.
This was described in
http://www.mail-archive.com/emacs-orgmode-mXXj517/zsQ@public.gmane.org/msg36976.html, and still
impossible -- for me! -- to do right now. But I'm not very sure I wrote my
thoughts in an enough understandable way -- maybe not clear enough to me?
Best regards,
Seb
--
Sébastien Vauban
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 19:55 ` Sebastien Vauban
@ 2011-05-24 23:51 ` Eric Schulte
2011-05-25 12:30 ` Sebastien Vauban
0 siblings, 1 reply; 13+ messages in thread
From: Eric Schulte @ 2011-05-24 23:51 UTC (permalink / raw)
To: Sebastien Vauban; +Cc: emacs-orgmode
"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:
> Hi Eric(s),
>
> Eric Schulte wrote:
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>> this is great to see as I use awk quite often. What is involved in
>>> extending this to be able to run an awk script on input from within the org
>>> file (output of another babel block, for instance, as my typical use of awk
>>> is to re-arrange output from another program...)? Or, if you wish, can you
>>> suggest one of the ob-XXX modules that best illustrates how to do this and
>>> I can give it a try?
>>
>> I've made a quick change so that any variable named "stdin" is treated
>> specially, in that, rather than using its value to replace strings of $stdin
>> in the text of the awk code, the value of the stdin variable is saved into
>> the file processed by awk. This allows awk to operate over Org-mode
>> references.
>>
>> If babel code block supported a pipe or an actual stdin header argument,
>> that would be the ideal way to add this behavior, but currently nothing of
>> that nature exists.
>>
>> Please let me know if this misses part of your suggestion, or more generally
>> what else may be advisable before we add this to the core.
>
> Could this be implemented for sh as well?
>
> AFAI understand, this is exactly the missing piece for me to be able to:
>
Hi Seb,
Unfortunately this simple hack for ob-awk does not address the need you
link to below -- which I am aware of and which is on my list of larger
longer-term Babel development items. I think that a future piping
implementation will be the ultimate solution to the issues you address.
Such an implementation -- allowing data to flow between concurrently
executing blocks utilizing posix pipes -- will require more
sophisticated processes interaction and possibly some form of
multi-threaded elisp execution.
Best -- Eric
>
> - run consecutive partial blocks of code in my Org buffer (seeing what
> they really do on input data),
>
> - export the full list of block as a script.
>
> This was described in
> http://www.mail-archive.com/emacs-orgmode@gnu.org/msg36976.html, and still
> impossible -- for me! -- to do right now. But I'm not very sure I wrote my
> thoughts in an enough understandable way -- maybe not clear enough to me?
>
> Best regards,
> Seb
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 23:51 ` Eric Schulte
@ 2011-05-25 12:30 ` Sebastien Vauban
2011-05-25 15:57 ` Eric Schulte
0 siblings, 1 reply; 13+ messages in thread
From: Sebastien Vauban @ 2011-05-25 12:30 UTC (permalink / raw)
To: emacs-orgmode-mXXj517/zsQ
Hi Eric,
Eric Schulte wrote:
> "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> Eric Schulte wrote:
>>> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes: I've made a quick change so that
>>> any variable named "stdin" is treated specially, in that, rather than
>>> using its value to replace strings of $stdin in the text of the awk code,
>>> the value of the stdin variable is saved into the file processed by awk.
>>> This allows awk to operate over Org-mode references.
>>>
>>> If babel code block supported a pipe or an actual stdin header argument,
>>> that would be the ideal way to add this behavior, but currently nothing of
>>> that nature exists.
>>>
>>> Please let me know if this misses part of your suggestion, or more
>>> generally what else may be advisable before we add this to the core.
>>
>> Could this be implemented for sh as well?
>>
>> AFAI understand, this is exactly the missing piece for me to be able to:
>
> Unfortunately this simple hack for ob-awk does not address the need you link
> to below -- which I am aware of and which is on my list of larger
> longer-term Babel development items. I think that a future piping
> implementation will be the ultimate solution to the issues you address.
Glad to hear you understand my wish. It's not always easy to express myself in
a very clean, with English not being my mother tongue, especially when trying
to tackle difficult subjects.
> Such an implementation -- allowing data to flow between concurrently
> executing blocks utilizing posix pipes -- will require more sophisticated
> processes interaction and possibly some form of multi-threaded elisp
> execution.
Just for the sake of clarity, I don't need concurrent or multi-threaded
execution of any kind.
My double-sided goal is:
1. to cut a shell script in small parts, and explain what every part does,
with a runnable example (=C-c C-v C-e=).
2. to tangle the executable script out of the Babel document, by concatenating
all its parts (=C-c C-v C-t=).
A quite "dumb" example follows. I've made it as _minimal_ and as _complete_ as
possible, to be able to _express my point_, for further reference.
* Abstract
This script "americanizes" a European CSV file.
* Sample data
The following is a sample CSV file:
#+results: sample-csv
#+begin_example
Date;Amount;Account
28-05-2010;-6.806,25;999-1974050-30
04-06-2009;420,00;999-1500974-23
24-02-2009;-54,93;999-1974050-30
#+end_example
* Script
What the script must do is:
** Load the data
Read the raw contents of the input file.
#+srcname: load-data
#+begin_src sh :var data=sample-csv :results output :exports both
echo "$data"
#+end_src
#+results: load-data
#+begin_example
Date;Amount;Account
28-05-2010;-6.806,25;999-1974050-30
04-06-2009;420,00;999-1500974-23
24-02-2009;-54,93;999-1974050-30
#+end_example
** Convert the date in American format
Convert the date in =MM/DD/YYYY= format.
#+srcname: convert-date
#+begin_src sh :var data=load-data :results output :exports both
echo "$data" |\
sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\
sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g'
#+end_src
#+results: convert-date
#+begin_example
Date;Amount;Account
28/05/202010;-6.806,25;999-1974050-30
04/06/202009;420,00;999-1500974-23
24/02/202009;-54,93;999-1974050-30
#+end_example
** Convert the separators
Apply the following operations in order to "americanize" the CSV file received
from the bank:
- remove the dot used as thousands separator (=.= -> ==)
- replace the comma used as decimal separator by a dot (=,= -> =.=)
- replace other commas by a dot (=,= -> =.=)
- replace the semi-comma used as field separator by a comma (=;= -> =,=)
#+srcname: convert-separators
#+begin_src sh :var data=convert-date :results output :exports both
echo "$data" |\
sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\
sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\
sed -r 's/,/./g' |\
sed -r 's/;/,/g'
#+end_src
#+results: convert-separators
#+begin_example
Date,Amount,Account
28/05/202010,-6806.25,999-1974050-30
04/06/202009,420.00,999-1500974-23
24/02/202009,-54.93,999-1974050-30
#+end_example
* Full code
The script is then:
#+begin_src sh :tangle americanize-csv.sh :noweb yes
#!/bin/bash
# americanize-csv.sh -- Convert CSV file to American format
# Usage: americanize-csv FILE.CSV
cat $1 |\
<<convert-date>> |\
<<convert-separators>>
exit 0
# americanize-csv.sh ends here
#+end_src
As you can see, the tangled script is not executable anymore, as I've been
forced to put =echo $data= commands, in every apart code block, as their first
command to run.
#+begin_src sh
#!/bin/bash
# americanize-csv.sh -- Convert CSV file to American format
# Usage: americanize-csv FILE.CSV
cat $1 |\
echo "$data" |\
sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\
sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g' |\
echo "$data" |\
sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\
sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\
sed -r 's/,/./g' |\
sed -r 's/;/,/g'
exit 0
# americanize-csv.sh ends here
#+end_src
Would I have the possibility to play with =stdin=, I could have "hidden" that
first line, and assume all the code I'm writing will be executed against
what's read on =stdin=. As well in the Org buffer, as in the stand-alone shell
script. Right?
#+begin_src sh
#!/bin/bash
# americanize-csv.sh -- Convert CSV file to American format
# Usage: americanize-csv FILE.CSV
cat $1 |\
sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\
sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g' |\
sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\
sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\
sed -r 's/,/./g' |\
sed -r 's/;/,/g'
exit 0
# americanize-csv.sh ends here
#+end_src
* Conclusions
As you can see, I did not really mean any concurrent execution. Simply being
able to execute parts of code in-situ, in the Org buffer, to document (and
test) what I'm writing.
And to be able to assemble all the parts in one single script file, by the
means of literate programming.
Best regards,
Seb
--
Sébastien Vauban
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-25 12:30 ` Sebastien Vauban
@ 2011-05-25 15:57 ` Eric Schulte
2011-05-26 11:18 ` Sebastien Vauban
0 siblings, 1 reply; 13+ messages in thread
From: Eric Schulte @ 2011-05-25 15:57 UTC (permalink / raw)
To: Sebastien Vauban; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 945 bytes --]
> * Conclusions
>
> As you can see, I did not really mean any concurrent execution. Simply being
> able to execute parts of code in-situ, in the Org buffer, to document (and
> test) what I'm writing.
>
> And to be able to assemble all the parts in one single script file, by the
> means of literate programming.
>
I see, you want to be able to construct a large pipe chain STDOUT>STDIN,
but you don't care if the parts of the chain (e.g., the code block)
execute in serial or concurrently (as they do in the shell).
The attached patch (can be applied with "git am") implements this
behavior as I understand it. The result is a new :stdin header argument
with which org-mode references can be passed to shell scripts as
standard input. Given the technique used in this patch, I'll probably
re-write part of ob-awk.el.
The following Org-mode snippet demonstrates it's use, please let me know
if this works for your use-case described above.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: example.org --]
[-- Type: text/x-org, Size: 316 bytes --]
** passing values through to STDIN of shell code blocks
#+results: square-table
| 1 | 2 | 3 |
| 4 | 5 | 6 |
| 7 | 8 | 9 |
#+source: first-col
#+begin_src sh :stdin square-table
awk '{print $1}'
#+end_src
#+begin_src sh :stdin first-col
sed 's/4/middle/g'
#+end_src
#+results:
| 1 |
| middle |
| 7 |
[-- Attachment #3: Type: text/plain, Size: 17 bytes --]
Cheers -- Eric
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0001-sh-allow-passing-references-through-STDIN-of-a-shell.patch --]
[-- Type: text/x-diff, Size: 4709 bytes --]
From 03aa327d7950176e50ebb779c3f153bb9192dea5 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Wed, 25 May 2011 09:50:36 -0600
Subject: [PATCH] sh: allow passing references through STDIN of a shell script
* lisp/ob-sh.el (ob-ref): Uses ob-ref to resolve the value of the
:stdin header argument.
(org-babel-execute:sh): Use the :stdin header argument.
(org-babel-sh-var-to-sh): Split the bulk of this function off into a
new sub-function.
(org-babel-sh-var-to-string): New function for converting elisp
values to strings that make sense for parsing with sh.
(org-babel-sh-evaluate): Adding "stdin" option to session and
external evaluation options.
---
lisp/ob-sh.el | 44 ++++++++++++++++++++++++++++++++------------
1 files changed, 32 insertions(+), 12 deletions(-)
diff --git a/lisp/ob-sh.el b/lisp/ob-sh.el
index 128924d..10c08d4 100644
--- a/lisp/ob-sh.el
+++ b/lisp/ob-sh.el
@@ -28,6 +28,7 @@
;;; Code:
(require 'ob)
+(require 'ob-ref)
(require 'ob-comint)
(require 'ob-eval)
(require 'shell)
@@ -57,10 +58,13 @@ This function is called by `org-babel-execute-src-block'."
(let* ((session (org-babel-sh-initiate-session
(cdr (assoc :session params))))
(result-params (cdr (assoc :result-params params)))
+ (stdin ((lambda (stdin) (when stdin (org-babel-sh-var-to-string
+ (org-babel-ref-resolve stdin))))
+ (cdr (assoc :stdin params))))
(full-body (org-babel-expand-body:generic
body params (org-babel-variable-assignments:sh params))))
(org-babel-reassemble-table
- (org-babel-sh-evaluate session full-body result-params)
+ (org-babel-sh-evaluate session full-body result-params stdin)
(org-babel-pick-name
(cdr (assoc :colname-names params)) (cdr (assoc :colnames params)))
(org-babel-pick-name
@@ -101,14 +105,17 @@ This function is called by `org-babel-execute-src-block'."
"Convert an elisp value to a shell variable.
Convert an elisp var into a string of shell commands specifying a
var of the same value."
+ (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep)))
+
+(defun org-babel-sh-var-to-string (var &optional sep)
+ "Convert an elisp value to a string."
(flet ((echo-var (v) (if (stringp v) v (format "%S" v))))
- ((lambda (var) (format org-babel-sh-var-quote-fmt var))
- (cond
- ((and (listp var) (listp (car var)))
- (orgtbl-to-generic var (list :sep (or sep "\t") :fmt #'echo-var)))
- ((listp var)
- (mapconcat #'echo-var var "\n"))
- (t (echo-var var))))))
+ (cond
+ ((and (listp var) (listp (car var)))
+ (orgtbl-to-generic var (list :sep (or sep "\t") :fmt #'echo-var)))
+ ((listp var)
+ (mapconcat #'echo-var var "\n"))
+ (t (echo-var var)))))
(defun org-babel-sh-table-or-results (results)
"Convert RESULTS to an appropriate elisp value.
@@ -128,7 +135,7 @@ Emacs-lisp table, otherwise return the results as a string."
(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
"String to indicate that evaluation has completed.")
-(defun org-babel-sh-evaluate (session body &optional result-params)
+(defun org-babel-sh-evaluate (session body &optional result-params stdin)
"Pass BODY to the Shell process in BUFFER.
If RESULT-TYPE equals 'output then return a list of the outputs
of the statements in BODY, if RESULT-TYPE equals 'value then
@@ -141,8 +148,19 @@ return the value of the last statement in BODY."
(let ((tmp-file (org-babel-temp-file "sh-")))
(with-temp-file tmp-file (insert results))
(org-babel-import-elisp-from-file tmp-file)))))
- (if (not session)
- (org-babel-eval org-babel-sh-command (org-babel-trim body))
+ (cond
+ (stdin ; external shell script w/STDIN
+ (let ((script-file (org-babel-temp-file "sh-script-"))
+ (stdin-file (org-babel-temp-file "sh-stdin-")))
+ (with-temp-file script-file (insert body))
+ (with-temp-file stdin-file (insert stdin))
+ (with-temp-buffer
+ (call-process-shell-command
+ (format "%s %s" org-babel-sh-command script-file)
+ stdin-file
+ (current-buffer))
+ (buffer-string))))
+ (session ; session evaluation
(mapconcat
#'org-babel-sh-strip-weird-long-prompt
(mapcar
@@ -156,7 +174,9 @@ return the value of the last statement in BODY."
(append
(split-string (org-babel-trim body) "\n")
(list org-babel-sh-eoe-indicator))))
- 2)) "\n"))))
+ 2)) "\n"))
+ ('otherwise ; external shell script
+ (org-babel-eval org-babel-sh-command (org-babel-trim body))))))
(defun org-babel-sh-strip-weird-long-prompt (string)
"Remove prompt cruft from a string of shell output."
--
1.7.4.1
[-- Attachment #5: Type: text/plain, Size: 74 bytes --]
>
> Best regards,
> Seb
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-25 15:57 ` Eric Schulte
@ 2011-05-26 11:18 ` Sebastien Vauban
2011-05-26 13:37 ` Eric Schulte
0 siblings, 1 reply; 13+ messages in thread
From: Sebastien Vauban @ 2011-05-26 11:18 UTC (permalink / raw)
To: emacs-orgmode-mXXj517/zsQ
Hi Eric,
Eric Schulte wrote:
>> As you can see, I did not really mean any concurrent execution. Simply
>> being able to execute parts of code in-situ, in the Org buffer, to document
>> (and test) what I'm writing.
>>
>> And to be able to assemble all the parts in one single script file, by the
>> means of literate programming.
>
> I see, you want to be able to construct a large pipe chain STDOUT>STDIN,
That's it!
> but you don't care if the parts of the chain (e.g., the code block) execute
> in serial or concurrently (as they do in the shell).
For me, there is no concept of serial or concurrent execution here, as I am
executing manually the calls, when writing the Org document.
Not sure to understand you...
Are you talking of what happens for the export, maybe?
Are you talking of the shell constructs which will be used in the way the
"full script" is assembled?
> The attached patch (can be applied with "git am") implements this
> behavior as I understand it. The result is a new :stdin header argument
> with which org-mode references can be passed to shell scripts as
> standard input. Given the technique used in this patch, I'll probably
> re-write part of ob-awk.el.
Your patch is simply wonderful. It completely meets my need! Thanks a lot.
Look with the updated (and, now working) example of yesterday.
* Abstract
This script "americanizes" a European CSV file.
* Sample data
The following is a sample CSV file:
#+results: sample-csv
#+begin_example
Date;Amount;Account
28-05-2010;-6.806,25;999-1974050-30
04-06-2009;420,00;999-1500974-23
24-02-2009;-54,93;999-1974050-30
#+end_example
This input data will be used to show what the results of the transformations
are.
* Script
What the script must do is:
** Convert the date in American format
Convert the date in =MM/DD/YYYY= format.
#+srcname: convert-date
#+begin_src sh :stdin sample-csv :results output :exports both
sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g'
#+end_src
#+results: convert-date
#+begin_example
Date;Amount;Account
05/28/2010;-6.806,25;999-1974050-30
06/04/2009;420,00;999-1500974-23
02/24/2009;-54,93;999-1974050-30
#+end_example
** Convert the separators
Apply the following operations in order to "americanize" the CSV file received
from the bank:
- remove the dot used as thousands separator (=.= -> ==)
- replace the comma used as decimal separator by a dot (=,= -> =.=)
- replace other commas by a dot (=,= -> =.=)
- replace the semi-comma used as field separator by a comma (=;= -> =,=)
#+srcname: convert-separators
#+begin_src sh :stdin convert-date :results output :exports both
sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\
sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\
sed -r 's/,/./g' |\
sed -r 's/;/,/g'
#+end_src
#+results: convert-separators
#+begin_example
Date,Amount,Account
05/28/2010,-6806.25,999-1974050-30
06/04/2009,420.00,999-1500974-23
02/24/2009,-54.93,999-1974050-30
#+end_example
* Full code
The script is then:
#+begin_src sh :tangle americanize-csv.sh :noweb yes
#!/bin/bash
# americanize-csv.sh -- Convert CSV file to American format
# Usage: americanize-csv FILE.CSV
cat $1 |\
<<convert-date>> |\
<<convert-separators>>
exit 0
# americanize-csv.sh ends here
#+end_src
* Conclusions
The new =stdin= option allows one to:
- execute parts of code in-situ, in the Org buffer, documenting (and testing)
them.
- assemble all the parts in one single script file, by the means of literate
programming.
Go for applying it!
Thanks a lot, Eric, for your time.
Best regards,
Seb
--
Sébastien Vauban
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-24 19:03 ` Eric Schulte
2011-05-24 19:55 ` Sebastien Vauban
@ 2011-05-26 13:03 ` Eric Schulte
2011-05-26 15:15 ` Eric S Fraga
1 sibling, 1 reply; 13+ messages in thread
From: Eric Schulte @ 2011-05-26 13:03 UTC (permalink / raw)
To: emacs-orgmode
Eric Schulte <schulte.eric@gmail.com> writes:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>> [...]
>>
>>> As an example, I've worked up an very simple ob-awk.el file from
>>> ob-template.el, it is attached along with an example org-mode file which
>>> demonstrates its usage.
>>
>> Eric,
>>
>> this is great to see as I use awk quite often. What is involved in
>> extending this to be able to run an awk script on input from within the
>> org file (output of another babel block, for instance, as my typical use
>> of awk is to re-arrange output from another program...)? Or, if you
>> wish, can you suggest one of the ob-XXX modules that best illustrates
>> how to do this and I can give it a try?
>>
>
> I've made a quick change so that any variable named "stdin" is treated
> specially, in that, rather than using its value to replace strings of
> $stdin in the text of the awk code, the value of the stdin variable is
> saved into the file processed by awk. This allows awk to operate over
> Org-mode references.
>
> See the attached example file.
>
> If babel code block supported a pipe or an actual stdin header argument,
> that would be the ideal way to add this behavior, but currently nothing
> of that nature exists.
>
> Please let me know if this misses part of your suggestion, or more
> generally what else may be advisable before we add this to the core.
>
I've now added ob-awk.el to the Org-mode core. The newest version
incorporates some change inspired by recent work with Sebastien, notably
:stdin is now its own header argument, rather than a special variable
name.
Best -- Eric
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-26 11:18 ` Sebastien Vauban
@ 2011-05-26 13:37 ` Eric Schulte
0 siblings, 0 replies; 13+ messages in thread
From: Eric Schulte @ 2011-05-26 13:37 UTC (permalink / raw)
To: Sebastien Vauban; +Cc: emacs-orgmode
>
> Go for applying it!
>
Great, happy it works. I've just pushed this up to the git repository.
>
> Thanks a lot, Eric, for your time.
>
Its my pleasure. Best -- Eric
>
> Best regards,
> Seb
--
Eric Schulte
http://cs.unm.edu/~eschulte/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk
2011-05-26 13:03 ` Eric Schulte
@ 2011-05-26 15:15 ` Eric S Fraga
0 siblings, 0 replies; 13+ messages in thread
From: Eric S Fraga @ 2011-05-26 15:15 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode
Eric Schulte <schulte.eric@gmail.com> writes:
[...]
> I've now added ob-awk.el to the Org-mode core. The newest version
> incorporates some change inspired by recent work with Sebastien, notably
> :stdin is now its own header argument, rather than a special variable
> name.
>
> Best -- Eric
Thanks Eric.
My apologies for not trying out your interim solution but I have been
bogged down by marking examination scripts... the temptation to play
with org + babel had to be resisted! Anyway, from the discussion in
this thread related to stdin for sh scripts, it sounds like the final
solution is cleaner and more consistent!
Thanks again,
eric
--
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.311.g5c1cc)
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-05-26 16:34 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-24 9:31 org babel support for tcl and awk orgmode
2011-05-24 12:51 ` Eric Schulte
2011-05-24 17:53 ` Eric S Fraga
2011-05-24 19:03 ` Eric Schulte
2011-05-24 19:55 ` Sebastien Vauban
2011-05-24 23:51 ` Eric Schulte
2011-05-25 12:30 ` Sebastien Vauban
2011-05-25 15:57 ` Eric Schulte
2011-05-26 11:18 ` Sebastien Vauban
2011-05-26 13:37 ` Eric Schulte
2011-05-26 13:03 ` Eric Schulte
2011-05-26 15:15 ` Eric S Fraga
2011-05-24 18:57 ` orgmode
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).