* org babel support for tcl and awk @ 2011-05-24 9:31 orgmode 2011-05-24 12:51 ` Eric Schulte 0 siblings, 1 reply; 13+ messages in thread From: orgmode @ 2011-05-24 9:31 UTC (permalink / raw) To: emacs-orgmode; +Cc: orgmode Hi, I am looking for support for Tcl (and AWK) for org-babel. Both have a supplied emacs mode and Tcl also has an inferior interpreter mode. I was trying to do it myself, however I am quite lost in the instructions. Is there someone with the knowledge and willingness to provide a support file for org babel for Tcl (and maybe awk)? thanks. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 9:31 org babel support for tcl and awk orgmode @ 2011-05-24 12:51 ` Eric Schulte 2011-05-24 17:53 ` Eric S Fraga 2011-05-24 18:57 ` orgmode 0 siblings, 2 replies; 13+ messages in thread From: Eric Schulte @ 2011-05-24 12:51 UTC (permalink / raw) To: orgmode@h-rd.org; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 671 bytes --] Hi, Are you aware of the ob-template.el file [1], which can be used as a jumping off point to simplify the addition of new languages? After globally replacing the term "template" with you language name, the only function that necessarily needs to be re-written is the main `org-babel-execute:template' function. I would recommend starting with only non-session based evaluation, and then slowly adding functionality. If you run into any specific problems I am happy to help trouble shoot. As an example, I've worked up an very simple ob-awk.el file from ob-template.el, it is attached along with an example org-mode file which demonstrates its usage. Best -- Eric [-- Attachment #2: ob-awk.el --] [-- Type: application/emacs-lisp, Size: 6815 bytes --] [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: example.org --] [-- Type: text/x-org, Size: 419 bytes --] * example use of ob-awk Header Arguments - cmd-line :: command line flags to pass to =awk= - in-file :: the text file on which to run the resulting =awk= script Currently only string results are returned directly from STDOUT. Currently only external (i.e., non-session) evaluation is supported. #+begin_src awk :in-file columns-of-numbers.txt {print $1} #+end_src #+results: : 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 [-- Attachment #4: Type: text/plain, Size: 554 bytes --] "orgmode@h-rd.org" <orgmode@h-rd.org> writes: > Hi, > > I am looking for support for Tcl (and AWK) for org-babel. Both have a > supplied emacs mode and Tcl also has an inferior interpreter mode. I > was trying to do it myself, however I am quite lost in the > instructions. Is there someone with the knowledge and willingness to > provide a support file for org babel for Tcl (and maybe awk)? > > thanks. > > > Footnotes: [1] http://repo.or.cz/w/Worg.git/blob/HEAD:/org-contrib/babel/ob-template.el -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 12:51 ` Eric Schulte @ 2011-05-24 17:53 ` Eric S Fraga 2011-05-24 19:03 ` Eric Schulte 2011-05-24 18:57 ` orgmode 1 sibling, 1 reply; 13+ messages in thread From: Eric S Fraga @ 2011-05-24 17:53 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode Eric Schulte <schulte.eric@gmail.com> writes: [...] > As an example, I've worked up an very simple ob-awk.el file from > ob-template.el, it is attached along with an example org-mode file which > demonstrates its usage. Eric, this is great to see as I use awk quite often. What is involved in extending this to be able to run an awk script on input from within the org file (output of another babel block, for instance, as my typical use of awk is to re-arrange output from another program...)? Or, if you wish, can you suggest one of the ob-XXX modules that best illustrates how to do this and I can give it a try? Thanks, eric -- : Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1 : using Org-mode version 7.5 (release_7.5.298.g7c436) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 17:53 ` Eric S Fraga @ 2011-05-24 19:03 ` Eric Schulte 2011-05-24 19:55 ` Sebastien Vauban 2011-05-26 13:03 ` Eric Schulte 0 siblings, 2 replies; 13+ messages in thread From: Eric Schulte @ 2011-05-24 19:03 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1351 bytes --] Eric S Fraga <e.fraga@ucl.ac.uk> writes: > Eric Schulte <schulte.eric@gmail.com> writes: > > [...] > >> As an example, I've worked up an very simple ob-awk.el file from >> ob-template.el, it is attached along with an example org-mode file which >> demonstrates its usage. > > Eric, > > this is great to see as I use awk quite often. What is involved in > extending this to be able to run an awk script on input from within the > org file (output of another babel block, for instance, as my typical use > of awk is to re-arrange output from another program...)? Or, if you > wish, can you suggest one of the ob-XXX modules that best illustrates > how to do this and I can give it a try? > I've made a quick change so that any variable named "stdin" is treated specially, in that, rather than using its value to replace strings of $stdin in the text of the awk code, the value of the stdin variable is saved into the file processed by awk. This allows awk to operate over Org-mode references. See the attached example file. If babel code block supported a pipe or an actual stdin header argument, that would be the ideal way to add this behavior, but currently nothing of that nature exists. Please let me know if this misses part of your suggestion, or more generally what else may be advisable before we add this to the core. Cheers -- Eric [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: example.org --] [-- Type: text/x-org, Size: 653 bytes --] * awk example #+results: simple-table | 1 | 2 | 3 | | 4 | 5 | 6 | | 7 | 8 | 9 | #+begin_src awk :var stdin=simple-table {print $1} #+end_src #+results: : 1 : 4 : 7 #+results: simple-text : Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do : eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enimad : minim veniam, quis nostrud exercitation ullamco laboris nisi ut : aliquip ex ea commodo consequat. Duis aute irure dolor in : reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla : pariatur. Excepteur sint occaecat cupidatat non proident, sunt in : culpa qui officia deserunt mollit anim id est laborum. [-- Attachment #3: ob-awk.el --] [-- Type: application/emacs-lisp, Size: 4665 bytes --] [-- Attachment #4: Type: text/plain, Size: 67 bytes --] > > Thanks, > eric -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 19:03 ` Eric Schulte @ 2011-05-24 19:55 ` Sebastien Vauban 2011-05-24 23:51 ` Eric Schulte 2011-05-26 13:03 ` Eric Schulte 1 sibling, 1 reply; 13+ messages in thread From: Sebastien Vauban @ 2011-05-24 19:55 UTC (permalink / raw) To: emacs-orgmode-mXXj517/zsQ Hi Eric(s), Eric Schulte wrote: > Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes: >> this is great to see as I use awk quite often. What is involved in >> extending this to be able to run an awk script on input from within the org >> file (output of another babel block, for instance, as my typical use of awk >> is to re-arrange output from another program...)? Or, if you wish, can you >> suggest one of the ob-XXX modules that best illustrates how to do this and >> I can give it a try? > > I've made a quick change so that any variable named "stdin" is treated > specially, in that, rather than using its value to replace strings of $stdin > in the text of the awk code, the value of the stdin variable is saved into > the file processed by awk. This allows awk to operate over Org-mode > references. > > If babel code block supported a pipe or an actual stdin header argument, > that would be the ideal way to add this behavior, but currently nothing of > that nature exists. > > Please let me know if this misses part of your suggestion, or more generally > what else may be advisable before we add this to the core. Could this be implemented for sh as well? AFAI understand, this is exactly the missing piece for me to be able to: - run consecutive partial blocks of code in my Org buffer (seeing what they really do on input data), - export the full list of block as a script. This was described in http://www.mail-archive.com/emacs-orgmode-mXXj517/zsQ@public.gmane.org/msg36976.html, and still impossible -- for me! -- to do right now. But I'm not very sure I wrote my thoughts in an enough understandable way -- maybe not clear enough to me? Best regards, Seb -- Sébastien Vauban ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 19:55 ` Sebastien Vauban @ 2011-05-24 23:51 ` Eric Schulte 2011-05-25 12:30 ` Sebastien Vauban 0 siblings, 1 reply; 13+ messages in thread From: Eric Schulte @ 2011-05-24 23:51 UTC (permalink / raw) To: Sebastien Vauban; +Cc: emacs-orgmode "Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes: > Hi Eric(s), > > Eric Schulte wrote: >> Eric S Fraga <e.fraga@ucl.ac.uk> writes: >>> this is great to see as I use awk quite often. What is involved in >>> extending this to be able to run an awk script on input from within the org >>> file (output of another babel block, for instance, as my typical use of awk >>> is to re-arrange output from another program...)? Or, if you wish, can you >>> suggest one of the ob-XXX modules that best illustrates how to do this and >>> I can give it a try? >> >> I've made a quick change so that any variable named "stdin" is treated >> specially, in that, rather than using its value to replace strings of $stdin >> in the text of the awk code, the value of the stdin variable is saved into >> the file processed by awk. This allows awk to operate over Org-mode >> references. >> >> If babel code block supported a pipe or an actual stdin header argument, >> that would be the ideal way to add this behavior, but currently nothing of >> that nature exists. >> >> Please let me know if this misses part of your suggestion, or more generally >> what else may be advisable before we add this to the core. > > Could this be implemented for sh as well? > > AFAI understand, this is exactly the missing piece for me to be able to: > Hi Seb, Unfortunately this simple hack for ob-awk does not address the need you link to below -- which I am aware of and which is on my list of larger longer-term Babel development items. I think that a future piping implementation will be the ultimate solution to the issues you address. Such an implementation -- allowing data to flow between concurrently executing blocks utilizing posix pipes -- will require more sophisticated processes interaction and possibly some form of multi-threaded elisp execution. Best -- Eric > > - run consecutive partial blocks of code in my Org buffer (seeing what > they really do on input data), > > - export the full list of block as a script. > > This was described in > http://www.mail-archive.com/emacs-orgmode@gnu.org/msg36976.html, and still > impossible -- for me! -- to do right now. But I'm not very sure I wrote my > thoughts in an enough understandable way -- maybe not clear enough to me? > > Best regards, > Seb -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 23:51 ` Eric Schulte @ 2011-05-25 12:30 ` Sebastien Vauban 2011-05-25 15:57 ` Eric Schulte 0 siblings, 1 reply; 13+ messages in thread From: Sebastien Vauban @ 2011-05-25 12:30 UTC (permalink / raw) To: emacs-orgmode-mXXj517/zsQ Hi Eric, Eric Schulte wrote: > "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes: >> Eric Schulte wrote: >>> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> writes: I've made a quick change so that >>> any variable named "stdin" is treated specially, in that, rather than >>> using its value to replace strings of $stdin in the text of the awk code, >>> the value of the stdin variable is saved into the file processed by awk. >>> This allows awk to operate over Org-mode references. >>> >>> If babel code block supported a pipe or an actual stdin header argument, >>> that would be the ideal way to add this behavior, but currently nothing of >>> that nature exists. >>> >>> Please let me know if this misses part of your suggestion, or more >>> generally what else may be advisable before we add this to the core. >> >> Could this be implemented for sh as well? >> >> AFAI understand, this is exactly the missing piece for me to be able to: > > Unfortunately this simple hack for ob-awk does not address the need you link > to below -- which I am aware of and which is on my list of larger > longer-term Babel development items. I think that a future piping > implementation will be the ultimate solution to the issues you address. Glad to hear you understand my wish. It's not always easy to express myself in a very clean, with English not being my mother tongue, especially when trying to tackle difficult subjects. > Such an implementation -- allowing data to flow between concurrently > executing blocks utilizing posix pipes -- will require more sophisticated > processes interaction and possibly some form of multi-threaded elisp > execution. Just for the sake of clarity, I don't need concurrent or multi-threaded execution of any kind. My double-sided goal is: 1. to cut a shell script in small parts, and explain what every part does, with a runnable example (=C-c C-v C-e=). 2. to tangle the executable script out of the Babel document, by concatenating all its parts (=C-c C-v C-t=). A quite "dumb" example follows. I've made it as _minimal_ and as _complete_ as possible, to be able to _express my point_, for further reference. * Abstract This script "americanizes" a European CSV file. * Sample data The following is a sample CSV file: #+results: sample-csv #+begin_example Date;Amount;Account 28-05-2010;-6.806,25;999-1974050-30 04-06-2009;420,00;999-1500974-23 24-02-2009;-54,93;999-1974050-30 #+end_example * Script What the script must do is: ** Load the data Read the raw contents of the input file. #+srcname: load-data #+begin_src sh :var data=sample-csv :results output :exports both echo "$data" #+end_src #+results: load-data #+begin_example Date;Amount;Account 28-05-2010;-6.806,25;999-1974050-30 04-06-2009;420,00;999-1500974-23 24-02-2009;-54,93;999-1974050-30 #+end_example ** Convert the date in American format Convert the date in =MM/DD/YYYY= format. #+srcname: convert-date #+begin_src sh :var data=load-data :results output :exports both echo "$data" |\ sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\ sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g' #+end_src #+results: convert-date #+begin_example Date;Amount;Account 28/05/202010;-6.806,25;999-1974050-30 04/06/202009;420,00;999-1500974-23 24/02/202009;-54,93;999-1974050-30 #+end_example ** Convert the separators Apply the following operations in order to "americanize" the CSV file received from the bank: - remove the dot used as thousands separator (=.= -> ==) - replace the comma used as decimal separator by a dot (=,= -> =.=) - replace other commas by a dot (=,= -> =.=) - replace the semi-comma used as field separator by a comma (=;= -> =,=) #+srcname: convert-separators #+begin_src sh :var data=convert-date :results output :exports both echo "$data" |\ sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\ sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\ sed -r 's/,/./g' |\ sed -r 's/;/,/g' #+end_src #+results: convert-separators #+begin_example Date,Amount,Account 28/05/202010,-6806.25,999-1974050-30 04/06/202009,420.00,999-1500974-23 24/02/202009,-54.93,999-1974050-30 #+end_example * Full code The script is then: #+begin_src sh :tangle americanize-csv.sh :noweb yes #!/bin/bash # americanize-csv.sh -- Convert CSV file to American format # Usage: americanize-csv FILE.CSV cat $1 |\ <<convert-date>> |\ <<convert-separators>> exit 0 # americanize-csv.sh ends here #+end_src As you can see, the tangled script is not executable anymore, as I've been forced to put =echo $data= commands, in every apart code block, as their first command to run. #+begin_src sh #!/bin/bash # americanize-csv.sh -- Convert CSV file to American format # Usage: americanize-csv FILE.CSV cat $1 |\ echo "$data" |\ sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\ sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g' |\ echo "$data" |\ sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\ sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\ sed -r 's/,/./g' |\ sed -r 's/;/,/g' exit 0 # americanize-csv.sh ends here #+end_src Would I have the possibility to play with =stdin=, I could have "hidden" that first line, and assume all the code I'm writing will be executed against what's read on =stdin=. As well in the Org buffer, as in the stand-alone shell script. Right? #+begin_src sh #!/bin/bash # americanize-csv.sh -- Convert CSV file to American format # Usage: americanize-csv FILE.CSV cat $1 |\ sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' |\ sed -r 's/^([[:digit:]]{2})\/([[:digit:]]{2})\/([[:digit:]]{2})/\2\/\1\/20\3/g' |\ sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\ sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\ sed -r 's/,/./g' |\ sed -r 's/;/,/g' exit 0 # americanize-csv.sh ends here #+end_src * Conclusions As you can see, I did not really mean any concurrent execution. Simply being able to execute parts of code in-situ, in the Org buffer, to document (and test) what I'm writing. And to be able to assemble all the parts in one single script file, by the means of literate programming. Best regards, Seb -- Sébastien Vauban ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-25 12:30 ` Sebastien Vauban @ 2011-05-25 15:57 ` Eric Schulte 2011-05-26 11:18 ` Sebastien Vauban 0 siblings, 1 reply; 13+ messages in thread From: Eric Schulte @ 2011-05-25 15:57 UTC (permalink / raw) To: Sebastien Vauban; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 945 bytes --] > * Conclusions > > As you can see, I did not really mean any concurrent execution. Simply being > able to execute parts of code in-situ, in the Org buffer, to document (and > test) what I'm writing. > > And to be able to assemble all the parts in one single script file, by the > means of literate programming. > I see, you want to be able to construct a large pipe chain STDOUT>STDIN, but you don't care if the parts of the chain (e.g., the code block) execute in serial or concurrently (as they do in the shell). The attached patch (can be applied with "git am") implements this behavior as I understand it. The result is a new :stdin header argument with which org-mode references can be passed to shell scripts as standard input. Given the technique used in this patch, I'll probably re-write part of ob-awk.el. The following Org-mode snippet demonstrates it's use, please let me know if this works for your use-case described above. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: example.org --] [-- Type: text/x-org, Size: 316 bytes --] ** passing values through to STDIN of shell code blocks #+results: square-table | 1 | 2 | 3 | | 4 | 5 | 6 | | 7 | 8 | 9 | #+source: first-col #+begin_src sh :stdin square-table awk '{print $1}' #+end_src #+begin_src sh :stdin first-col sed 's/4/middle/g' #+end_src #+results: | 1 | | middle | | 7 | [-- Attachment #3: Type: text/plain, Size: 17 bytes --] Cheers -- Eric [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: 0001-sh-allow-passing-references-through-STDIN-of-a-shell.patch --] [-- Type: text/x-diff, Size: 4709 bytes --] From 03aa327d7950176e50ebb779c3f153bb9192dea5 Mon Sep 17 00:00:00 2001 From: Eric Schulte <schulte.eric@gmail.com> Date: Wed, 25 May 2011 09:50:36 -0600 Subject: [PATCH] sh: allow passing references through STDIN of a shell script * lisp/ob-sh.el (ob-ref): Uses ob-ref to resolve the value of the :stdin header argument. (org-babel-execute:sh): Use the :stdin header argument. (org-babel-sh-var-to-sh): Split the bulk of this function off into a new sub-function. (org-babel-sh-var-to-string): New function for converting elisp values to strings that make sense for parsing with sh. (org-babel-sh-evaluate): Adding "stdin" option to session and external evaluation options. --- lisp/ob-sh.el | 44 ++++++++++++++++++++++++++++++++------------ 1 files changed, 32 insertions(+), 12 deletions(-) diff --git a/lisp/ob-sh.el b/lisp/ob-sh.el index 128924d..10c08d4 100644 --- a/lisp/ob-sh.el +++ b/lisp/ob-sh.el @@ -28,6 +28,7 @@ ;;; Code: (require 'ob) +(require 'ob-ref) (require 'ob-comint) (require 'ob-eval) (require 'shell) @@ -57,10 +58,13 @@ This function is called by `org-babel-execute-src-block'." (let* ((session (org-babel-sh-initiate-session (cdr (assoc :session params)))) (result-params (cdr (assoc :result-params params))) + (stdin ((lambda (stdin) (when stdin (org-babel-sh-var-to-string + (org-babel-ref-resolve stdin)))) + (cdr (assoc :stdin params)))) (full-body (org-babel-expand-body:generic body params (org-babel-variable-assignments:sh params)))) (org-babel-reassemble-table - (org-babel-sh-evaluate session full-body result-params) + (org-babel-sh-evaluate session full-body result-params stdin) (org-babel-pick-name (cdr (assoc :colname-names params)) (cdr (assoc :colnames params))) (org-babel-pick-name @@ -101,14 +105,17 @@ This function is called by `org-babel-execute-src-block'." "Convert an elisp value to a shell variable. Convert an elisp var into a string of shell commands specifying a var of the same value." + (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep))) + +(defun org-babel-sh-var-to-string (var &optional sep) + "Convert an elisp value to a string." (flet ((echo-var (v) (if (stringp v) v (format "%S" v)))) - ((lambda (var) (format org-babel-sh-var-quote-fmt var)) - (cond - ((and (listp var) (listp (car var))) - (orgtbl-to-generic var (list :sep (or sep "\t") :fmt #'echo-var))) - ((listp var) - (mapconcat #'echo-var var "\n")) - (t (echo-var var)))))) + (cond + ((and (listp var) (listp (car var))) + (orgtbl-to-generic var (list :sep (or sep "\t") :fmt #'echo-var))) + ((listp var) + (mapconcat #'echo-var var "\n")) + (t (echo-var var))))) (defun org-babel-sh-table-or-results (results) "Convert RESULTS to an appropriate elisp value. @@ -128,7 +135,7 @@ Emacs-lisp table, otherwise return the results as a string." (defvar org-babel-sh-eoe-output "org_babel_sh_eoe" "String to indicate that evaluation has completed.") -(defun org-babel-sh-evaluate (session body &optional result-params) +(defun org-babel-sh-evaluate (session body &optional result-params stdin) "Pass BODY to the Shell process in BUFFER. If RESULT-TYPE equals 'output then return a list of the outputs of the statements in BODY, if RESULT-TYPE equals 'value then @@ -141,8 +148,19 @@ return the value of the last statement in BODY." (let ((tmp-file (org-babel-temp-file "sh-"))) (with-temp-file tmp-file (insert results)) (org-babel-import-elisp-from-file tmp-file))))) - (if (not session) - (org-babel-eval org-babel-sh-command (org-babel-trim body)) + (cond + (stdin ; external shell script w/STDIN + (let ((script-file (org-babel-temp-file "sh-script-")) + (stdin-file (org-babel-temp-file "sh-stdin-"))) + (with-temp-file script-file (insert body)) + (with-temp-file stdin-file (insert stdin)) + (with-temp-buffer + (call-process-shell-command + (format "%s %s" org-babel-sh-command script-file) + stdin-file + (current-buffer)) + (buffer-string)))) + (session ; session evaluation (mapconcat #'org-babel-sh-strip-weird-long-prompt (mapcar @@ -156,7 +174,9 @@ return the value of the last statement in BODY." (append (split-string (org-babel-trim body) "\n") (list org-babel-sh-eoe-indicator)))) - 2)) "\n")))) + 2)) "\n")) + ('otherwise ; external shell script + (org-babel-eval org-babel-sh-command (org-babel-trim body)))))) (defun org-babel-sh-strip-weird-long-prompt (string) "Remove prompt cruft from a string of shell output." -- 1.7.4.1 [-- Attachment #5: Type: text/plain, Size: 74 bytes --] > > Best regards, > Seb -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-25 15:57 ` Eric Schulte @ 2011-05-26 11:18 ` Sebastien Vauban 2011-05-26 13:37 ` Eric Schulte 0 siblings, 1 reply; 13+ messages in thread From: Sebastien Vauban @ 2011-05-26 11:18 UTC (permalink / raw) To: emacs-orgmode-mXXj517/zsQ Hi Eric, Eric Schulte wrote: >> As you can see, I did not really mean any concurrent execution. Simply >> being able to execute parts of code in-situ, in the Org buffer, to document >> (and test) what I'm writing. >> >> And to be able to assemble all the parts in one single script file, by the >> means of literate programming. > > I see, you want to be able to construct a large pipe chain STDOUT>STDIN, That's it! > but you don't care if the parts of the chain (e.g., the code block) execute > in serial or concurrently (as they do in the shell). For me, there is no concept of serial or concurrent execution here, as I am executing manually the calls, when writing the Org document. Not sure to understand you... Are you talking of what happens for the export, maybe? Are you talking of the shell constructs which will be used in the way the "full script" is assembled? > The attached patch (can be applied with "git am") implements this > behavior as I understand it. The result is a new :stdin header argument > with which org-mode references can be passed to shell scripts as > standard input. Given the technique used in this patch, I'll probably > re-write part of ob-awk.el. Your patch is simply wonderful. It completely meets my need! Thanks a lot. Look with the updated (and, now working) example of yesterday. * Abstract This script "americanizes" a European CSV file. * Sample data The following is a sample CSV file: #+results: sample-csv #+begin_example Date;Amount;Account 28-05-2010;-6.806,25;999-1974050-30 04-06-2009;420,00;999-1500974-23 24-02-2009;-54,93;999-1974050-30 #+end_example This input data will be used to show what the results of the transformations are. * Script What the script must do is: ** Convert the date in American format Convert the date in =MM/DD/YYYY= format. #+srcname: convert-date #+begin_src sh :stdin sample-csv :results output :exports both sed -r 's/^([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{4})/\2\/\1\/\3/g' #+end_src #+results: convert-date #+begin_example Date;Amount;Account 05/28/2010;-6.806,25;999-1974050-30 06/04/2009;420,00;999-1500974-23 02/24/2009;-54,93;999-1974050-30 #+end_example ** Convert the separators Apply the following operations in order to "americanize" the CSV file received from the bank: - remove the dot used as thousands separator (=.= -> ==) - replace the comma used as decimal separator by a dot (=,= -> =.=) - replace other commas by a dot (=,= -> =.=) - replace the semi-comma used as field separator by a comma (=;= -> =,=) #+srcname: convert-separators #+begin_src sh :stdin convert-date :results output :exports both sed -r 's/([[:digit:]])\.([[:digit:]]{3})/\1\2/g' |\ sed -r 's/([[:digit:]]),([[:digit:]]{2})/\1.\2/g' |\ sed -r 's/,/./g' |\ sed -r 's/;/,/g' #+end_src #+results: convert-separators #+begin_example Date,Amount,Account 05/28/2010,-6806.25,999-1974050-30 06/04/2009,420.00,999-1500974-23 02/24/2009,-54.93,999-1974050-30 #+end_example * Full code The script is then: #+begin_src sh :tangle americanize-csv.sh :noweb yes #!/bin/bash # americanize-csv.sh -- Convert CSV file to American format # Usage: americanize-csv FILE.CSV cat $1 |\ <<convert-date>> |\ <<convert-separators>> exit 0 # americanize-csv.sh ends here #+end_src * Conclusions The new =stdin= option allows one to: - execute parts of code in-situ, in the Org buffer, documenting (and testing) them. - assemble all the parts in one single script file, by the means of literate programming. Go for applying it! Thanks a lot, Eric, for your time. Best regards, Seb -- Sébastien Vauban ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-26 11:18 ` Sebastien Vauban @ 2011-05-26 13:37 ` Eric Schulte 0 siblings, 0 replies; 13+ messages in thread From: Eric Schulte @ 2011-05-26 13:37 UTC (permalink / raw) To: Sebastien Vauban; +Cc: emacs-orgmode > > Go for applying it! > Great, happy it works. I've just pushed this up to the git repository. > > Thanks a lot, Eric, for your time. > Its my pleasure. Best -- Eric > > Best regards, > Seb -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 19:03 ` Eric Schulte 2011-05-24 19:55 ` Sebastien Vauban @ 2011-05-26 13:03 ` Eric Schulte 2011-05-26 15:15 ` Eric S Fraga 1 sibling, 1 reply; 13+ messages in thread From: Eric Schulte @ 2011-05-26 13:03 UTC (permalink / raw) To: emacs-orgmode Eric Schulte <schulte.eric@gmail.com> writes: > Eric S Fraga <e.fraga@ucl.ac.uk> writes: > >> Eric Schulte <schulte.eric@gmail.com> writes: >> >> [...] >> >>> As an example, I've worked up an very simple ob-awk.el file from >>> ob-template.el, it is attached along with an example org-mode file which >>> demonstrates its usage. >> >> Eric, >> >> this is great to see as I use awk quite often. What is involved in >> extending this to be able to run an awk script on input from within the >> org file (output of another babel block, for instance, as my typical use >> of awk is to re-arrange output from another program...)? Or, if you >> wish, can you suggest one of the ob-XXX modules that best illustrates >> how to do this and I can give it a try? >> > > I've made a quick change so that any variable named "stdin" is treated > specially, in that, rather than using its value to replace strings of > $stdin in the text of the awk code, the value of the stdin variable is > saved into the file processed by awk. This allows awk to operate over > Org-mode references. > > See the attached example file. > > If babel code block supported a pipe or an actual stdin header argument, > that would be the ideal way to add this behavior, but currently nothing > of that nature exists. > > Please let me know if this misses part of your suggestion, or more > generally what else may be advisable before we add this to the core. > I've now added ob-awk.el to the Org-mode core. The newest version incorporates some change inspired by recent work with Sebastien, notably :stdin is now its own header argument, rather than a special variable name. Best -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-26 13:03 ` Eric Schulte @ 2011-05-26 15:15 ` Eric S Fraga 0 siblings, 0 replies; 13+ messages in thread From: Eric S Fraga @ 2011-05-26 15:15 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode Eric Schulte <schulte.eric@gmail.com> writes: [...] > I've now added ob-awk.el to the Org-mode core. The newest version > incorporates some change inspired by recent work with Sebastien, notably > :stdin is now its own header argument, rather than a special variable > name. > > Best -- Eric Thanks Eric. My apologies for not trying out your interim solution but I have been bogged down by marking examination scripts... the temptation to play with org + babel had to be resisted! Anyway, from the discussion in this thread related to stdin for sh scripts, it sounds like the final solution is cleaner and more consistent! Thanks again, eric -- : Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1 : using Org-mode version 7.5 (release_7.5.311.g5c1cc) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: org babel support for tcl and awk 2011-05-24 12:51 ` Eric Schulte 2011-05-24 17:53 ` Eric S Fraga @ 2011-05-24 18:57 ` orgmode 1 sibling, 0 replies; 13+ messages in thread From: orgmode @ 2011-05-24 18:57 UTC (permalink / raw) To: emacs-orgmode, Eric Schulte; +Cc: orgmode Hi Eric, yes I am aware of op-template and tried to use it. However it was not clear to me how to proceed and I looked into ob-perl, ob-ruby, ob-scheme and ob-python. But it seemed to me they use a different structure than op-template and I was stuck. I also saw that the file ob-template is documented, however I am not so good in org-emacs-speak that I can decipher it. Really thanks for the awk example, I will tru to study it. thanks. Quoting Eric Schulte <schulte.eric@gmail.com>: > Hi, > > Are you aware of the ob-template.el file [1], which can be used as a > jumping off point to simplify the addition of new languages? After > globally replacing the term "template" with you language name, the only > function that necessarily needs to be re-written is the main > `org-babel-execute:template' function. > > I would recommend starting with only non-session based evaluation, and > then slowly adding functionality. If you run into any specific problems > I am happy to help trouble shoot. > > As an example, I've worked up an very simple ob-awk.el file from > ob-template.el, it is attached along with an example org-mode file which > demonstrates its usage. > > Best -- Eric > > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-05-26 16:34 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-05-24 9:31 org babel support for tcl and awk orgmode 2011-05-24 12:51 ` Eric Schulte 2011-05-24 17:53 ` Eric S Fraga 2011-05-24 19:03 ` Eric Schulte 2011-05-24 19:55 ` Sebastien Vauban 2011-05-24 23:51 ` Eric Schulte 2011-05-25 12:30 ` Sebastien Vauban 2011-05-25 15:57 ` Eric Schulte 2011-05-26 11:18 ` Sebastien Vauban 2011-05-26 13:37 ` Eric Schulte 2011-05-26 13:03 ` Eric Schulte 2011-05-26 15:15 ` Eric S Fraga 2011-05-24 18:57 ` orgmode
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).