emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [babel] How to kill two birds with one stone?
@ 2011-02-04 16:00 Sébastien Vauban
  2011-02-04 17:43 ` Dan Davison
  0 siblings, 1 reply; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-04 16:00 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

#+TITLE:     Document a shell script as separate blocks
#+DATE:      2011-02-04
#+LANGUAGE:  en_US

* Abstract

When writing shell scripts, I'd like to kill *two* birds with one Babel stone:

- Be able to *execute the script in situ*, so that I get a copy of the results
  stored in (and "versioned" with) my documentation.

- Be able to *chain the code blocks*, so that I tangle a "natural" script into
  a file for later execution.

I guess I currently miss some points in order to reach that goal in a clean
way. Could you give me advice on how to get a step further down the route?

* Sample code

For the sake of clarity, let's take a simple problem: I'd like to generate a
DOT graph of links between all files from a directory tree.

The procedure:

1. (Recursively) list all files inside the directory.

2. For each file, (recursively) search for its name referenced in all the
   files.

3. Generate a DOT representation of the link between files.

** List all files under current directory

#+srcname: dw-file-tree
#+begin_src sh :results output
cd ~/Some-Project-Dir
find . -type f -print | grep -v .svn | head -n 5
#+end_src

#+results: dw-file-tree
#+begin_example
./.cvsignore
./charge_dim
./charge_fct
./compte
./controle_config
#+end_example

Here, I voluntary limit the number of results to the first 5 files, for the
compactness of this example. This sample does not include any file from
subdirectories, but it doesn't matter.

** Search recursively for anything about a file

Search through all files (ignoring =.svn= directories) for any reference to
filename given as parameter.

#+srcname: search-files-pointing-to-this-file
#+begin_src sh :results output :var f="charge_dim"
cd ~/Some-Project-Dir
find . -not \( -name .svn -prune \) -type f -print0 |\
xargs -0 grep -i --files-with-matches "$f"
#+end_src

#+results: search-files-pointing-to-this-file
#+begin_example
./.cvsignore
./charge_dim
./compte
./IFP/Chrg_dim
./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12
./principal.env
./29Aalst/Chrg_dim
./29Aalst/Chrg_dim_interactif
./29Aalst/Publ_dim_interactif
#+end_example

HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE
RESULTS.

** Convert to a DOT representation

For every file pointing to the file given in parameter, generate an "arrow"
(edge) in DOT representation.

#+srcname: dot-arrow-from-files-pointing-to-this-file
#+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file
for i in $(echo "$data"); do echo "    $(basename $i) -> $f"; done
#+end_src

#+results: dot-arrow-from-files-pointing-to-this-file
#+begin_example
    .cvsignore -> charge_dim
    charge_dim -> charge_dim
    compte -> charge_dim
    Chrg_dim -> charge_dim
    Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim
    principal.env -> charge_dim
    Chrg_dim -> charge_dim
    Chrg_dim_interactif -> charge_dim
    Publ_dim_interactif -> charge_dim
#+end_example

HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND
DOCUMENTATION) PURPOSE.

* Problem

All of the above nicely answer my first goal ("Be able to execute the script
in situ, so that I get a copy of the results stored in my documentation").

It does not allow for the second one: *how to chain the calls together*?

For example, just for chaining steps 2 and 3 (the easier to chain, I think),
I'd like to be able to write something like this:

#+srcname: search-links-and-generate-dot-arrow
#+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file :noweb yes
for i in (
    <<search-files-pointing-to-this-file>>);
do echo "    $(basename $i) -> $f"; done
#+end_src

#+results: search-links-and-generate-dot-arrow

But it yields an error:

#+begin_src text :eval no
sh: line 14: syntax error near unexpected token `('
sh: line 14: `for i in ('
#+end_src

Note, in the latter code block, that I did not even tried to really chain
steps 2 and 3: I'm rewriting step 3, including step 2 inside it.

*I certainly miss a smarter way* to achieve the above.

Quite important as well, is that the tangled file seems natural to read (for
someone that would not read it from the Org file). This means we would have to
minimize the use of "echo" and "tee" commands, among others -- but that's a
nice to have.

Do you see my point?

Do you have ideas on how to go that way?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-04 16:00 [babel] How to kill two birds with one stone? Sébastien Vauban
@ 2011-02-04 17:43 ` Dan Davison
  2011-02-04 22:23   ` Sébastien Vauban
  2011-02-06 16:51   ` Sébastien Vauban
  0 siblings, 2 replies; 15+ messages in thread
From: Dan Davison @ 2011-02-04 17:43 UTC (permalink / raw)
  To: Sébastien Vauban; +Cc: emacs org-mode mailing list

Hi Seb,

Cool post. I hope someone has some good ideas in this thread. Some quick
responses / questions below.

> #+TITLE:     Document a shell script as separate blocks
> #+DATE:      2011-02-04
> #+LANGUAGE:  en_US
>
> * Abstract
>
> When writing shell scripts, I'd like to kill *two* birds with one Babel stone:
>
> - Be able to *execute the script in situ*, so that I get a copy of the results
>   stored in (and "versioned" with) my documentation.
>
> - Be able to *chain the code blocks*, so that I tangle a "natural" script into
>   a file for later execution.
>
> I guess I currently miss some points in order to reach that goal in a clean
> way. Could you give me advice on how to get a step further down the route?
>
> * Sample code
>
> For the sake of clarity, let's take a simple problem: I'd like to generate a
> DOT graph of links between all files from a directory tree.

So if file a.org contains

-----
jhasg
[[file:b.org]]
[[file:c.org]]
hbjgv
-----

then you want an arrow from node a to node b and from node a to node c, right?

>
> The procedure:
>
> 1. (Recursively) list all files inside the directory.
>
> 2. For each file, (recursively) search for its name referenced in all the
>    files.
>
> 3. Generate a DOT representation of the link between files.
>
> ** List all files under current directory
>
> #+srcname: dw-file-tree
> #+begin_src sh :results output
> cd ~/Some-Project-Dir
> find . -type f -print | grep -v .svn | head -n 5
> #+end_src
>
> #+results: dw-file-tree
> #+begin_example
> ./.cvsignore
> ./charge_dim
> ./charge_fct
> ./compte
> ./controle_config
> #+end_example
>
> Here, I voluntary limit the number of results to the first 5 files, for the
> compactness of this example. This sample does not include any file from
> subdirectories, but it doesn't matter.
>
> ** Search recursively for anything about a file
>
> Search through all files (ignoring =.svn= directories) for any reference to
> filename given as parameter.
>
> #+srcname: search-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim"
> cd ~/Some-Project-Dir
> find . -not \( -name .svn -prune \) -type f -print0 |\
> xargs -0 grep -i --files-with-matches "$f"
> #+end_src
>
> #+results: search-files-pointing-to-this-file
> #+begin_example
> ./.cvsignore
> ./charge_dim
> ./compte
> ./IFP/Chrg_dim
> ./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12
> ./principal.env
> ./29Aalst/Chrg_dim
> ./29Aalst/Chrg_dim_interactif
> ./29Aalst/Publ_dim_interactif
> #+end_example
>
> HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE
> RESULTS.
>
> ** Convert to a DOT representation
>
> For every file pointing to the file given in parameter, generate an "arrow"
> (edge) in DOT representation.
>
> #+srcname: dot-arrow-from-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file
> for i in $(echo "$data"); do echo "    $(basename $i) -> $f"; done
> #+end_src
>
> #+results: dot-arrow-from-files-pointing-to-this-file
> #+begin_example
>     .cvsignore -> charge_dim
>     charge_dim -> charge_dim
>     compte -> charge_dim
>     Chrg_dim -> charge_dim
>     Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim
>     principal.env -> charge_dim
>     Chrg_dim -> charge_dim
>     Chrg_dim_interactif -> charge_dim
>     Publ_dim_interactif -> charge_dim
> #+end_example
>
> HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND
> DOCUMENTATION) PURPOSE.
>
> * Problem
>
> All of the above nicely answer my first goal ("Be able to execute the script
> in situ, so that I get a copy of the results stored in my documentation").
>
> It does not allow for the second one: *how to chain the calls together*?
>
> For example, just for chaining steps 2 and 3 (the easier to chain, I think),
> I'd like to be able to write something like this:
>
> #+srcname: search-links-and-generate-dot-arrow
> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file :noweb yes
> for i in (
>     <<search-files-pointing-to-this-file>>);
> do echo "    $(basename $i) -> $f"; done
> #+end_src

Don't forget about C-c C-v C-v `org-babel-expand-src-block' when
debugging this problem. I've started using that a lot.

My shell scripting may be too weak to help here. (Which is one point,
I'd be tempted to bail out to another language on this one) What exactly
are you doing there? Executing the code from the other block in a
subshell and using the stdout as the set to loop over? You're sure you
don't want parentheses <<search-files-pointing-to-this-file>>() ? Or $(
... ) / backticks ? How are you going to pass arguments to the
search-files-pointing src block?

>
> #+results: search-links-and-generate-dot-arrow
>
> But it yields an error:
>
> #+begin_src text :eval no
> sh: line 14: syntax error near unexpected token `('
> sh: line 14: `for i in ('
> #+end_src
>
> Note, in the latter code block, that I did not even tried to really chain
> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>
> *I certainly miss a smarter way* to achieve the above.

I think a relevant point here is that Org doesn't yet have the ability
to pass data on standard input to a code block. I.e. a :stdin header
arg. I don't think it's that hard, someone would just need to rework
`org-babel-eval' so that it puts the code into a temporary file, freeing
up stdin to be used for data (and therefore we would no longer be able
to use shell-command-on-region but some other command (call-process I
think?).)


Then you'd be able to do something like

#+srcname: search-links-and-generate-dot-arrow
#+header: :stdin search-files-pointing-to-this-file
#+begin_src sh :results output :var f="charge_dim"
while read f; do
    echo "    $(basename $i) -> $f";
done
#+end_src


I'll be interested to see the solution to all this.

Dan

>
> Quite important as well, is that the tangled file seems natural to read (for
> someone that would not read it from the Org file). This means we would have to
> minimize the use of "echo" and "tee" commands, among others -- but that's a
> nice to have.
>
> Do you see my point?
>
> Do you have ideas on how to go that way?
>
> Best regards,
>   Seb

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-04 17:43 ` Dan Davison
@ 2011-02-04 22:23   ` Sébastien Vauban
  2011-02-06 16:51   ` Sébastien Vauban
  1 sibling, 0 replies; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-04 22:23 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Dan,

Myself quickly reacting on this...

Dan Davison wrote:
> Cool post. I hope someone has some good ideas in this thread.

Thanks. Once solved, this one (and many more experiments I'm jotting down)
could become "case studies" on Worg, or so.

> Some quick responses / questions below.
>
>> #+TITLE:     Document a shell script as separate blocks
>> #+DATE:      2011-02-04
>> #+LANGUAGE:  en_US
>>
>> * Abstract
>>
>> When writing shell scripts, I'd like to kill *two* birds with one Babel stone:
>>
>> - Be able to *execute the script in situ*, so that I get a copy of the results
>>   stored in (and "versioned" with) my documentation.
>>
>> - Be able to *chain the code blocks*, so that I tangle a "natural" script into
>>   a file for later execution.
>>
>> I guess I currently miss some points in order to reach that goal in a clean
>> way. Could you give me advice on how to get a step further down the route?
>>
>> * Sample code
>>
>> For the sake of clarity, let's take a simple problem: I'd like to generate a
>> DOT graph of links between all files from a directory tree.
>
> So if file a.org contains
>
> -----
> jhasg
> [[file:b.org]]
> [[file:c.org]]
> hbjgv
> -----
>
> then you want an arrow from node a to node b and from node a to node c, right?

Exactly. Though, this is not limited to Org files, but any file (and any
filename) in fact...

>> The procedure:
>>
>> 1. (Recursively) list all files inside the directory.
>>
>> 2. For each file, (recursively) search for its name referenced in all the
>>    files.
>>
>> 3. Generate a DOT representation of the link between files.
>>
>> ** List all files under current directory
>>
>> #+srcname: dw-file-tree
>> #+begin_src sh :results output
>> cd ~/Some-Project-Dir
>> find . -type f -print | grep -v .svn | head -n 5
>> #+end_src
>>
>> #+results: dw-file-tree
>> #+begin_example
>> ./.cvsignore
>> ./charge_dim
>> ./charge_fct
>> ./compte
>> ./controle_config
>> #+end_example
>>
>> Here, I voluntary limit the number of results to the first 5 files, for the
>> compactness of this example. This sample does not include any file from
>> subdirectories, but it doesn't matter.
>>
>> ** Search recursively for anything about a file
>>
>> Search through all files (ignoring =.svn= directories) for any reference to
>> filename given as parameter.
>>
>> #+srcname: search-files-pointing-to-this-file
>> #+begin_src sh :results output :var f="charge_dim"
>> cd ~/Some-Project-Dir
>> find . -not \( -name .svn -prune \) -type f -print0 |\
>> xargs -0 grep -i --files-with-matches "$f"
>> #+end_src
>>
>> #+results: search-files-pointing-to-this-file
>> #+begin_example
>> ./.cvsignore
>> ./charge_dim
>> ./compte
>> ./IFP/Chrg_dim
>> ./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12
>> ./principal.env
>> ./29Aalst/Chrg_dim
>> ./29Aalst/Chrg_dim_interactif
>> ./29Aalst/Publ_dim_interactif
>> #+end_example
>>
>> HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE
>> RESULTS.
>>
>> ** Convert to a DOT representation
>>
>> For every file pointing to the file given in parameter, generate an "arrow"
>> (edge) in DOT representation.
>>
>> #+srcname: dot-arrow-from-files-pointing-to-this-file
>> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file
>> for i in $(echo "$data"); do echo "    $(basename $i) -> $f"; done
>> #+end_src
>>
>> #+results: dot-arrow-from-files-pointing-to-this-file
>> #+begin_example
>>     .cvsignore -> charge_dim
>>     charge_dim -> charge_dim
>>     compte -> charge_dim
>>     Chrg_dim -> charge_dim
>>     Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim
>>     principal.env -> charge_dim
>>     Chrg_dim -> charge_dim
>>     Chrg_dim_interactif -> charge_dim
>>     Publ_dim_interactif -> charge_dim
>> #+end_example
>>
>> HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND
>> DOCUMENTATION) PURPOSE.
>>
>> * Problem
>>
>> All of the above nicely answer my first goal ("Be able to execute the script
>> in situ, so that I get a copy of the results stored in my documentation").
>>
>> It does not allow for the second one: *how to chain the calls together*?
>>
>> For example, just for chaining steps 2 and 3 (the easier to chain, I think),
>> I'd like to be able to write something like this:
>>
>> #+srcname: search-links-and-generate-dot-arrow
>> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file :noweb yes
>> for i in (
>>     <<search-files-pointing-to-this-file>>);
>> do echo "    $(basename $i) -> $f"; done
>> #+end_src
>
> Don't forget about C-c C-v C-v `org-babel-expand-src-block' when
> debugging this problem. I've started using that a lot.
>
> My shell scripting may be too weak to help here. (Which is one point,
> I'd be tempted to bail out to another language on this one)

That's another point. Once written in Bash scripts, I'd be interested (and I
think it is very interesting from a Babel point of view) to see the same
functionality implemented in other languages.

> What exactly are you doing there? Executing the code from the other block in
> a subshell and using the stdout as the set to loop over?

I'm not especially wanting to try executing the code block in a subshell. Yes
for the "in situ execution" point of view, no from the "code to be tangled"
point of view...

> You're sure you don't want parentheses
> <<search-files-pointing-to-this-file>>() ? Or $( ... ) / backticks ?

Parenthesis would run the code... But I need the code to be written into this
block of code (in another valid form) so that the tangled file will be runable
in any directory, with any structure below it, at any point in time.

> How are you going to pass arguments to the search-files-pointing src block?

Good question. Unanswered at this point in time...

>> #+results: search-links-and-generate-dot-arrow
>>
>> But it yields an error:
>>
>> #+begin_src text :eval no
>> sh: line 14: syntax error near unexpected token `('
>> sh: line 14: `for i in ('
>> #+end_src
>>
>> Note, in the latter code block, that I did not even tried to really chain
>> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>>
>> *I certainly miss a smarter way* to achieve the above.
>
> I think a relevant point here is that Org doesn't yet have the ability
> to pass data on standard input to a code block. I.e. a :stdin header
> arg. I don't think it's that hard, someone would just need to rework
> `org-babel-eval' so that it puts the code into a temporary file, freeing
> up stdin to be used for data (and therefore we would no longer be able
> to use shell-command-on-region but some other command (call-process I
> think?).)
>
> Then you'd be able to do something like
>
> #+srcname: search-links-and-generate-dot-arrow
> #+header: :stdin search-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim"
> while read f; do
>     echo "    $(basename $i) -> $f";
> done
> #+end_src

I must admit it's too late now for me to be sure to grasp this latter
paragraphs. I'll read them tomorrow when I'll be able to think about it.

> I'll be interested to see the solution to all this.

So am I, with or without any extension to the current Babel. I'm not sure yet
that it is, or that it is not, doable right now... Maybe a mind swap is just
necessary to view the problem under new light...

Thanks for your help.

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-04 17:43 ` Dan Davison
  2011-02-04 22:23   ` Sébastien Vauban
@ 2011-02-06 16:51   ` Sébastien Vauban
  2011-02-20  8:57     ` Eric Schulte
  1 sibling, 1 reply; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-06 16:51 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Dan,

Dan Davison wrote:
> Cool post. I hope someone has some good ideas in this thread. Some quick
> responses / questions below.
>
>> Note, in the latter code block, that I did not even tried to really chain
>> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>>
>> *I certainly miss a smarter way* to achieve the above.
>
> I think a relevant point here is that Org doesn't yet have the ability to
> pass data on standard input to a code block. I.e. a :stdin header arg. I
> don't think it's that hard, someone would just need to rework
> `org-babel-eval' so that it puts the code into a temporary file, freeing up
> stdin to be used for data

I think you exactly spotted the problem.

> (and therefore we would no longer be able to use
> shell-command-on-region but some other command (call-process I think?).)

I just don't understand this last sentence, by lack of knowledge on Babel's
internals.

> Then you'd be able to do something like
>
> #+srcname: search-links-and-generate-dot-arrow
> #+header: :stdin search-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim"
> while read f; do
>     echo "    $(basename $i) -> $f";
> done
> #+end_src
>
> I'll be interested to see the solution to all this.

I've tried to rewrite the example in a much cleaner way, showing what it
should like in the best of the worlds, with a working code at hand...

BTW, I think the following could be of use, with maybe slight modifications,
even for projects like Worg: identifying all relationships between files,
showing files that aren't referenced, etc.

#+TITLE:     Graph file dependencies
#+DATE:      2011-02-06
#+BABEL:    :dir ~/src/Worg

* Context

We want to demonstrate how to document a script in a very neat way (IMHO),
that is:

- By defining and explaining multiple small code blocks, using them later in a
  tangle file for constructing the "full code".

- By showing the effect of every small code block, that is what it returns
  when applied on test input data.

The latter is the problem, as the code has to be able to take a results set as
if it would come from =stdin=.

* Code

For the sake of clarity, a real-life example that graph dependencies between
files (based on their /basename/).

** List all files

Simple file command, ignoring =.svn= directories.

#+srcname: file-tree
#+begin_src sh :results output
find . -not \( -name .svn -prune \) -type f -print | head -n 5
#+end_src

#+results: file-tree
#+begin_example
./digraph.dot
./full-code.sh
./graph-circo.pdf
./graph-dot.pdf
./graph-fdp.pdf
#+end_example

** Search recursively for anything about a file

Grep-search through files, ignoring =.svn= directories.

#+srcname: search-files-pointing-to-this-file
#+begin_src sh :results output :var toname="charge_dim"
find . -not \( -name .svn -prune \) -type f -print0 |\
xargs -0 grep -i --files-with-matches "$toname"
#+end_src

#+results: search-files-pointing-to-this-file
: ./graph-file-dependencies.txt

** Convert to DOT

In real life, the following block of code must read its input from =stdin=.

For /in situ execution/, I should be able to say that =stdin= is equal to any
results set (here: =search-files-pointing-to-this-file=).

#+srcname: make-dot-arrow-for-files-pointing-to-this-file
#+begin_src sh :results output :var toname="charge_dim"
while read -r fromname
do
    echo "    \"${fromname##*/}\" -> \"$toname\""
done
#+end_src

#+results: make-dot-arrow-for-files-pointing-to-this-file

** Full code: generate the DOT file

#+begin_src sh :results output :file digraph.dot :noweb yes :tangle full-code.sh
echo 'digraph G {'
echo '    node [shape=diamond,style=filled,color=lightgrey]; ".cvsignore";'
echo '    node [shape=box,color=yellow];'
echo ''

<<file-tree>> |\
while read -r fname
do
    toname="${fname##*/}" # basename of fname
    echo "# Files pointing to \"$toname\"..."
    <<search-files-pointing-to-this-file>> |\
    <<make-dot-arrow-for-files-pointing-to-this-file>>
    echo ""
done

echo '}'
#+end_src

#+results:
[[file:digraph.dot]]

** Draw multiple graphs

#+srcname: draw-graphs
#+begin_src sh :dir ~/Projects :var infile="digraph.dot" :var outfileprefix="digraph"
for cmd in dot neato twopi circo fdp sfdp;
do
    echo $cmd...
    time $cmd -Gcharset=latin1 -Tpdf -o$outfileprefix-$cmd.pdf $infile
    echo ""
done
#+end_src

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Re: [babel] How to kill two birds with one stone?
  2011-02-06 16:51   ` Sébastien Vauban
@ 2011-02-20  8:57     ` Eric Schulte
  2011-02-25 14:27       ` Sébastien Vauban
  2011-02-28 15:16       ` [babel] How to kill two birds with one stone? Sébastien Vauban
  0 siblings, 2 replies; 15+ messages in thread
From: Eric Schulte @ 2011-02-20  8:57 UTC (permalink / raw)
  To: Sébastien Vauban; +Cc: emacs-orgmode

Hi,

I haven't followed this discussion very closely, but I'm not sure why it
would be necessary to pass data through STDIN rather than through a
variable or an external file.

I took a shot at the dot graph example you proposed, the following works
for me over a simple example directory.

Best -- Eric

directory to search
#+results: graph-dir
: graph-dir

list all files in dir
#+source: graph-files
#+begin_src sh :results vector :var dir=graph-dir
  find $dir -type f -exec basename {} \;
#+end_src

#+results: graph-files
| other |
| dan   |
| eric  |
| seb   |

association of files with mentions
#+source: graph-associations
#+begin_src sh :var dir=graph-dir :var files=graph-files
  for i in $files; do
      for j in `grep -l -r $i $dir`;do
          echo $i, `basename $j`
      done
  done
#+end_src

#+results: graph-associations
| other | eric |
| other | seb  |
| dan   | eric |
| eric  | seb  |
| seb   | dan  |

graphing with dot
#+source: to-dot
#+begin_src sh :var associations=graph-associations :results scalar
  echo "$associations"|awk '{print $1, "->", $2}'
#+end_src

#+results: to-dot
: other -> eric
: other -> seb
: dan -> eric
: eric -> seb
: seb -> dan

#+begin_src dot :var data=to-dot :file files.png
  digraph G{
    $data
  }
#+end_src

#+results:
[[file:files.png]]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:

> Hi Dan,
>
> Dan Davison wrote:
>> Cool post. I hope someone has some good ideas in this thread. Some quick
>> responses / questions below.
>>
>>> Note, in the latter code block, that I did not even tried to really chain
>>> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>>>
>>> *I certainly miss a smarter way* to achieve the above.
>>
>> I think a relevant point here is that Org doesn't yet have the ability to
>> pass data on standard input to a code block. I.e. a :stdin header arg. I
>> don't think it's that hard, someone would just need to rework
>> `org-babel-eval' so that it puts the code into a temporary file, freeing up
>> stdin to be used for data
>
> I think you exactly spotted the problem.
>
>> (and therefore we would no longer be able to use
>> shell-command-on-region but some other command (call-process I think?).)
>
> I just don't understand this last sentence, by lack of knowledge on Babel's
> internals.
>
>> Then you'd be able to do something like
>>
>> #+srcname: search-links-and-generate-dot-arrow
>> #+header: :stdin search-files-pointing-to-this-file
>> #+begin_src sh :results output :var f="charge_dim"
>> while read f; do
>>     echo "    $(basename $i) -> $f";
>> done
>> #+end_src
>>
>> I'll be interested to see the solution to all this.
>
> I've tried to rewrite the example in a much cleaner way, showing what it
> should like in the best of the worlds, with a working code at hand...
>
> BTW, I think the following could be of use, with maybe slight modifications,
> even for projects like Worg: identifying all relationships between files,
> showing files that aren't referenced, etc.
>
> #+TITLE:     Graph file dependencies
> #+DATE:      2011-02-06
> #+BABEL:    :dir ~/src/Worg
>
> * Context
>
> We want to demonstrate how to document a script in a very neat way (IMHO),
> that is:
>
> - By defining and explaining multiple small code blocks, using them later in a
>   tangle file for constructing the "full code".
>
> - By showing the effect of every small code block, that is what it returns
>   when applied on test input data.
>
> The latter is the problem, as the code has to be able to take a results set as
> if it would come from =stdin=.
>
> * Code
>
> For the sake of clarity, a real-life example that graph dependencies between
> files (based on their /basename/).
>
> ** List all files
>
> Simple file command, ignoring =.svn= directories.
>
> #+srcname: file-tree
> #+begin_src sh :results output
> find . -not \( -name .svn -prune \) -type f -print | head -n 5
> #+end_src
>
> #+results: file-tree
> #+begin_example
> ./digraph.dot
> ./full-code.sh
> ./graph-circo.pdf
> ./graph-dot.pdf
> ./graph-fdp.pdf
> #+end_example
>
> ** Search recursively for anything about a file
>
> Grep-search through files, ignoring =.svn= directories.
>
> #+srcname: search-files-pointing-to-this-file
> #+begin_src sh :results output :var toname="charge_dim"
> find . -not \( -name .svn -prune \) -type f -print0 |\
> xargs -0 grep -i --files-with-matches "$toname"
> #+end_src
>
> #+results: search-files-pointing-to-this-file
> : ./graph-file-dependencies.txt
>
> ** Convert to DOT
>
> In real life, the following block of code must read its input from =stdin=.
>
> For /in situ execution/, I should be able to say that =stdin= is equal to any
> results set (here: =search-files-pointing-to-this-file=).
>
> #+srcname: make-dot-arrow-for-files-pointing-to-this-file
> #+begin_src sh :results output :var toname="charge_dim"
> while read -r fromname
> do
>     echo "    \"${fromname##*/}\" -> \"$toname\""
> done
> #+end_src
>
> #+results: make-dot-arrow-for-files-pointing-to-this-file
>
> ** Full code: generate the DOT file
>
> #+begin_src sh :results output :file digraph.dot :noweb yes :tangle full-code.sh
> echo 'digraph G {'
> echo '    node [shape=diamond,style=filled,color=lightgrey]; ".cvsignore";'
> echo '    node [shape=box,color=yellow];'
> echo ''
>
> <<file-tree>> |\
> while read -r fname
> do
>     toname="${fname##*/}" # basename of fname
>     echo "# Files pointing to \"$toname\"..."
>     <<search-files-pointing-to-this-file>> |\
>     <<make-dot-arrow-for-files-pointing-to-this-file>>
>     echo ""
> done
>
> echo '}'
> #+end_src
>
> #+results:
> [[file:digraph.dot]]
>
> ** Draw multiple graphs
>
> #+srcname: draw-graphs
> #+begin_src sh :dir ~/Projects :var infile="digraph.dot" :var outfileprefix="digraph"
> for cmd in dot neato twopi circo fdp sfdp;
> do
>     echo $cmd...
>     time $cmd -Gcharset=latin1 -Tpdf -o$outfileprefix-$cmd.pdf $infile
>     echo ""
> done
> #+end_src
>
> Best regards,
>   Seb

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-20  8:57     ` Eric Schulte
@ 2011-02-25 14:27       ` Sébastien Vauban
  2011-02-25 22:44         ` Nick Dokos
  2011-02-26  0:24         ` Eric Schulte
  2011-02-28 15:16       ` [babel] How to kill two birds with one stone? Sébastien Vauban
  1 sibling, 2 replies; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-25 14:27 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Eric,

First, thanks for answering this open thread!

"Eric Schulte" wrote:
> I haven't followed this discussion very closely, but I'm not sure why it
> would be necessary to pass data through STDIN rather than through a variable
> or an external file.

I'll comment on the full problem (or solution) later on. But I'd like to share
a first problem with you.

> I took a shot at the dot graph example you proposed, the following works for
> me over a simple example directory.

I've redone (almost) the same file structure as yours, so that our results
must be similar, if not equal.

> directory to search
> #+results: graph-dir
> : graph-dir
>
> list all files in dir
> #+source: graph-files
> #+begin_src sh :results vector :var dir=graph-dir
>   find $dir -type f -exec basename {} \;
> #+end_src
>
> #+results: graph-files
> | other |
> | dan   |
> | eric  |
> | seb   |
>
> association of files with mentions
> #+source: graph-associations
> #+begin_src sh :var dir=graph-dir :var files=graph-files
>   for i in $files; do
>       for j in `grep -l -r $i $dir`;do
>           echo $i, `basename $j`
>       done
>   done
> #+end_src
>
> #+results: graph-associations
> | other | eric |
> | other | seb  |
> | dan   | eric |
> | eric  | seb  |
> | seb   | dan  |

My adapted version of your first paragraphs:

* Directory to search

#+results: graph-dir
: graph-dir

* List all files in dir (version of Eric)

#+source: graph-files
#+begin_src sh :results vector :var dir=graph-dir
  find $dir -type f -exec basename {} \;
#+end_src

#+results: graph-files
| dan   |        |
| eric  |        |
| other |        |
| seb   | vauban |

* List all files in dir (version of Seb)

My code was a bit more complex... because I need to be able to correctly take
care of filenames containing spaces inside them (I'm on Windows, I never do
such a thing, but there are well spaces on the files I wanna graph).

#+srcname: graph-files-seb
#+begin_src sh :results vector :var dir=graph-dir
  find $dir -type f -print |\
  while read -r name
  do
      echo "\"${name##*/}\""
  done
#+end_src

#+results: graph-files-seb
| dan   |         |
| eric  |         |
| other |         |
| "seb  | vauban" |

Spaces are used as colon delimiters: they win over my double quotes.

* List all files in dir (version of Seb)

Just to show, this code prints a semi-colon after every filename.

#+srcname: graph-files-seb2
#+begin_src sh :results vector :var dir=graph-dir
  find $dir -type f -print |\
  while read -r name
  do
      echo "\"${name##*/}\";"
  done
#+end_src

#+results: graph-files-seb2
| dan   |          |
| eric  |          |
| other |          |
| "seb  | vauban"; |

In most cases, these have been eaten as well...

Is it possible to circumvent this problem, and get my filenames (even those
with spaces in them) in one column?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Re: [babel] How to kill two birds with one stone?
  2011-02-25 14:27       ` Sébastien Vauban
@ 2011-02-25 22:44         ` Nick Dokos
  2011-02-25 22:55           ` Nick Dokos
  2011-02-28 13:59           ` Sébastien Vauban
  2011-02-26  0:24         ` Eric Schulte
  1 sibling, 2 replies; 15+ messages in thread
From: Nick Dokos @ 2011-02-25 22:44 UTC (permalink / raw)
  To: =?utf-8?Q?S=C3=A9bastien_Vauban?=; +Cc: nicholas.dokos, emacs-orgmode

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:

> ... 
> * List all files in dir (version of Seb)
> 
> My code was a bit more complex... because I need to be able to correctly take
> care of filenames containing spaces inside them (I'm on Windows, I never do
> such a thing, but there are well spaces on the files I wanna graph).
> 
> #+srcname: graph-files-seb
> #+begin_src sh :results vector :var dir=graph-dir
>   find $dir -type f -print |\
>   while read -r name
>   do
>       echo "\"${name##*/}\""
>   done
> #+end_src
> 
> #+results: graph-files-seb
> | dan   |         |
> | eric  |         |
> | other |         |
> | "seb  | vauban" |
> 

I suspect that this is a losing battle: spaces in filenames are legal,
they are common on Windows systems, but they are a PITA. The main reason
is that a *lot* of tools (particularly Unix tools of a certain age)
assume that spaces in filenames will not occur and break in mysterious
and unexpected ways when presented with a directory structure that
contains such.

There are various workarounds (the most important of which, practically speaking,
is the idiom

   find ... -print0 | xargs -0 ....

which causes ``find'' to use a null byte as a separator and ``xargs'' to
search for same in order to split the list into its constituent
components - null bytes being illegal in filenames), and there is a
long, fairly exhaustive discusssion of such matters in David Wheeler's
enlightening essay:

   http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html

but none of these would help in this case, because the culprit here
turns out to be org-table-convert-region:

,----
| org-table-convert-region is an interactive Lisp function in
| `org-table.el'.
| 
| (org-table-convert-region BEG0 END0 &optional SEPARATOR)
| 
| Convert region to a table.
| The region goes from BEG0 to END0, but these borders will be moved
| slightly, to make sure a beginning of line in the first line is included.
| 
| SEPARATOR specifies the field separator in the lines.  It can have the
| following values:
| 
| '(4)     Use the comma as a field separator
| '(16)    Use a TAB as field separator
| integer  When a number, use that many spaces as field separator
| nil      When nil, the command tries to be smart and figure out the
|          separator in the following way:
|          - when each line contains a TAB, assume TAB-separated material
|          - when each line contains a comma, assume CSV material
|          - else, assume one or more SPACE characters as separator.
| 
`----

It is called with a nil separator so it uses its "smart" mode and counts
one or more whitespace characters as the separator (I wonder
what would happen with a filename that contains a comma :-)

In any case, the region has the filenames one per line, so if
org-table-convert-region could parse a newline-separated list (and if
there was a way to specify the newline separator from higher levels)
everything would be hunky dory; there might be a way to specify the
separator using dynamic scoping, but org-table-convert-region would
require some changes to take advantage of it.

Nick

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Re: [babel] How to kill two birds with one stone?
  2011-02-25 22:44         ` Nick Dokos
@ 2011-02-25 22:55           ` Nick Dokos
  2011-02-28 13:59           ` Sébastien Vauban
  1 sibling, 0 replies; 15+ messages in thread
From: Nick Dokos @ 2011-02-25 22:55 UTC (permalink / raw)
  To: nicholas.dokos; +Cc: =?utf-8?Q?S=C3=A9bastien_Vauban?=, emacs-orgmode

Nick Dokos <nicholas.dokos@hp.com> wrote:

> ...
> It is called with a nil separator so it uses its "smart" mode and counts
> one or more whitespace characters as the separator (I wonder
> what would happen with a filename that contains a comma :-)
> 
Answer: nothing much.

Please disregard the comma comment: flippant stupidity on my part...

Nick

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Re: [babel] How to kill two birds with one stone?
  2011-02-25 14:27       ` Sébastien Vauban
  2011-02-25 22:44         ` Nick Dokos
@ 2011-02-26  0:24         ` Eric Schulte
  2011-02-26  9:56           ` Closing #+results: with #+end declaration? Bastien
  1 sibling, 1 reply; 15+ messages in thread
From: Eric Schulte @ 2011-02-26  0:24 UTC (permalink / raw)
  To: Sébastien Vauban; +Cc: emacs-orgmode

> * List all files in dir (version of Seb)
>
> Just to show, this code prints a semi-colon after every filename.
>
> #+srcname: graph-files-seb2
> #+begin_src sh :results vector :var dir=graph-dir
>   find $dir -type f -print |\
>   while read -r name
>   do
>       echo "\"${name##*/}\";"
>   done
> #+end_src
>
> #+results: graph-files-seb2
> | dan   |          |
> | eric  |          |
> | other |          |
> | "seb  | vauban"; |
>
> In most cases, these have been eaten as well...
>
> Is it possible to circumvent this problem, and get my filenames (even those
> with spaces in them) in one column?
>

Hi Seb,

My first idea was to use the ":results list" header argument, to return
a simple list (rather than a table in which the last element has two
columns)

#+begin_src sh :results list
  echo "eric schulte"
  echo "dan davison"
  echo "seb vauban"
#+end_src

#+results:
- ("eric" "schulte")
- ("dan" "davison")
- ("seb" "vauban")

But that didn't work out quite as well as expected.  Perhaps list
results should be made "smarter" by concatenating list elements that
happen to be lists themselves into strings...

I then tried the following, which should work, enabling you to split the
raw results on newline in subsequent code blocks.  Note this approach
will also preserve things like ";"'s which may have been eaten by org
table import.

#+begin_src sh :results scalar
  echo "eric schulte"
  echo "dan davison"
  echo "seb vauban"
#+end_src

#+results:
: eric schulte
: dan davison
: seb vauban

Hope this helps -- Eric

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Closing #+results: with #+end declaration?
  2011-02-26  0:24         ` Eric Schulte
@ 2011-02-26  9:56           ` Bastien
  2011-02-27 20:00             ` Eric Schulte
  0 siblings, 1 reply; 15+ messages in thread
From: Bastien @ 2011-02-26  9:56 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Sébastien Vauban, emacs-orgmode

Hi Eric,

"Eric Schulte" <schulte.eric@gmail.com> writes:

> #+begin_src sh :results list
>   echo "eric schulte"
>   echo "dan davison"
>   echo "seb vauban"
> #+end_src
>
> #+results:
> - ("eric" "schulte")
> - ("dan" "davison")
> - ("seb" "vauban")

reading this, I wonder if we should consider use this instead:

#+results:
- ("eric" "schulte")
- ("dan" "davison")
- ("seb" "vauban")
#+end

or better:

#+begin_results
- ("eric" "schulte")
- ("dan" "davison")
- ("seb" "vauban")
#+end_results

Looks more consistent with the rest of the #+begin* statements.

More generally, I've sometimes wondered why we need to use 

#+begin_*
#+end_*

instead of just

#+begin_*
#+end

Unless we allow nested #+begin blocks (and AFAIK we don't), there 
is no real need for being specific about what #+end ends.

What do you think?

-- 
 Bastien

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Closing #+results: with #+end declaration?
  2011-02-26  9:56           ` Closing #+results: with #+end declaration? Bastien
@ 2011-02-27 20:00             ` Eric Schulte
  2011-02-28 13:54               ` Sébastien Vauban
  2011-03-03 11:11               ` Bastien
  0 siblings, 2 replies; 15+ messages in thread
From: Eric Schulte @ 2011-02-27 20:00 UTC (permalink / raw)
  To: Bastien; +Cc: Sébastien Vauban, emacs-orgmode

Bastien <bzg@altern.org> writes:

> Hi Eric,
>
> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> #+begin_src sh :results list
>>   echo "eric schulte"
>>   echo "dan davison"
>>   echo "seb vauban"
>> #+end_src
>>
>> #+results:
>> - ("eric" "schulte")
>> - ("dan" "davison")
>> - ("seb" "vauban")
>
> reading this, I wonder if we should consider use this instead:
>
> #+results:
> - ("eric" "schulte")
> - ("dan" "davison")
> - ("seb" "vauban")
> #+end
>
> or better:
>
> #+begin_results
> - ("eric" "schulte")
> - ("dan" "davison")
> - ("seb" "vauban")
> #+end_results
>
> Looks more consistent with the rest of the #+begin* statements.
>

This has come up before, and there are now options which allow wrapping
of results, e.g.,

#+begin_src emacs-lisp :results wrap :exports both
  (mapcar (lambda (el) (list el (+ 1 (* el el)))) (number-sequence 0 10))
#+end_src

#+results:
#+BEGIN_RESULT
|  0 |   1 |
|  1 |   2 |
|  2 |   5 |
|  3 |  10 |
|  4 |  17 |
|  5 |  26 |
|  6 |  37 |
|  7 |  50 |
|  8 |  65 |
|  9 |  82 |
| 10 | 101 |
#+END_RESULT

I've just added documentation of the "wrap" header argument to the
manual.

>
> More generally, I've sometimes wondered why we need to use
>
> #+begin_*
> #+end_*
>
> instead of just
>
> #+begin_*
> #+end
>
> Unless we allow nested #+begin blocks (and AFAIK we don't), there 
> is no real need for being specific about what #+end ends.
>
> What do you think?

I agree that (possibly aside from clarity) there is no real need for the
end block to specify its type.  However as I use helper methods
(e.g. yasnippets) for all block creation, then extra characters
represent no real typing burden.

Best -- Eric

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Closing #+results: with #+end declaration?
  2011-02-27 20:00             ` Eric Schulte
@ 2011-02-28 13:54               ` Sébastien Vauban
  2011-03-03 11:11               ` Bastien
  1 sibling, 0 replies; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-28 13:54 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Eric and Bastien,

"Eric Schulte" wrote:
>> reading this, I wonder if we should consider use this instead:
>>
>> #+results:
>> - ("eric" "schulte")
>> - ("dan" "davison")
>> - ("seb" "vauban")
>> #+end
>>
>> or better:
>>
>> #+begin_results
>> - ("eric" "schulte")
>> - ("dan" "davison")
>> - ("seb" "vauban")
>> #+end_results
>>
>> Looks more consistent with the rest of the #+begin* statements.
>
> This has come up before, and there are now options which allow wrapping of
> results, e.g.,
>
> #+begin_src emacs-lisp :results wrap :exports both
>   (mapcar (lambda (el) (list el (+ 1 (* el el)))) (number-sequence 0 2))
> #+end_src
>
> #+results:
> #+BEGIN_RESULT
> |  0 |   1 |
> |  1 |   2 |
> |  2 |   5 |
> #+END_RESULT
>
>> More generally, I've sometimes wondered why we need to use
>>
>> #+begin_*
>> #+end_*
>>
>> instead of just
>>
>> #+begin_*
>> #+end
>>
>> Unless we allow nested #+begin blocks (and AFAIK we don't), there is no
>> real need for being specific about what #+end ends.
>> What do you think?

I'm always in favor of (a bit) too much of information, rather than not
enough. Properly closing the results block (with a meaningful name) has the
following advantages IMHO:

- close to LaTeX and HTML styles -- though, a detail;

- allows inserting blocks in other blocks (of a different nature);

- maybe if numbered, and if there is a strong use case, would allow to insert
  blocks inside blocks of the same type.

Remember a discussion we had where we would even be able to specify the name
of the wrap environment to be used -- though not yet implemented.

> I agree that (possibly aside from clarity) there is no real need for the end
> block to specify its type. However as I use helper methods (e.g. yasnippets)
> for all block creation, then extra characters represent no real typing
> burden.

If I had to choose (a right I don't have ;-)), I would more opt for the
disappearance of the alone `#+results':' line, and only have `results' blocks
such as:

#+begin_results
#+end_results

That would simplify the way results are shown when wrapped, and avoid some
problems (inserting results with blanks lines, not being able to say where it
ends).

This would stop the redundancy of the results line, and allow as well an easy
conversion to nicely formatted LaTeX or HTML.

My 2 cents.

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-25 22:44         ` Nick Dokos
  2011-02-25 22:55           ` Nick Dokos
@ 2011-02-28 13:59           ` Sébastien Vauban
  1 sibling, 0 replies; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-28 13:59 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Nick,

Nick Dokos wrote:
> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> wrote:
>> My code was a bit more complex... because I need to be able to correctly
>> take care of filenames containing spaces inside them (I'm on Windows, I
>> never do such a thing, but there are well spaces on the files I wanna
>> graph).
>> 
>> #+results: graph-files-seb
>> | dan   |         |
>> | eric  |         |
>> | other |         |
>> | "seb  | vauban" |
>
> I suspect that this is a losing battle: spaces in filenames are legal, they
> are common on Windows systems, but they are a PITA. The main reason is that
> a *lot* of tools (particularly Unix tools of a certain age) assume that
> spaces in filenames will not occur and break in mysterious and unexpected
> ways when presented with a directory structure that contains such.
>
> There are various workarounds (the most important of which, practically
> speaking, is the idiom
>
>    find ... -print0 | xargs -0 ....
>
> which causes ``find'' to use a null byte as a separator and ``xargs'' to
> search for same in order to split the list into its constituent components -
> null bytes being illegal in filenames), and there is a long, fairly
> exhaustive discusssion of such matters in David Wheeler's enlightening
> essay:
>
>    http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html
>
> but none of these would help in this case, because the culprit here turns
> out to be org-table-convert-region:
>
> ,----
> | (org-table-convert-region BEG0 END0 &optional SEPARATOR)
> | 
> | Convert region to a table.
> | SEPARATOR specifies the field separator in the lines.  It can have the
> | following values:
> | 
> | '(4)     Use the comma as a field separator
> | '(16)    Use a TAB as field separator
> | integer  When a number, use that many spaces as field separator
> | nil      When nil, the command tries to be smart and figure out the
> |          separator in the following way:
> |          - when each line contains a TAB, assume TAB-separated material
> |          - when each line contains a comma, assume CSV material
> |          - else, assume one or more SPACE characters as separator.
> `----
>
> It is called with a nil separator so it uses its "smart" mode and counts one
> or more whitespace characters as the separator (I wonder what would happen
> with a filename that contains a comma :-)
>
> In any case, the region has the filenames one per line, so if
> org-table-convert-region could parse a newline-separated list (and if there
> was a way to specify the newline separator from higher levels) everything
> would be hunky dory; there might be a way to specify the separator using
> dynamic scoping, but org-table-convert-region would require some changes to
> take advantage of it.

If I follow you correctly, another approach would be to enhance
`org-table-convert-region' so that it could take `\0' as field separator?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [babel] How to kill two birds with one stone?
  2011-02-20  8:57     ` Eric Schulte
  2011-02-25 14:27       ` Sébastien Vauban
@ 2011-02-28 15:16       ` Sébastien Vauban
  1 sibling, 0 replies; 15+ messages in thread
From: Sébastien Vauban @ 2011-02-28 15:16 UTC (permalink / raw)
  To: emacs-orgmode-mXXj517/zsQ

Hi Eric,

"Eric Schulte" wrote:
> I haven't followed this discussion very closely, but I'm not sure why it
> would be necessary to pass data through STDIN rather than through a variable
> or an external file.
>
> I took a shot at the dot graph example you proposed, the following works for
> me over a simple example directory.

For sure, your version works, but it does not address the two goals -- I hope
they don't exclude each other -- I wanna try to reach.

This should clarify the subject.

#+TITLE:     graph-dir-eric-schulte
#+DATE:      2011-02-28
#+LANGUAGE:  en_US

* Directory to search

#+results: graph-dir
: graph-dir

* List all files in dir

#+source: graph-files
#+begin_src sh :results vector :var dir=graph-dir
  find $dir -type f -exec basename {} \;
#+end_src

#+results: graph-files
| dan        |
| eric       |
| other      |
| seb_vauban |

* Association of files with mentions

#+source: graph-associations
#+begin_src sh :var dir=graph-dir :var files=graph-files
  for i in $files; do
      for j in `grep -l -r $i $dir`; do
          echo $i, `basename $j`
      done
  done
#+end_src

#+results: graph-associations
| dan        | eric       |
| eric       | seb_vauban |
| other      | eric       |
| other      | seb_vauban |
| seb_vauban | dan        |

* Ultimate goal

The first goal of this document was to write small snippets of code, with a
clear and concise specification, and test its results when applied on some
input data. This goal is clearly met.

Though, the following goal, the ultimate one, is to be able to output an
independent shell script -- out of this stuff --, so that the program can be
run independently by anybody, just from a simple shell.

How could I write such a "combined" shell script?

Something in the spirit of:

#+source: graph-associations-sh-script
#+begin_src sh :var dir=graph-dir :noweb yes
  cd $dir
  for i in `<<graph-files>>`; do
      for j in `grep -l -r $i $dir`; do
          echo $i, `basename $j`
      done
  done
#+end_src

#+results: graph-associations-sh-script

But, of course (because of the =dir= var?), the above does not work, at least
for my first goal: seeing "in situ" (part of) the results it returns when it
is run.

Maybe it's possible to reach both goals, maybe it's not -- I've no definite
clue about this.

- Maybe we need =stdin= to be properly "handled" for this, nothing more?

- Maybe we need to add some extra hard constraints on how to write the little
  program parts, such as "don't use the Babel :var mechanism"?

- Etc...

* Executive summary

To sum up what I'm trying to explain, I'd like the ability to write *and run*
small snippets of code, such as:

#+source: block-1
#+begin_src sh
... do this...
#+end_src

#+source: block-2
#+begin_src sh
... do that...
#+end_src

*and* to be able to produce the shell script that would run them all from a
shell (no need for Emacs), something like:

#+source: full-code
#+begin_src sh :tangle yes
<<block-1>>
<<block-2>>
#+end_src

or

#+source: full-code
#+begin_src sh :tangle yes
<<block-1>> | <<block-2>>
#+end_src

or

#+source: full-code
#+begin_src sh :tangle yes
for i in `<<block-1>>`; do
    <<block-2>>
done
#+end_src

or ... (depending on what the code does) ...

Which conditions would allow one to make both dreams true at the same time?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Re: Closing #+results: with #+end declaration?
  2011-02-27 20:00             ` Eric Schulte
  2011-02-28 13:54               ` Sébastien Vauban
@ 2011-03-03 11:11               ` Bastien
  1 sibling, 0 replies; 15+ messages in thread
From: Bastien @ 2011-03-03 11:11 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Sébastien Vauban, emacs-orgmode

Hi Eric,

"Eric Schulte" <schulte.eric@gmail.com> writes:

> #+begin_src emacs-lisp :results wrap :exports both
>   (mapcar (lambda (el) (list el (+ 1 (* el el)))) (number-sequence 0 10))
> #+end_src
>
> #+results:
> #+BEGIN_RESULT
> |  0 |   1 |
> |  1 |   2 |
> |  2 |   5 |
> |  3 |  10 |
> |  4 |  17 |
> |  5 |  26 |
> |  6 |  37 |
> |  7 |  50 |
> |  8 |  65 |
> |  9 |  82 |
> | 10 | 101 |
> #+END_RESULT
>
> I've just added documentation of the "wrap" header argument to the
> manual.

Thanks -- IMHO wrapping with begin/end_result could be the default, 
or even the only option avaiable (but it would break current code.)

> I agree that (possibly aside from clarity) there is no real need for the
> end block to specify its type.  However as I use helper methods
> (e.g. yasnippets) for all block creation, then extra characters
> represent no real typing burden.

Let's think about this later, no real emergency here.

Thanks for your feedback,

-- 
 Bastien

-- 
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2011-03-03 11:12 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-02-04 16:00 [babel] How to kill two birds with one stone? Sébastien Vauban
2011-02-04 17:43 ` Dan Davison
2011-02-04 22:23   ` Sébastien Vauban
2011-02-06 16:51   ` Sébastien Vauban
2011-02-20  8:57     ` Eric Schulte
2011-02-25 14:27       ` Sébastien Vauban
2011-02-25 22:44         ` Nick Dokos
2011-02-25 22:55           ` Nick Dokos
2011-02-28 13:59           ` Sébastien Vauban
2011-02-26  0:24         ` Eric Schulte
2011-02-26  9:56           ` Closing #+results: with #+end declaration? Bastien
2011-02-27 20:00             ` Eric Schulte
2011-02-28 13:54               ` Sébastien Vauban
2011-03-03 11:11               ` Bastien
2011-02-28 15:16       ` [babel] How to kill two birds with one stone? Sébastien Vauban

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).