[babel] noweb does not work (as expected)

[babel] noweb does not work (as expected)

[Torsten Wagner]

Hi everyone,

I tried to use the noweb-syntax to tangle all the pieces of source code blocks 
together as proposed by Tom.

However, for some reasons it does not work. 
I notice there are two different syntax for noweb. One propsed on worg and the 
other in examples here in the mailing list. Which one will be correct ? or are 
both correct but with different purpose ?

I used the latest git-version and even tried the branch noweb-evaluate

What I used was a source code block at the end of my org-file like:
All other blocks are defined above.

#+srcname: makefile()
#+begin_src python :tangle simulate_ofdm :exports none
  <<block1>>
  <<block2>>
  <<block3>>
  <<block4>>
  <<block5>>
#+end_src
#+resname: makefile

However
the tangled file results only in 

#!/usr/bin/env python
# generated by org-babel-tangle
# [[file:~/test_noweb.org::*Result][makefile]]

# makefile ends here

An execution of block shows nothing on the *Shell output* buffer
It seems somehow I missunderstand which part of a block will replace a 
<<noweb>>  variable. I thouht it will be replaced by the complete source block 
content during tangle. 

Any ideas ?

Thanks,

Torsten

Re: [babel] noweb does not work (as expected)

[Thomas S. Dye]

[-- Attachment #1.1: Type: text/plain, Size: 3312 bytes --]

On Nov 8, 2009, at 5:47 PM, Torsten Wagner wrote:

> Hi everyone,
>
> I tried to use the noweb-syntax to tangle all the pieces of source  
> code blocks
> together as proposed by Tom.
>
> However, for some reasons it does not work.
> I notice there are two different syntax for noweb. One propsed on  
> worg and the
> other in examples here in the mailing list. Which one will be  
> correct ? or are
> both correct but with different purpose ?
>
> I used the latest git-version and even tried the branch noweb-evaluate
>
> What I used was a source code block at the end of my org-file like:
> All other blocks are defined above.
>
> #+srcname: makefile()
> #+begin_src python :tangle simulate_ofdm :exports none
>  <<block1>>
>  <<block2>>
>  <<block3>>
>  <<block4>>
>  <<block5>>
> #+end_src
> #+resname: makefile
>
> However
> the tangled file results only in
>
> #!/usr/bin/env python
> # generated by org-babel-tangle
> # [[file:~/test_noweb.org::*Result][makefile]]
>
> # makefile ends here
>
> An execution of block shows nothing on the *Shell output* buffer
> It seems somehow I missunderstand which part of a block will replace a
> <<noweb>>  variable. I thouht it will be replaced by the complete  
> source block
> content during tangle.
>
> Any ideas ?
>
> Thanks,
>
> Torsten
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Hi Torsten,

I've been working successfully with your makefile idea.  Here is one  
that works for me:

--------------------------------------

#+begin_src R :noweb :session 135cultural
    <<r-load-libraries>>
    <<r-connect>>
    # run queries
    <<r-135-artifacts-wt-cultural>>
    <<r-135-artifacts-ct-cultural>>
    <<r-135-inverts-cultural>>
    <<r-135-verts-cultural>>
    # reshape query results and run PCA
    <<r-135-reshape>>
    <<r-135-merge>>
    <<r-135-prcomp>>
    # make biplots
    <<r-135-plot-pca-1-2-cultural>>
    <<r-135-plot-pca-2-3-cultural>>
    # segment plot
    <<r-135-stars-cultural>>
    # make tables for LaTeX
   # <<r-135-table-pca>>
    <<r-disconnect>>
#+end_src

-------------------------------------------------

A typical source block looks like this:

-------------------------------------------------

#+srcname: r-load-libraries
#+begin_src R
   library(RMySQL)
   library(reshape)
   library(xtable)
#+end_src

-------------------------------------------------

Note that I don't :tangle this, but use the :noweb feature, instead.   
Also, my "makefile" runs all its :noweb source blocks in a :session,  
where the environment set up by each source block can persist and be  
referenced by other source blocks.  This particular makefile has run  
successfully dozens of times, as I fix little things and add new ones.

I don't know if this is the best way to accomplish this, or if it is  
what you are trying to achieve at the moment.  But, it looks to me as  
if you might want to :noweb instead of :tangle and perhaps establish  
a :session in case your source blocks need to communicate with one  
another.

HTH,
Tom

Thomas S. Dye, Ph.D.
T. S. Dye & Colleagues, Archaeologists, Inc.
Phone: (808) 529-0866 Fax: (808) 529-0884
http://www.tsdye.com



[-- Attachment #1.2: Type: text/html, Size: 6916 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] noweb does not work (as expected)

[Torsten Wagner]

Hi Tom,

If I try to use the noweb way, I always got error messages which tells me that 
org-babel can not read the result correctly

> 
> #+srcname: r-load-libraries
> #+begin_src R
>    library(RMySQL)
>    library(reshape)
>    library(xtable)
> #+end_src


As I see you do not use any special header here which means results is set to 
value, right? However, there is no special return line to org-babel either. In 
my understanding and according to my error messages  library(xtable) would be 
evaluated as the result of this block...

However this results in error messages at least for python.

Which org-babel version due you use (which branch)?

Maybe you or someone else can bring some more light into this :results topic.
I somehow miss the option :results none to avoid any results which should be 
useful if the source code block is just a piece of a bigger arrangement. For 
literate programming and RR it might be essential to spread blocks over the 
file which will be tangled together (by tangle or a noweb block). However not 
all need provide a result but just need to run in the same session. Maybe just 
the last block of such an arrangement will come up with a result suitable for 
org-babel.
     
Actually I prefer the idea of :noweb instead of tangle, which seems to me more 
flexible and faster.

Just need to get it running

Thanks 

Torsten

Re: [babel] noweb does not work (as expected)

[Thomas S. Dye]

On Nov 8, 2009, at 9:38 PM, Torsten Wagner wrote:

> Hi Tom,
>
> If I try to use the noweb way, I always got error messages which  
> tells me that
> org-babel can not read the result correctly
>
>>
>> #+srcname: r-load-libraries
>> #+begin_src R
>>   library(RMySQL)
>>   library(reshape)
>>   library(xtable)
>> #+end_src
>
>
> As I see you do not use any special header here which means results  
> is set to
> value, right? However, there is no special return line to org-babel  
> either. In
> my understanding and according to my error messages  library(xtable)  
> would be
> evaluated as the result of this block...
>
> However this results in error messages at least for python.
>
> Which org-babel version due you use (which branch)?
>
> Maybe you or someone else can bring some more light into  
> this :results topic.
> I somehow miss the option :results none to avoid any results which  
> should be
> useful if the source code block is just a piece of a bigger  
> arrangement. For
> literate programming and RR it might be essential to spread blocks  
> over the
> file which will be tangled together (by tangle or a noweb block).  
> However not
> all need provide a result but just need to run in the same session.  
> Maybe just
> the last block of such an arrangement will come up with a result  
> suitable for
> org-babel.
>
> Actually I prefer the idea of :noweb instead of tangle, which seems  
> to me more
> flexible and faster.
>
> Just need to get it running
>
> Thanks
>
> Torsten

Hi Torsten,

It's not clear to me what outcome you desire.  Tangling should result  
in a source file that can serve as input to a compiler or  
interpreter.  The combination of :noweb and :session lets you write  
literate programs that are sent directly to an interpreter, which  
presumably creates some useful output along the way (my example makes  
some graphs after a lot of data reshaping), but might be used just to  
set up an environment in the interpreter, which then can be  
manipulated directly in the session buffer.

I presumed that your reference to makefile meant that you were heading  
for the :noweb, :session route.  I find this useful because I can  
query the state of variables in the session and put the results in a  
LaTeX source block. Of course, the same thing would be possible to  
insert the results into the org file.

I'm using the version of org-babel that comes with org-mode version  
6.32trans.  I think I last updated about a week ago.  I'm not  
competent to work at the bleeding edge.  Besides that, I find the org- 
babel distributed with org-mode to be mature for such a young piece of  
software, and it does pretty much everything I ask it to do.

HTH,
Tom

Re: [babel] noweb does not work (as expected)

[Torsten Wagner]

> Hi Torsten,
Hi Tom,
 
> It's not clear to me what outcome you desire.  Tangling should result  
> in a source file that can serve as input to a compiler or  
> interpreter.  The combination of :noweb and :session lets you write  
> literate programs that are sent directly to an interpreter, which  
> presumably creates some useful output along the way (my example makes  
> some graphs after a lot of data reshaping), but might be used just to  
> set up an environment in the interpreter, which then can be  
> manipulated directly in the session buffer.

Actually I'm looking for a way to execute several of my source code buffers in 
a comfortable and semiautomatic way to do exactly what you described. Joggling 
around with my measurement data... bend badly statistics until it fit to my 
world ;) and finally create a plot, table or even a single number. The blocks 
should be aware of each other (same session) and I like to have a single local 
place to control the order in which the source code blocks are executed and be 
able to tweak around with some variables or add quickly.
Thus, this sound really like :noweb and :session will do this.

However, sometimes I would prefer to see the code all tangled together in one 
chunk. This might be esp. interesting for several purposes.
a) Give the pure code to someone else,
b) Make a somehow stand-alone-version without org-babel e.g. to execute it 
over and over again on another org-babel free machine,
c) Debugging and bug searching might be easier since you can clearly see the 
interfaces between the source code blocks without to much org-mode in between.

I guess at the moment I could create two blocks ... one :noweb, :nosession for 
the interactive work on the project and one with :tangle which put all this 
together into a single file. I never tried :tangle and :noweb together. 
Actually, they should not conflict so fare. However, at the worg-page all 
options are given as mutual exclusive. 

I will give at a trial... as soon as I make my file working under both methods.

Best regards,

Torsten

Re: [babel] noweb does not work (as expected)

[Dan Davison]

Hi Torsten,

Torsten Wagner <torsten.wagner@gmail.com> writes:

> Hi everyone,
>
> I tried to use the noweb-syntax to tangle all the pieces of source code blocks 
> together as proposed by Tom.
>
> However, for some reasons it does not work. 
> I notice there are two different syntax for noweb. One propsed on worg and the 
> other in examples here in the mailing list. Which one will be correct ? or are 
> both correct but with different purpose ?

<<block-1>> inserts the body of block 'block-1'. <<block-1()> inserts
the *result* of evaluating block-1, and <<block-1(x=3.3)>> inserts the
result of evaluating it with the specified argument value. I've just
added documentation of this to Worg.

>
> I used the latest git-version and even tried the branch noweb-evaluate
>
> What I used was a source code block at the end of my org-file like:
> All other blocks are defined above.

Can you provide the exact contents of the file you're having trouble
with? If I have

----------------------------------------------------------------
* test tangle
#+srcname: block1
#+begin_src python 
  x = 1
#+end_src

#+srcname: makefile()
#+begin_src python :tangle simulate_ofdm :exports none
  This is makefile
  <<block1>>
#+end_src
----------------------------------------------------------------

then for me org-babel-tangle produces

----------------------------------------------------------------
#!/usr/bin/env python
# generated by org-babel-tangle
# [[file:/tmp/zz.org::*test%20tangle][makefile]]
This is makefile
x = 1
# makefile ends here
----------------------------------------------------------------

Dan

>
> #+srcname: makefile()
> #+begin_src python :tangle simulate_ofdm :exports none
>   <<block1>>
>   <<block2>>
>   <<block3>>
>   <<block4>>
>   <<block5>>
> #+end_src
> #+resname: makefile
>
> However
> the tangled file results only in 
>
> #!/usr/bin/env python
> # generated by org-babel-tangle
> # [[file:~/test_noweb.org::*Result][makefile]]
>
> # makefile ends here
>
> An execution of block shows nothing on the *Shell output* buffer
> It seems somehow I missunderstand which part of a block will replace a 
> <<noweb>>  variable. I thouht it will be replaced by the complete source block 
> content during tangle. 
>
> Any ideas ?
>
> Thanks,
>
> Torsten
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] noweb does not work (as expected)

[Torsten Wagner]

Hi Dan, 

thanks for trying helping me out
actually your example works for me after switching back to git-babel master 
branch.
However, my file does still not work and I can not see a difference between your 
and mine file.

I will try to minimize it now to the point where it either works or still 
fails and post it here. 
Could you please tell me which of the branches in the babel-git includes the 
best working version for noweb usage.
I got a bit confused with all this branches.
Maybe I do something wrong during the installation from git. I actually only 
run standard make; make install; 
and I set a symlink of the contrib folder into my .emacs.d from where emacs 
will find the babel files. Actually I'm wondering why the contribs are not 
copied by make into the elisp-path.

Furthermore, I noticed that babel-git uses a complete org-mode branch for the 
development of org-babel. Is this to keep org-mode freezed and to control 
merging with the main org-mode developments? 
I once read about git submodules [1]. Maybe this is interesting for org-babel.
 
Thanks again for help

Torsten

[1] http://book.git-scm.com/5_submodules.html

Re: [babel] noweb does not work (as expected)

[Dan Davison]

Hi Torsten,

Torsten Wagner <torsten.wagner@gmail.com> writes:

<...>
> Could you please tell me which of the branches in the babel-git includes the 
> best working version for noweb usage.

Important: The only version of org-babel that's intended for users is
the version in current org-mode. I.e. the current master branch of
Carsten's org-mode repository. The other repository (the babel repo) is
for development only. Any stable improvements in there are rapidly
merged into Carsten's repo.

> I got a bit confused with all this branches.
> Maybe I do something wrong during the installation from git. I actually only 
> run standard make; make install; 

Personally I don't bother with make or make install when switching
between git branches. I just load straight from the .el files. But like
I said, unless you're making changes to the code, just use the main
org-mode git branch in whatever way you normally would.

> and I set a symlink of the contrib folder into my .emacs.d from where emacs 
> will find the babel files. Actually I'm wondering why the contribs are not 
> copied by make into the elisp-path.
>
> Furthermore, I noticed that babel-git uses a complete org-mode branch for the 
> development of org-babel. Is this to keep org-mode freezed and to control 
> merging with the main org-mode developments? 
> I once read about git submodules [1]. Maybe this is interesting for org-babel.

Disclaimer: I only started using git submodules last night.

I think the main point is that the current set up means that org-mode
and org-babel share a common history of commits. My current idea of git
submodules is that I would include something as a submodule when it is a
module that gets used by multiple different projects, and so doesn't
'belong' to any project in particular. But org-babel belongs to org-mode
and has no independent existence and I think it makes sense for it to be
developed in an org-mode repo.

But away from org-mode, git submodules look really cool -- they seem to
be exactly what I was looking for to organize projects which share some
code in common.

Dan

>  
> Thanks again for help
>
> Torsten
>
> [1] http://book.git-scm.com/5_submodules.html
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] noweb does not work (as expected)

[Torsten Wagner]

Hi Dan
> Important: The only version of org-babel that's intended for users is
> the version in current org-mode. I.e. the current master branch of
> Carsten's org-mode repository. The other repository (the babel repo) is
> for development only. Any stable improvements in there are rapidly
> merged into Carsten's repo.

O.k. thanks for the clarification. I started using babel-git since in an older 
thread someone refereed that the functionality is no in a branch of babel-git. 
I did not know that you merge quickly with Carstens org-mode git.

Depending on the speed of merging, you might like to use tags in babel-git 
signaling at which commit you merged org-babel back to org-mode. Then people 
can quickly run a git diff to see if the particular change they are looking for 
is already in merged to org-mode or not. 

> I think the main point is that the current set up means that org-mode
> and org-babel share a common history of commits. My current idea of git
> submodules is that I would include something as a submodule when it is a
> module that gets used by multiple different projects...

As far as I understand git submodules (but I never get in real use to them 
yet) they can be considered as a part of a project which keeps it own history. 
If that is right it might be perfect for org-mode and org-babel. You can 
develop on top of the org-mode git and does not cluttering the history of org-
mode git.
However, as far as I read there are some potential flaws which, if used wrong, 
can create some serious problems. 

Many thanks for helping me so fare

Torsten

Re: [babel] noweb does not work (as expected)

[Eric Schulte]

Torsten Wagner <torsten.wagner@gmail.com> writes:

>> I think the main point is that the current set up means that org-mode
>> and org-babel share a common history of commits. My current idea of git
>> submodules is that I would include something as a submodule when it is a
>> module that gets used by multiple different projects...
>
> As far as I understand git submodules (but I never get in real use to
> them yet) they can be considered as a part of a project which keeps it
> own history.  If that is right it might be perfect for org-mode and
> org-babel. You can develop on top of the org-mode git and does not
> cluttering the history of org- mode git.  However, as far as I read
> there are some potential flaws which, if used wrong, can create some
> serious problems.
>

Hi Torsten,

Having used git submodules in a couple of projects I would shy away from
the added complexity which submodules introduce.  In my experience the
implicit requirement of submodules for users to execute 'git submodule
init' and 'git submodule update' commands whenever they update their
source code can lead to a lot of confusion.

Best -- Eric

Re: [babel] (solved) noweb does not work (as expected)

[Torsten Wagner]

Hi,

finally I found the problem. As Dan mentioned there is a difference for 
<<block1>>
<<block1()>>
<<block1(a=1)>>

For some reason I was sticking with block1() and override it over and over 
again. No it seems to work go into the right direction. Python still seems not 
to like the code if its created by :noweb. However :tangle code seems to run 
fine.

I will try to sort this out now.

By the way, :noweb and :tangle seems to work nicely together which gives the 
freedom to go one or the other way to execute the source code blocks

During fiddling around the last day, I found some quirks and "nice to have" 
requests. Should I simply post them on the list ?

Thanks again for helping me

Torsten