[babel]: Some feedback after the first week usage

[babel]: Some feedback after the first week usage

[Torsten Wagner]

Hi everyone,
I would like to report the usage of org-babel after a 1 week usage in 
"serious" project work. 

First of all org-babel is a masterpiece. Many thanks to the developers. I 
never saw a project which was such useful in this very early stage (how old is 
org-babel?). For me it tends out that org-babel is for my scientific-
programming tasks what org-mode is for organisation of the administrative part 
of those projects... plain, simple and just helpful at the right moment.

I like to make some remarks and depict some small glitches which I found over 
the last week. As for the glitches, for some there might be a solution 
already, maybe some arouses because I'm still using something wrong. Thus, I 
thought it might be good to share it here.  

First another statement which could be help to show how useful is org-babel.

Do you spend  sometimes days or weeks in front of the PC to fiddle around with 
some scientific simulation or some other programming task?
If your colleagues and esp. your boss are not deep inside what it means to 
perform this kind of work (e.g., because they might be from another, 
programming free, major), they pretty fast become nervous about why you are 
going to loose so much time in front of the keyboard. 

With org-babel and a mixture of literate programming and RR, you will have a 
new way to handle those requests.   
Actually all you need is 
C-c C-e d
and sent it to the printer....
The result will be a professional looking comprehensive status report. You can 
bring it to the meetings, file them as lab-book, use them in many ways to show 
what are you doing right at the moment. It includes everything from theory, 
explanations, calculations up to graphs, tables  and even the source code.  
And all you need is the above command which takes 1.2 seconds... :D

Compare this with the "classical" approach (which is teached unfortunately in 
most universities now). Bring your project to a point where all individual 
parts work, create some figures, data and results of your project, create a new 
MS Power-point presentation, copy and place figures and results inside, add 
text and tables, make everything looks more or less uniform, print it.... this 
might consume 1-2 days or ... 172800 seconds !!! and you will still miss many 
information (e.g. which fitting function did you use for the graph ?).


As I said already, I'm really amazed that org-babel works all out so smoothly. 
However, I noticed the following points which I like to publish here for 
discussion.

1. Is there another possible way for creating the final statement for the 
source block result ? 
At the moment I have to write 

#+srcname: plot()
#+begin_src python :results file 
  plot(x,y)
  filename = "figure.pdf"
  savefig(filename)
  filename
#+end_src

#+resname: [[figure.pdf]]

But from the python point of view the last command is not necessary. If I 
tangle the source blocks or copy and paste them I might even become some 
problems (e.g., if the content is a very large string it would be displayed on 
the ipython console). In addition this commands will be included in the export 
but have not really a function beside of the wrapping for org-babel. I would 
like to use something like this: 

#+srcname: sin_sum()
#+begin_src python :results file 
  plot(x,y)
  filename = "figure.pdf"
  savefig(filename)
#+end_src filename

#+resname: [[figure.pdf]]

By this way I could clearly distinguish between what is pure python and what 
is needed to wrap it in org-mode

2. Is it possible (by a keyboard-shortcut) to execute all blocks of a session 
starting from top to down? Sometimes I make small changes here and there and I 
just like to say o.k. tangle all this together and execute it, refresh and 
show me all new results. This might be even combined by a org-mode variable 
which allow to define which results need to be refreshed before saving the org-
mode file (something like #+EXECBYSAVE: name_of_session).
I press C-x C-s rather frequently without much thinking of it. If I want make 
sure that e.g., all generated output files (e.g. figures) are in sync with the 
code it would be nice to say C-x C-s  and all external figures will be 
recreated if necessary.
 
3. If somethings went bad during execution of a block, org-babel will report 
"no output"... is there any way to catch errors and place a warning or/and 
even jump to the code block in question?

4. Is there a way to see all source code blocks of a single session tangled 
together in a single buffer and changes in this buffer will be placed back after 
leaving into the individual blocks again 
Similar like C-c ' just for a complete session with minimal comments to split 
it later:
e.g.,

#[[blockname1]]
a= 1
b= 2
#[[blockname2]]
result = a +b

would allow me to work on the code in a single file in python mode and later it 
will be displayed again in the individual source blocks in my org-file.

I guess that was all I noticed during using org-bable over the last days.
It would be nice to hear how others think about the above points. Did you find 
a better solution already ? Any tricks or any functions I might missing?
Again many thanks for this really nice org-mode contribution.


Best regards

Torsten
 

Re: [babel]: Some feedback after the first week usage

[Thomas S. Dye]

[-- Attachment #1.1: Type: text/plain, Size: 1211 bytes --]

On Nov 4, 2009, at 10:27 PM, Torsten Wagner wrote:
> ...

> 2. Is it possible (by a keyboard-shortcut) to execute all blocks of  
> a session
> starting from top to down? Sometimes I make small changes here and  
> there and I
> just like to say o.k. tangle all this together and execute it,  
> refresh and
> show me all new results. This might be even combined by a org-mode  
> variable
> which allow to define which results need to be refreshed before  
> saving the org-
> mode file (something like #+EXECBYSAVE: name_of_session).
> I press C-x C-s rather frequently without much thinking of it. If I  
> want make
> sure that e.g., all generated output files (e.g. figures) are in  
> sync with the
> code it would be nice to say C-x C-s  and all external figures will be
> recreated if necessary.

> ...

Aloha Torsten,

Would something like run-code-blocks (below) do what you want?  I put  
a source block like this at the top of my org-babel file, where I can  
find it quickly.

#+srcrname: run-code-blocks
#+begin_src python :noweb
<<block-1>>
<<block-2>>
#+end_src

#+srcname: block-1
#+begin_src python
a = 1
a
#+end_src

#+srcname: block-2
#+begin_src python
b = 1
b
#+end_src

All the best,
Tom 

[-- Attachment #1.2: Type: text/html, Size: 1998 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel]: Some feedback after the first week usage

[Dan Davison]

Torsten Wagner <torsten.wagner@gmail.com> writes:

<...>
> Do you spend  sometimes days or weeks in front of the PC to fiddle around with 
> some scientific simulation or some other programming task?

Yes.

> If your colleagues and esp. your boss are not deep inside what it means to 
> perform this kind of work (e.g., because they might be from another, 
> programming free, major), they pretty fast become nervous about why you are 
> going to loose so much time in front of the keyboard. 

Yes.

>
> With org-babel and a mixture of literate programming and RR, you will have a 
> new way to handle those requests.   
> Actually all you need is 
> C-c C-e d
> and sent it to the printer....
> The result will be a professional looking comprehensive status report. You can 
> bring it to the meetings, file them as lab-book, use them in many ways to show 
> what are you doing right at the moment. It includes everything from theory, 
> explanations, calculations up to graphs, tables  and even the source code.  
> And all you need is the above command which takes 1.2 seconds... :D
>
> Compare this with the "classical" approach (which is teached unfortunately in 
> most universities now). Bring your project to a point where all individual 
> parts work, create some figures, data and results of your project, create a new 
> MS Power-point presentation, copy and place figures and results inside, add 
> text and tables, make everything looks more or less uniform, print it.... this 
> might consume 1-2 days or ... 172800 seconds !!! and you will still miss many 
> information (e.g. which fitting function did you use for the graph ?).

Nice description Torsten. That is exactly the situation that (a) I am
always in and (b) originally motivated the project for me.

>
>
> As I said already, I'm really amazed that org-babel works all out so smoothly. 
> However, I noticed the following points which I like to publish here for 
> discussion.
>
> 1. Is there another possible way for creating the final statement for the 
> source block result ? 
> At the moment I have to write 
>
> #+srcname: plot()
> #+begin_src python :results file 
>   plot(x,y)
>   filename = "figure.pdf"
>   savefig(filename)
>   filename
> #+end_src
>
> #+resname: [[figure.pdf]]
>
> But from the python point of view the last command is not necessary. If I 
> tangle the source blocks or copy and paste them I might even become some 
> problems (e.g., if the content is a very large string it would be displayed on 
> the ipython console). In addition this commands will be included in the export 
> but have not really a function beside of the wrapping for org-babel. I would 
> like to use something like this: 
>
> #+srcname: sin_sum()
> #+begin_src python :results file 
>   plot(x,y)
>   filename = "figure.pdf"
>   savefig(filename)
> #+end_src filename
>
> #+resname: [[figure.pdf]]

Yes, I think we do want a version of this for python and ruby et al. In
your example, the filename is created in python. I suggest doing it
slightly differently, something like this.

#+srcname: fileoutput
#+begin_src python :file outfile.txt
  def savetofile(result, filename):
      with open(filename, 'w') as f:
          f.write(str(result))
  savetofile(78, 'outfile.txt')
  55
#+end_src

#+resname: fileoutput
[[file:outfile.txt]]

This functionality is now available for ruby & python in branch
ded-babel of git://repo.or.cz/org-mode/babel.git.

So here, if you specify :file <filepath> ruby/python blindly outputs a
link to <filepath>, regardless of the contents of the
code. Responsibility for creating useful contents of <filepath> lies
with the code. Notice that with this you have to specify the output file
twice: once as an org-babel directive, and once in the python code. This
is in contrast to the graphics languages (dot, ditaa, asymptote), where
the results *automatically* get sent to the file specified by :file. The
same is also true now for graphical output from R.

The difference with python, ruby et al is that they might create file
output in a variety of ways which we can't anticipate, so we can't
automatically send output to the file. In contrast, the graphics
language *always* create file output and always do it in the same
way. [And in R it is possible to divert all graphical output to file] A
possible extension of the above might be to use a "magic variable" so
that a python variable is created e.g. __org_babel_output_file__ that
always holds a string corresponding to the file specified by :file. Eric
may have further ideas / views here. 


>
> By this way I could clearly distinguish between what is pure python and what 
> is needed to wrap it in org-mode
>
> 2. Is it possible (by a keyboard-shortcut) to execute all blocks of a session 
> starting from top to down?

In addition to Tom's suggestion, the functions org-babel-execute-buffer
and org-babel-execute-subtree have existed all along, although I'm not
sure how much use they've had so far. Please let us know whether there
are any problems in using them and how exactly this sort of
functionality should be extended (e.g. w.r.t. executing in specific
sessions, etc)

> Sometimes I make small changes here and there and I 
> just like to say o.k. tangle all this together and execute it, refresh and 
> show me all new results. This might be even combined by a org-mode variable 
> which allow to define which results need to be refreshed before saving the org-
> mode file (something like #+EXECBYSAVE: name_of_session).
> I press C-x C-s rather frequently without much thinking of it. If I want make 
> sure that e.g., all generated output files (e.g. figures) are in sync with the 
> code it would be nice to say C-x C-s  and all external figures will be 
> recreated if necessary.
>  
> 3. If somethings went bad during execution of a block, org-babel will report 
> "no output"... is there any way to catch errors and place a warning or/and 
> even jump to the code block in question?

I intend to work on this when I get time. But of course if anyone else
would like to help out... I think we will want a uniform approach across
all languages. Something like, at the end of evaluation, org-babel has a
stdout and stderr object, and it checks that stderr is empty before
giving "normal results". The mechanism for filling stderr and stdout
would vary across languages and according to whether evaluation is in an
external shell process or emacs comint. E.g. in session evaluation it
would use try...except in python, and try() in R. In external process
evaluation it would use the output stream redirection mechanisms
available in the language or in emacs.

>
> 4. Is there a way to see all source code blocks of a single session tangled 
> together in a single buffer and changes in this buffer will be placed back after 
> leaving into the individual blocks again 

It sounds nice, but very challenging. Would such a mechanism be robust
to very disruptive edits in the tangled code buffer (e.g. moving large
blocks of text around)? Would it be able to meaningfully maintain a
mapping between buffer regions and code blocks? You'd want to start off
by looking at the code in org-src.el to see whether you thought it could
be extended.

Thanks a lot for the ideas and feedback.

Dan

> Similar like C-c ' just for a complete session with minimal comments to split 
> it later:
> e.g.,
>
> #[[blockname1]]
> a= 1
> b= 2
> #[[blockname2]]
> result = a +b
>
> would allow me to work on the code in a single file in python mode and later it 
> will be displayed again in the individual source blocks in my org-file.
>
> I guess that was all I noticed during using org-bable over the last days.
> It would be nice to hear how others think about the above points. Did you find 
> a better solution already ? Any tricks or any functions I might missing?
> Again many thanks for this really nice org-mode contribution.
>
>
> Best regards
>
> Torsten
>  
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel]: Some feedback after the first week usage

[Tom Short]

Dan Davison <davison <at> stats.ox.ac.uk> writes:

> 
> Torsten Wagner <torsten.wagner <at> gmail.com> writes:
> 
> <...>
> > 4. Is there a way to see all source code blocks of a single session tangled 
> > together in a single buffer and changes in this buffer will be placed back
after 
> > leaving into the individual blocks again 
> 
> It sounds nice, but very challenging. Would such a mechanism be robust
> to very disruptive edits in the tangled code buffer (e.g. moving large
> blocks of text around)? Would it be able to meaningfully maintain a
> mapping between buffer regions and code blocks? You'd want to start off
> by looking at the code in org-src.el to see whether you thought it could
> be extended.

One way this could be implemented is with an indirect buffer. In the indirect
buffer, you could use an overlay to hide all the org parts and just leave the
code parts visible. You're not really tangling, just hiding the non-code stuff.
You might even be able to operate the indirect buffer in ess-mode or python mode
if you wanted.

- Tom

Re: [babel]: Some feedback after the first week usage

[Eric Schulte]

Torsten Wagner <torsten.wagner@gmail.com> writes:

> Hi everyone,
> I would like to report the usage of org-babel after a 1 week usage in 
> "serious" project work. 
>
> First of all org-babel is a masterpiece. Many thanks to the developers. I 
> never saw a project which was such useful in this very early stage

Thanks, that's very kind.  It has been incredibly gratifying to see that
people are finding this to be useful, and are doing cooler things with
it than we had anticipated.

> (how old is org-babel?).

I think the official start date for org-babel would be

Thu, 5 Feb 2009 11:14:24 -0500

the date of an email sent out by Dan proposing this work.  I think it
was another month or two before we actually started generating code.

Thanks -- Eric

Re: [babel]: Some feedback after the first week usage

[Torsten Wagner]

Hi Tom

> Would something like run-code-blocks (below) do what you want?  I put  
> a source block like this at the top of my org-babel file, where I can  
> find it quickly.
> 
> #+srcrname: run-code-blocks
> #+begin_src python :noweb
> <<block-1>>
> <<block-2>>
> #+end_src
> 
> #+srcname: block-1
> #+begin_src python
> a = 1
> a
> #+end_src
> 
> #+srcname: block-2
> #+begin_src python
> b = 1
> b
> #+end_src

Actually, that is a very nice and smart idea :)
It could be something like a Makefile allowing me to switch on and off different 
blocks (e.g., for debugging if there is an error somewhere), switch between 
different blocks (e.g. real data from file and test data set) or even modify 
temporarily variables in between 

For example I could write:

#+srcrname: run-code-blocks
#+begin_src python :noweb
 # <<block-1>> # choose between real data
 <<block-test>> # and test data	
 # wondering whether it would work with negative values
 a = -1 	
 <<block-2>>
#+end_src

#+srcname: block-1
#+begin_src python
a = open("foo.txt","r")
a
#+end_src

#+srcname: block-test
#+begin_src python
a = 1
a
#+end_src

#+srcname: block-2
#+begin_src python
b = 1
b
#+end_src

Nice, will try it and many thanks for sharing

Greetings

Torsten

Re: [babel]: Some feedback after the first week usage

[Thomas S. Dye]

On Nov 5, 2009, at 7:27 AM, Dan Davison wrote:
> ...
> Yes, I think we do want a version of this for python and ruby et al.  
> In
> your example, the filename is created in python. I suggest doing it
> slightly differently, something like this.
>
> #+srcname: fileoutput
> #+begin_src python :file outfile.txt
>  def savetofile(result, filename):
>      with open(filename, 'w') as f:
>          f.write(str(result))
>  savetofile(78, 'outfile.txt')
>  55
> #+end_src
>
> #+resname: fileoutput
> [[file:outfile.txt]]
>
> This functionality is now available for ruby & python in branch
> ded-babel of git://repo.or.cz/org-mode/babel.git.
>
> So here, if you specify :file <filepath> ruby/python blindly outputs a
> link to <filepath>, regardless of the contents of the
> code. Responsibility for creating useful contents of <filepath> lies
> with the code. Notice that with this you have to specify the output  
> file
> twice: once as an org-babel directive, and once in the python code.  
> This
> is in contrast to the graphics languages (dot, ditaa, asymptote),  
> where
> the results *automatically* get sent to the file specified by :file.  
> The
> same is also true now for graphical output from R.
>
> The difference with python, ruby et al is that they might create file
> output in a variety of ways which we can't anticipate, so we can't
> automatically send output to the file. In contrast, the graphics
> language *always* create file output and always do it in the same
> way. [And in R it is possible to divert all graphical output to  
> file] A
> possible extension of the above might be to use a "magic variable" so
> that a python variable is created e.g. __org_babel_output_file__ that
> always holds a string corresponding to the file specified by :file.  
> Eric
> may have further ideas / views here.
> ...


Aloha Dan,

Nice idea.  It is great to have a link to the graphic in the org file.

With the graphics file name available to org-babel, it might be useful  
to have

<<fileoutput()>>

or some such thing, return the file name.  Then one could do something  
like this:

#+begin_src latex
...
\includegraphics{<<fileoutput()>>}
...
#+end_src

This way the file name would only have to be specified twice, and not  
every time other code wanted to refer to the output of fileoutput.

All the best,
Tom