Re: [babel]: Some feedback after the first week usage

[Dan Davison <davison@stats.ox.ac.uk>]

Torsten Wagner <torsten.wagner@gmail.com> writes:

<...>
> Do you spend  sometimes days or weeks in front of the PC to fiddle around with 
> some scientific simulation or some other programming task?

Yes.

> If your colleagues and esp. your boss are not deep inside what it means to 
> perform this kind of work (e.g., because they might be from another, 
> programming free, major), they pretty fast become nervous about why you are 
> going to loose so much time in front of the keyboard. 

Yes.

>
> With org-babel and a mixture of literate programming and RR, you will have a 
> new way to handle those requests.   
> Actually all you need is 
> C-c C-e d
> and sent it to the printer....
> The result will be a professional looking comprehensive status report. You can 
> bring it to the meetings, file them as lab-book, use them in many ways to show 
> what are you doing right at the moment. It includes everything from theory, 
> explanations, calculations up to graphs, tables  and even the source code.  
> And all you need is the above command which takes 1.2 seconds... :D
>
> Compare this with the "classical" approach (which is teached unfortunately in 
> most universities now). Bring your project to a point where all individual 
> parts work, create some figures, data and results of your project, create a new 
> MS Power-point presentation, copy and place figures and results inside, add 
> text and tables, make everything looks more or less uniform, print it.... this 
> might consume 1-2 days or ... 172800 seconds !!! and you will still miss many 
> information (e.g. which fitting function did you use for the graph ?).

Nice description Torsten. That is exactly the situation that (a) I am
always in and (b) originally motivated the project for me.

>
>
> As I said already, I'm really amazed that org-babel works all out so smoothly. 
> However, I noticed the following points which I like to publish here for 
> discussion.
>
> 1. Is there another possible way for creating the final statement for the 
> source block result ? 
> At the moment I have to write 
>
> #+srcname: plot()
> #+begin_src python :results file 
>   plot(x,y)
>   filename = "figure.pdf"
>   savefig(filename)
>   filename
> #+end_src
>
> #+resname: [[figure.pdf]]
>
> But from the python point of view the last command is not necessary. If I 
> tangle the source blocks or copy and paste them I might even become some 
> problems (e.g., if the content is a very large string it would be displayed on 
> the ipython console). In addition this commands will be included in the export 
> but have not really a function beside of the wrapping for org-babel. I would 
> like to use something like this: 
>
> #+srcname: sin_sum()
> #+begin_src python :results file 
>   plot(x,y)
>   filename = "figure.pdf"
>   savefig(filename)
> #+end_src filename
>
> #+resname: [[figure.pdf]]

Yes, I think we do want a version of this for python and ruby et al. In
your example, the filename is created in python. I suggest doing it
slightly differently, something like this.

#+srcname: fileoutput
#+begin_src python :file outfile.txt
  def savetofile(result, filename):
      with open(filename, 'w') as f:
          f.write(str(result))
  savetofile(78, 'outfile.txt')
  55
#+end_src

#+resname: fileoutput
[[file:outfile.txt]]

This functionality is now available for ruby & python in branch
ded-babel of git://repo.or.cz/org-mode/babel.git.

So here, if you specify :file <filepath> ruby/python blindly outputs a
link to <filepath>, regardless of the contents of the
code. Responsibility for creating useful contents of <filepath> lies
with the code. Notice that with this you have to specify the output file
twice: once as an org-babel directive, and once in the python code. This
is in contrast to the graphics languages (dot, ditaa, asymptote), where
the results *automatically* get sent to the file specified by :file. The
same is also true now for graphical output from R.

The difference with python, ruby et al is that they might create file
output in a variety of ways which we can't anticipate, so we can't
automatically send output to the file. In contrast, the graphics
language *always* create file output and always do it in the same
way. [And in R it is possible to divert all graphical output to file] A
possible extension of the above might be to use a "magic variable" so
that a python variable is created e.g. __org_babel_output_file__ that
always holds a string corresponding to the file specified by :file. Eric
may have further ideas / views here. 


>
> By this way I could clearly distinguish between what is pure python and what 
> is needed to wrap it in org-mode
>
> 2. Is it possible (by a keyboard-shortcut) to execute all blocks of a session 
> starting from top to down?

In addition to Tom's suggestion, the functions org-babel-execute-buffer
and org-babel-execute-subtree have existed all along, although I'm not
sure how much use they've had so far. Please let us know whether there
are any problems in using them and how exactly this sort of
functionality should be extended (e.g. w.r.t. executing in specific
sessions, etc)

> Sometimes I make small changes here and there and I 
> just like to say o.k. tangle all this together and execute it, refresh and 
> show me all new results. This might be even combined by a org-mode variable 
> which allow to define which results need to be refreshed before saving the org-
> mode file (something like #+EXECBYSAVE: name_of_session).
> I press C-x C-s rather frequently without much thinking of it. If I want make 
> sure that e.g., all generated output files (e.g. figures) are in sync with the 
> code it would be nice to say C-x C-s  and all external figures will be 
> recreated if necessary.
>  
> 3. If somethings went bad during execution of a block, org-babel will report 
> "no output"... is there any way to catch errors and place a warning or/and 
> even jump to the code block in question?

I intend to work on this when I get time. But of course if anyone else
would like to help out... I think we will want a uniform approach across
all languages. Something like, at the end of evaluation, org-babel has a
stdout and stderr object, and it checks that stderr is empty before
giving "normal results". The mechanism for filling stderr and stdout
would vary across languages and according to whether evaluation is in an
external shell process or emacs comint. E.g. in session evaluation it
would use try...except in python, and try() in R. In external process
evaluation it would use the output stream redirection mechanisms
available in the language or in emacs.

>
> 4. Is there a way to see all source code blocks of a single session tangled 
> together in a single buffer and changes in this buffer will be placed back after 
> leaving into the individual blocks again 

It sounds nice, but very challenging. Would such a mechanism be robust
to very disruptive edits in the tangled code buffer (e.g. moving large
blocks of text around)? Would it be able to meaningfully maintain a
mapping between buffer regions and code blocks? You'd want to start off
by looking at the code in org-src.el to see whether you thought it could
be extended.

Thanks a lot for the ideas and feedback.

Dan

> Similar like C-c ' just for a complete session with minimal comments to split 
> it later:
> e.g.,
>
> #[[blockname1]]
> a= 1
> b= 2
> #[[blockname2]]
> result = a +b
>
> would allow me to work on the code in a single file in python mode and later it 
> will be displayed again in the individual source blocks in my org-file.
>
> I guess that was all I noticed during using org-bable over the last days.
> It would be nice to hear how others think about the above points. Did you find 
> a better solution already ? Any tricks or any functions I might missing?
> Again many thanks for this really nice org-mode contribution.
>
>
> Best regards
>
> Torsten
>  
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode