update on missing :parameters in code blocks

update on missing :parameters in code blocks

[John Kitchin]

I did some more digging on the missing :parameters during export, and it
seems that the src block itself is different during export than in the
buffer.  Below illustrates what I mean. In the buffer, if I look at the
contents of the code block using output from org-element-at-point, it
looks as I expect, with the complete header. But, using the same method
in the export, it is clear that the header args are not there.

I have not figured out where they disappear yet.

* Bug report 2 for org-mode

Here is a named table
#+tblname: tbl-data
| x | y |
|---+---|
| 1 | 1 |
| 2 | 4 |
| 3 | 9 |

Here is a named src-block with a defined var
#+name: print-table
#+BEGIN_SRC python :var data=tbl-data :results value
return data 
#+END_SRC


Now, we illustrate that the src-block does have :parameters.
#+name: get-properties
#+BEGIN_SRC emacs-lisp
(save-excursion
  (re-search-backward "#\\+name: print-table")
  (buffer-substring
   (org-element-property :begin (org-element-at-point))
   (org-element-property :end (org-element-at-point))))
#+END_SRC

#+RESULTS: get-properties
: #+name: print-table
: #+BEGIN_SRC python :var data=tbl-data :results value
: return data 
: #+END_SRC
: 
: 

In the export machinery, though it appears the src-block is not the same.

#+BEGIN_SRC emacs-lisp :results output :exports none
(defun my-src-block (src-block contents info)

  (princ (buffer-substring
	  (org-element-property :begin src-block)
	  (org-element-property :end src-block)))
  
    (org-export-format-code-default src-block info))

(org-export-define-derived-backend 'my-html 'html
  :translate-alist '((src-block . my-src-block)))

(org-export-to-file 'my-html "custom-src-table-export.html")
#+END_SRC
#+RESULTS:
#+begin_example
#+name: print-table
#+BEGIN_SRC python
return data 
#+END_SRC


#+name: get-properties
#+BEGIN_SRC emacs-lisp
(save-excursion
  (re-search-backward "#\\+name: print-table")
  (buffer-substring
   (org-element-property :begin (org-element-at-point))
   (org-element-property :end (org-element-at-point))))
#+END_SRC

#+end_example

The parameters seem to be gone in the python block.

-- 
-----------------------------------
John Kitchin
http://kitchingroup.cheme.cmu.edu

Re: update on missing :parameters in code blocks

[Charles Berry]

John Kitchin <jkitchin <at> andrew.cmu.edu> writes:

> 
> I did some more digging on the missing :parameters during export, and it
> seems that the src block itself is different during export than in the
> buffer.  Below illustrates what I mean. In the buffer, if I look at the
> contents of the code block using output from org-element-at-point, it
> looks as I expect, with the complete header. But, using the same method
> in the export, it is clear that the header args are not there.
> 
> I have not figured out where they disappear yet.
> 
[snip]

Instrument org-export-as. Then step thru it.

Before this line:
 
 (org-export-execute-babel-code)

the buffer copy ( my-buffer<2>, say ) will show the parameters/headers.

After that line is executed, they are gone.

You can work around this by `advicing' various babel functions and inserting
things you want to save as attributes, but it would be nice not to need
to do that.

HTH,

Chuck

Re: update on missing :parameters in code blocks

[John Kitchin]

Charles Berry <ccberry@ucsd.edu> writes:

Thanks for the tips. I did not see a way to do the advice here. For what
I need, we need to save parameters for each code block. But it appears
they are all affected by one function org-babel-exp-process-buffer, and
advising this kind of function is tantamount to rewriting the function!

my solution is here:
http://kitchingroup.cheme.cmu.edu/blog/2014/09/22/Showing-what-data-went-into-a-code-block-on-export/

I agree it would be nice if that was unnecessary, but it is not too
terrible for now. 


> John Kitchin <jkitchin <at> andrew.cmu.edu> writes:
>
>> 
>> I did some more digging on the missing :parameters during export, and it
>> seems that the src block itself is different during export than in the
>> buffer.  Below illustrates what I mean. In the buffer, if I look at the
>> contents of the code block using output from org-element-at-point, it
>> looks as I expect, with the complete header. But, using the same method
>> in the export, it is clear that the header args are not there.
>> 
>> I have not figured out where they disappear yet.
>> 
> [snip]
>
> Instrument org-export-as. Then step thru it.
>
> Before this line:
>  
>  (org-export-execute-babel-code)
>
> the buffer copy ( my-buffer<2>, say ) will show the parameters/headers.
>
> After that line is executed, they are gone.
>
> You can work around this by `advicing' various babel functions and inserting
> things you want to save as attributes, but it would be nice not to need
> to do that.
>
> HTH,
>
> Chuck
>
>
>
>
>

-- 
-----------------------------------
John Kitchin
Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu

Re: update on missing :parameters in code blocks

[Nicolas Goaziou]

Hello,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> Thanks for the tips. I did not see a way to do the advice here. For what
> I need, we need to save parameters for each code block. But it appears
> they are all affected by one function org-babel-exp-process-buffer, and
> advising this kind of function is tantamount to rewriting the
> function!

Actually, this is not related to this function, which is but a big
dispatcher. Block transformation happens at a lower level (e.g.,
`org-babel-exp-src-block' and deeper).


Regards,

-- 
Nicolas Goaziou

Re: update on missing :parameters in code blocks

[John Kitchin]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Thanks for the clarification. Is there any chance you know why this
transformation is done? Does it avoid some problem?

> Hello,
>
> John Kitchin <jkitchin@andrew.cmu.edu> writes:
>
>> Thanks for the tips. I did not see a way to do the advice here. For what
>> I need, we need to save parameters for each code block. But it appears
>> they are all affected by one function org-babel-exp-process-buffer, and
>> advising this kind of function is tantamount to rewriting the
>> function!
>
> Actually, this is not related to this function, which is but a big
> dispatcher. Block transformation happens at a lower level (e.g.,
> `org-babel-exp-src-block' and deeper).
>
>
> Regards,

-- 
-----------------------------------
John Kitchin
Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu

Re: update on missing :parameters in code blocks

[Nicolas Goaziou]

Hello,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> Thanks for the clarification. Is there any chance you know why this
> transformation is done? Does it avoid some problem?

No idea.


Regards,

-- 
Nicolas Goaziou

Re: update on missing :parameters in code blocks

[Aaron Ecay]

Hi John,

Look at the functions ‘org-babel-exp-src-block’ which calls
‘org-babel-exp-do-export’, which calls ‘org-babel-exp-code’.  The tl;dr
version is that indeed the babel export machinery does change the code
block in substantial ways, including the removal of parts of it.

This plays merry hell with the cache mechanism, as you might imagine
(different header args at different points -> the sha1 hash changes).  A
year or more ago I worked on a patch to overhaul this system.  I got
partway through before giving up, because it turned into a massive
undertaking and because it became clear that the cache mechanism would
not be very reliable/useful for my needs anyway.  But IMHO it remains an
imperfection in the interface between babel and the new parser, and it
might be possible to avoid the necessity of doing this sort of
destructive modification during export.  Along the way simplification of
the code might also be possible.

Let me know if you’re interested; I may be able to dig the old
half-patch out of a disused git branch somewhere.  It may have bitrotted
some, but it may also be useful.

-- 
Aaron Ecay

Re: update on missing :parameters in code blocks

[John Kitchin]

Aaron Ecay <aaronecay@gmail.com> writes:

Thanks for the confirmation this happens, and the pointer to where it
happens.

I guess there was at one point a good reason to do this, but I cannot
see it directly.

I found a way to do it with filters and preprocessing, which is
illustrated here:

http://kitchingroup.cheme.cmu.edu/blog/2014/09/22/Showing-what-data-went-into-a-code-block-on-export/

It works, but I feel like it is a workaround that should not be
needed. It does look like some effort to change this, and first it would
be good to know why the modification is being done.

Thanks again for the hints.



> Hi John,
>
> Look at the functions ‘org-babel-exp-src-block’ which calls
> ‘org-babel-exp-do-export’, which calls ‘org-babel-exp-code’.  The tl;dr
> version is that indeed the babel export machinery does change the code
> block in substantial ways, including the removal of parts of it.
>
> This plays merry hell with the cache mechanism, as you might imagine
> (different header args at different points -> the sha1 hash changes).  A
> year or more ago I worked on a patch to overhaul this system.  I got
> partway through before giving up, because it turned into a massive
> undertaking and because it became clear that the cache mechanism would
> not be very reliable/useful for my needs anyway.  But IMHO it remains an
> imperfection in the interface between babel and the new parser, and it
> might be possible to avoid the necessity of doing this sort of
> destructive modification during export.  Along the way simplification of
> the code might also be possible.
>
> Let me know if you’re interested; I may be able to dig the old
> half-patch out of a disused git branch somewhere.  It may have bitrotted
> some, but it may also be useful.

-- 
-----------------------------------
John Kitchin
http://kitchingroup.cheme.cmu.edu