Bug in new exporter with babel blocks

Bug in new exporter with babel blocks

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

Hello,

I’m dealing with a puzzling bug in the new exporter.  As background,
I’ve written a custom function to process special-blocks, to replicate
in the new exporter the functionality of the org-exp-blocks package
(http://orgmode.org/worg/org-contrib/org-exp-blocks.html).  Because I
need the un-processed text inside the special-block (not parsed into an
org-elements structure nor escaped for LaTeX special characters), I use
the :contents-begin and -end properties of the element to extract this
information.

These values aren’t accurate in the presence of babel source blocks.  In
order to reproduce this, extract the attached elisp and org files into
your home directory, adjust the paths to point to an up-to-date git
checkout of the org sources (I am on today’s commit 196c579), and
eval-buffer the lisp file in an emacs -Q (I am using an up-to-date trunk
checkout of the meacs sources).  Then look in the *Messages* buffer to
see what follows “RESULT IS: “.  It should be “bar”; I get something
else (specifically “ 1)” from the middle of the babel block).

Deleting the babel block from the org file and re-running the export
call delivers the expected “RESULT IS: bar” message.

(Note that the final output is correct because the test elisp falls back
to the original definition of ‘org-e-latex-special-block’, but in my use
the output is wrong because I rely on the result of the buffer-substring
call.)

I’m not sure if this is a long-standing problem or not; this is the
first time I’ve combined babel with my custom special-blocks code.  I’m
also at a loss of how to debug the internals of org-elements and
org-export, two complex bits of machinery.  I guess the value of the
:contents-begin and -end properties needs to be fixed in any case.  I’d
also be happy to discover another, better way of getting the raw text
content of the special-block that doesn’t succumb to this problem.

Thanks in advance,
Aaron Ecay


[-- Attachment #2: org-bug.el --]
[-- Type: application/emacs-lisp, Size: 708 bytes --]

[-- Attachment #3: org-bug.org --]
[-- Type: text/plain, Size: 128 bytes --]

* Intro

#+name: setup
#+begin_src elisp :results output :exports both
  (+ 1 1)
#+end_src

foo

#+begin_foo
bar
#+end_foo

baz

Re: Bug in new exporter with babel blocks

[Nicolas Goaziou]

Hello,


Aaron Ecay <aaronecay@gmail.com> writes:

> I’m dealing with a puzzling bug in the new exporter.  As background,
> I’ve written a custom function to process special-blocks, to replicate
> in the new exporter the functionality of the org-exp-blocks package
> (http://orgmode.org/worg/org-contrib/org-exp-blocks.html).

You needn't. org-exp-blocks functionalities are supported by the new
exporter out of the box.

> Because I need the un-processed text inside the special-block (not
> parsed into an org-elements structure nor escaped for LaTeX special
> characters),

Special blocks are de facto, recursive, much like drawers. Their
contents have to be parsed.

> I’m not sure if this is a long-standing problem or not; this is the
> first time I’ve combined babel with my custom special-blocks code.  I’m
> also at a loss of how to debug the internals of org-elements and
> org-export, two complex bits of machinery.  I guess the value of the
> :contents-begin and -end properties needs to be fixed in any case.

During export, the buffer being parsed isn't the original buffer,
because of babel block, include keywords and macros expansion. You may
want to have a look at `org-export-as' source code.

Hence, :contents-begin and :contents-end properties do not refer to real
buffer positions in the buffer from which `org-e-latex-special-block' is
called.

> I’d also be happy to discover another, better way of getting the raw
> text content of the special-block that doesn’t succumb to this
> problem.

If you must, you can try:

  (org-element-interpret-data (org-element-contents special-block))

from `org-e-latex-special-block'.


Regards,

-- 
Nicolas Goaziou

Re: Bug in new exporter with babel blocks

[Aaron Ecay]

Nicolas,

Thank you for your explanations, which were very helpful.

2013ko urtarrilak 23an, Nicolas Goaziou-ek idatzi zuen:
> You needn't. org-exp-blocks functionalities are supported by the new
> exporter out of the box.

Can you say more about this?  I looked for but did not find a
replacement to the org-export-blocks variable (an alist associating
block types with functions to export them).  I found it very easy to
hook into the new exporter, but perhaps I missed something?

> Special blocks are de facto, recursive, much like drawers. Their
> contents have to be parsed.

For parsing, yes.  But for export I want a way to say “I don’t care what
Org thinks the export of this block is.  Give me the raw contents, and I
will tell you what the export should be.”

This is how the ditaa special-block code used to work; I see that it has
now morphed into a babel language, which makes some kind of sense.  I’m
not sure it does in general.

My use case is glossed examples for linguistics: my special block
contains three lines, which are a sentence in a foreign language and a
translation.  By inserting markup in a way which is easy to automate,
you can get LaTeX to align the words of one language with the words of
the other.  I don’t want any org processing of the text of the examples:
it might contain backslashes, stars, etc., all of which should be passed
verbatim to LaTeX.  This does not feel like source code, it cannot be
evaluated or tangled, I would not want these blocks to be included in
org-babel-next-src-block, etc.

>> I’d also be happy to discover another, better way of getting the raw
>> text content of the special-block that doesn’t succumb to this
>> problem.
> 
> If you must, you can try:
> 
>   (org-element-interpret-data (org-element-contents special-block))
> 
> from `org-e-latex-special-block'.

I would up patching org-elements to add a :contents property to
special-block elements, which is populated when parsing the original
buffer (and thus dodges the different-buffer-for-export problem).  I can
then retrieve this in my export backend function.  It is a very simple
patch:
-----------cut-here-----------
diff --git i/lisp/org-element.el w/lisp/org-element.el
index 3dc1e72..b67e5e6 100644
--- i/lisp/org-element.el
+++ w/lisp/org-element.el
@@ -1389,6 +1389,9 @@ Assume point is at the beginning of the block."
 			 :hiddenp hidden
 			 :contents-begin contents-begin
 			 :contents-end contents-end
+			 :contents (and contents-begin contents-end
+					(buffer-substring-no-properties
+					 contents-begin contents-end))
 			 :post-blank (count-lines pos-before-blank end)
 			 :post-affiliated post-affiliated)
 		   (cdr affiliated)))))))))
-----------cut-here-----------

Is including support for special blocks that should be exported “raw” a
compelling reason to install such a patch?  I think the only downside
would be increased memory usage/decreased speed for parsed objects
(since they are now storing an extra string), but I think that would be
very small (though I haven’t benchmarked anything).

Thanks,

-- 
Aaron Ecay

Re: Bug in new exporter with babel blocks

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> 2013ko urtarrilak 23an, Nicolas Goaziou-ek idatzi zuen:
>> You needn't. org-exp-blocks functionalities are supported by the new
>> exporter out of the box.
>
> Can you say more about this?  I looked for but did not find a
> replacement to the org-export-blocks variable (an alist associating
> block types with functions to export them).  I found it very easy to
> hook into the new exporter, but perhaps I missed something?

No, you didn't miss anything. I was thinking about
`org-special-blocks.el'. The exporter doesn't implement org-exp-blocks
functionalities.

Though, IIRC, org-exp-blocks was mostly deprecated when Babel was
introduced. That's why old ditaa and dot "exp-blocks" were moved to src
block languages.

> For parsing, yes.  But for export I want a way to say “I don’t care what
> Org thinks the export of this block is.  Give me the raw contents, and I
> will tell you what the export should be.”
>
> This is how the ditaa special-block code used to work; I see that it has
> now morphed into a babel language, which makes some kind of sense.  I’m
> not sure it does in general.
>
> My use case is glossed examples for linguistics: my special block
> contains three lines, which are a sentence in a foreign language and a
> translation.  By inserting markup in a way which is easy to automate,
> you can get LaTeX to align the words of one language with the words of
> the other.  I don’t want any org processing of the text of the examples:
> it might contain backslashes, stars, etc., all of which should be passed
> verbatim to LaTeX.  This does not feel like source code, it cannot be
> evaluated or tangled, I would not want these blocks to be included in
> org-babel-next-src-block, etc.

It is data. Source code is for the processing function.

Anyway, just store your text in example blocks, and create a src block
in any language to do the processing. Then you can call the processing
function on all of your example blocks.

I suggest to keep the example blocks in a non-exportable section. You
can also remove all named example blocks before export, via a hook. Your
call.

#+begin_src org
* Data                                                                  :noexport:
  
  #+name: words
  #+BEGIN_EXAMPLE
  a b c
  #+END_EXAMPLE
  
  #+name: 
  #+BEGIN_SRC emacs-lisp :var x=words
  (upcase x)
  #+END_SRC

* Contents

  #+call: words(:var x=words)
#+end_src


Regards,

-- 
Nicolas Goaziou