Macro expansion in new exporter

Macro expansion in new exporter

[T.F. Torrey]

Hello all,

I really like the new exporter, and I appreciate the hard work of
everyone involved.

Right now, though, it's giving me a small problem: in the export to
HTML, macro's are not expanded, so I have {{{title}}}, for instance, in
the HTML output.

I haven't been following the list as closely as I'd like, so I'm hoping
I missed something relevant in the changeover.

If anyone has any ideas, I'd appreciate them before I go digging.

Emacs  : GNU Emacs 24.3.50.1 (i686-pc-linux-gnu, GTK+ Version 3.6.0)
 of 2012-12-24 on menkib, modified by Debian
Package: Org-mode version 7.9.2+ (7.9.2+-GNU-Emacs-24-3 (commit 488eea) @ mixed installation! /usr/share/emacs/24.3.50/lisp/org/ and /home/tftorrey/.emacs.d/elisp/org/lisp/)

Terry
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nicolas Goaziou]

Hello,

tftorrey@tftorrey.com (T.F. Torrey) writes:

> Right now, though, it's giving me a small problem: in the export to
> HTML, macro's are not expanded, so I have {{{title}}}, for instance, in
> the HTML output.
>
> I haven't been following the list as closely as I'd like, so I'm hoping
> I missed something relevant in the changeover.
>
> If anyone has any ideas, I'd appreciate them before I go digging.

Macro expansion happens before export back-ends kick-in, as does Babel
code evaluation and file inclusion through #+include keywords. So the
problem (if there's one) doesn't come from ox-html.el.

On that topic, the main difference with the previous exporter is that
macros are now required to be in a context that can be parsed. Thus, for
example, the following is not a macro:

  ~{{{title}}}~

> Emacs  : GNU Emacs 24.3.50.1 (i686-pc-linux-gnu, GTK+ Version 3.6.0)
>  of 2012-12-24 on menkib, modified by Debian
> Package: Org-mode version 7.9.2+ (7.9.2+-GNU-Emacs-24-3 (commit 488eea) @ mixed installation! /usr/share/emacs/24.3.50/lisp/org/ and /home/tftorrey/.emacs.d/elisp/org/lisp/)

You have a mixed installation. You should perhaps fix this before trying
again.


Regards,

-- 
Nicolas Goaziou

Re: Macro expansion in new exporter

[T.F. Torrey]

Hello again,

Like many others, I'm adapting my workflow to the new exporter.  Like
Carsten (but apparently few others), I've been using macros extensively.
Though I've spent several days digging through the mailing list and
code, I still don't have the answers I need, but hopefully I can ask
intelligent questions.

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> On that topic, the main difference with the previous exporter is that
> macros are now required to be in a context that can be parsed. Thus, for
> example, the following is not a macro:
>
>   ~{{{title}}}~

What is meant by "a context that can be parsed"?

In my work, it has been very useful to use macros for snippets of
text.  Then, instead of changing the text everywhere when I want a
change, I would just change the macro.  So...

For instance, I used to be able to do this:

#+MACRO: status Draft
#+HTML: <p class="status">{{{status}}}</p>

And on export to HTML, I would get what you would expect:
<p class="status">Draft</p>

With the new exporter, the macro is left unexpanded in the output:
<p class="status">{{{status}}}</p>

Of course, I could also put the {{{status}}} in any ordinary text and
have it there as well.

In extensive experiments, I have not found any combination of input that
would produce the old output using macros.

The old behavior had an elegant, one-line solution.  Perhaps the
functionality could be duplicated with babel, but surely not as simply
and directly as with the old macro system.

Is there a way to replicate the old behavior in the new export engine?

Also, in your response to Carsten's question about macros, you suggested
this:

#+MACRO: thumbright @@html:<img src="./Content/$2/thumb.jpg" style="float:right;width:$1;margin:0px 20px 0px 20px;" alt="./Content/$2/thumb.jpg" />@@

The "@@" syntax looks new to me.  Can you tell me what the function of
the "@@" is?  Is this documented somewhere?

Best regards,
Terry
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nicolas Goaziou]

Hello,

tftorrey@tftorrey.com (T.F. Torrey) writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> On that topic, the main difference with the previous exporter is that
>> macros are now required to be in a context that can be parsed. Thus, for
>> example, the following is not a macro:
>>
>>   ~{{{title}}}~
>
> What is meant by "a context that can be parsed"?

Anywhere but in verbatim areas like:

  - example blocks
  - src blocks, inline src-blocks
  - fixed-width area
  - keywords (excepted CAPTION, DATE, TITLE, AUTHOR, EMAIL and MACRO).
  - verbatim and code emphasis.
  - block boundaries

> In my work, it has been very useful to use macros for snippets of
> text.  Then, instead of changing the text everywhere when I want a
> change, I would just change the macro.  So...
>
> For instance, I used to be able to do this:
>
> #+MACRO: status Draft
> #+HTML: <p class="status">{{{status}}}</p>
>
> And on export to HTML, I would get what you would expect:
> <p class="status">Draft</p>
>
> With the new exporter, the macro is left unexpanded in the output:
> <p class="status">{{{status}}}</p>

Because the macro is in a keyword that means: "do not parse this
contents, it is for the html back-end only".

> Of course, I could also put the {{{status}}} in any ordinary text and
> have it there as well.
>
> In extensive experiments, I have not found any combination of input that
> would produce the old output using macros.

It's possible, macros are not exactly the same. Some of their old
functionalities are meant to be reached through Babel now. It may sounds
bad, but it allows a better control on macro expansion.

> The old behavior had an elegant, one-line solution.  Perhaps the
> functionality could be duplicated with babel, but surely not as simply
> and directly as with the old macro system.
>
> Is there a way to replicate the old behavior in the new export engine?

I didn't check thoroughly, but I would say this kind of feature requires
some Babel code.

> Also, in your response to Carsten's question about macros, you suggested
> this:
>
> #+MACRO: thumbright @@html:<img src="./Content/$2/thumb.jpg" style="float:right;width:$1;margin:0px 20px 0px 20px;" alt="./Content/$2/thumb.jpg" />@@
>
> The "@@" syntax looks new to me.  Can you tell me what the function of
> the "@@" is?  Is this documented somewhere?

It is called an export-snippet. @@html:...@@ syntax is to #+begin_html
blocks what =...= is to #+begin_example blocks. It replaces old @<...>
syntax, which was HTML only. Now you can have @@latex:...@@ or
@@beamer:...@@ syntax.


Regards,

-- 
Nicolas Goaziou

Re: Macro expansion in new exporter

[T.F. Torrey]

Hello Nicolas,

Thank you for your thoughtful clarification about macros in the new
exporter.  Probably in the long run I will find happiness using babel
for what I want to do.  In the meantime, however, I have a few more
questions:

1. You wrote before that macros were scaled back because what they did
could be done by babel.  Is that really a good reason for removing
functionality?  Everything that Org does could be done in other tools,
and yet ... we have Org.

2. You wrote to Carsten that macros could no longer contain newlines.
That seems like an arbitrary limitation.  Is it?

3. Is there really a reason why macro expansion is limited to a few
keywords rather than all?  Who would that trip up?  Ditto for verbatim
and code emphasis.

4. Given that macro values are easy to find in the source document, and
unexpanded macros are easy to find in the output document, couldn't I
just add a filter to the exporter to find and expand any unexpanded
macros (and lingering newline indicators)?  Is there an easy method for
adding such a filter?

5. Actually, why do macros need to be an exporter problem at all?
Couldn't the macro functionality be put into a separate package that
used hooks and filters to connect itself into the export routine and the
various back-ends (if even necessary)?  Then macros could be made to do
interesting things without burdening the export engine (and its
maintainer) at all.

Thanks again for your amazing work.

Best regards,
Terry
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nicolas Goaziou]

tftorrey@tftorrey.com (T.F. Torrey) writes:

> 1. You wrote before that macros were scaled back because what they did
> could be done by babel.  Is that really a good reason for removing
> functionality?  Everything that Org does could be done in other tools,
> and yet ... we have Org.

Macros were scaled down because the parser has to be able to know when
it looks at one of them. This limits the places where a macro can be
found. For example, it doesn't make much sense to expect a macro to be
found in an example block.

But scaling down macros made some of its features disappear. Since Babel
code could provide them anyway, it generated no real regression.

To sum it up, they were not downgraded because of Babel, but if Babel
hadn't been there, this wouldn't have happened.

> 2. You wrote to Carsten that macros could no longer contain newlines.
> That seems like an arbitrary limitation.  Is it?

In the parser, there's a difference between objects and elements.
Distinction is made clear in the comments lines of "org-element.el". To
put it simply, objects cannot contain blank lines.

Macros are objects. If I allow a newline in a macro, I allow two and
therefore a blank line. For example a macro within a paragraph could
potentially split it into two paragraphs. In this case, there would be
a mismatch between what the user can see in the buffer, and what will
happen during export.

Babel code already has got this privilege (of breaking structure of the
document). During alpha-test phase, bugs were reported because of
unanticipated discrepancies between Babel code in original buffer and
results in export buffer.

Allowing macros to do the same would be asking for more trouble.

> 3. Is there really a reason why macro expansion is limited to a few
> keywords rather than all?  Who would that trip up?  Ditto for verbatim
> and code emphasis.

Most keywords are simple key and values couples. The parser mustn't look
into the values, as it may find constructs that do not exist.

On the other hand some selected keywords have to be parsed, because
their value is really Org data. They are defined in "org-element.el".
See `org-element-document-properties' and `org-element-parsed-keywords'
for an exhaustive list.

Verbatim and code emphasis contents are never parsed, by definition.

> 4. Given that macro values are easy to find in the source document, and
> unexpanded macros are easy to find in the output document, couldn't I
> just add a filter to the exporter to find and expand any unexpanded
> macros (and lingering newline indicators)?  Is there an easy method for
> adding such a filter?

It may be possible, although very hackish. Functions in filters are, at
the moment, called from the export buffer. So you can probably read
macro definitions there. I think a proper filter to use for that would
be `org-export-filter-final-output-functions'.

I still strongly suggest to use Babel instead.

> 5. Actually, why do macros need to be an exporter problem at all?
> Couldn't the macro functionality be put into a separate package that
> used hooks and filters to connect itself into the export routine and the
> various back-ends (if even necessary)?  Then macros could be made to do
> interesting things without burdening the export engine (and its
> maintainer) at all.

Macros are an exporter problem, albeit a limited one, because their
expansion happens during the export process. Also, some macros are very
export oriented and have to be handled at the export framework level
(e.g. "{{{author}}}"). Though, an export back-end actually never sees
any macro, as these are expanded before its translator functions are
called.

Most of the macro code lives in org.el, and will probably be moved into
a separate library. Again, their limitations come from "org-element.el",
not "ox.el". Parsing them means "no crazy stuff allowed". For crazy
stuff, look towards Babel.


Regards,

-- 
Nicolas Goaziou

Re: Macro expansion in new exporter

[T.F. Torrey]

Hello Nicolas,

Thank your for your thoughtful reply.

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> tftorrey@tftorrey.com (T.F. Torrey) writes:
>
>> 1. You wrote before that macros were scaled back because what they did
>> could be done by babel.  Is that really a good reason for removing
>> functionality?  Everything that Org does could be done in other tools,
>> and yet ... we have Org.
>
> Macros were scaled down because the parser has to be able to know when
> it looks at one of them. This limits the places where a macro can be
> found. For example, it doesn't make much sense to expect a macro to be
> found in an example block.
>
> But scaling down macros made some of its features disappear. Since Babel
> code could provide them anyway, it generated no real regression.

Perhaps.  We still know of no easy/straightforward way at all to
replicate using babel the behavior I had (creating '<p
class="foo">bar</p>' with a macro), let alone in a pair of single lines.

> To sum it up, they were not downgraded because of Babel, but if Babel
> hadn't been there, this wouldn't have happened.
>
>> 2. You wrote to Carsten that macros could no longer contain newlines.
>> That seems like an arbitrary limitation.  Is it?
>
> In the parser, there's a difference between objects and elements.
> Distinction is made clear in the comments lines of "org-element.el". To
> put it simply, objects cannot contain blank lines.

Thanks for the tip.  I've been very impressed with the comments in your
code.  Great work!

> Macros are objects. If I allow a newline in a macro, I allow two and
> therefore a blank line. For example a macro within a paragraph could
> potentially split it into two paragraphs. In this case, there would be
> a mismatch between what the user can see in the buffer, and what will
> happen during export.
>
> Babel code already has got this privilege (of breaking structure of the
> document). During alpha-test phase, bugs were reported because of
> unanticipated discrepancies between Babel code in original buffer and
> results in export buffer.
>
> Allowing macros to do the same would be asking for more trouble.

Macros are so straightforward compared to babel, I wonder how much
trouble they /could/ generate.

>> 3. Is there really a reason why macro expansion is limited to a few
>> keywords rather than all?  Who would that trip up?  Ditto for verbatim
>> and code emphasis.
>
> Most keywords are simple key and values couples. The parser mustn't look
> into the values, as it may find constructs that do not exist.
>
> On the other hand some selected keywords have to be parsed, because
> their value is really Org data. They are defined in "org-element.el".
> See `org-element-document-properties' and `org-element-parsed-keywords'
> for an exhaustive list.

Thanks for this valuable tip.

> Verbatim and code emphasis contents are never parsed, by definition.

I was reading that wrong.  "Code emphasis" is not "emphasis" (italics).
Sorry for the noise.

>> 4. Given that macro values are easy to find in the source document, and
>> unexpanded macros are easy to find in the output document, couldn't I
>> just add a filter to the exporter to find and expand any unexpanded
>> macros (and lingering newline indicators)?  Is there an easy method for
>> adding such a filter?
>
> It may be possible, although very hackish. Functions in filters are, at

I have a lot of code that could be described as "hackish", at best.  I
don't mind because producing beautiful code is pretty far down on my
list of goals.  I suspect, though, it would make you either laugh or
feel nauseous, probably both.

> the moment, called from the export buffer. So you can probably read
> macro definitions there. I think a proper filter to use for that would
> be `org-export-filter-final-output-functions'.

That's one I'm familiar with.  Thanks.

> I still strongly suggest to use Babel instead.

I wonder if an entry in the (underused) Library of Babel could replicate
the macro functionality and eliminate the need for macros as separate
things.  I will have to look into that when I have more time.

>> 5. Actually, why do macros need to be an exporter problem at all?
>> Couldn't the macro functionality be put into a separate package that
>> used hooks and filters to connect itself into the export routine and the
>> various back-ends (if even necessary)?  Then macros could be made to do
>> interesting things without burdening the export engine (and its
>> maintainer) at all.
>
> Macros are an exporter problem, albeit a limited one, because their
> expansion happens during the export process. Also, some macros are very
> export oriented and have to be handled at the export framework level
> (e.g. "{{{author}}}"). Though, an export back-end actually never sees
> any macro, as these are expanded before its translator functions are
> called.

I'm pretty sure I could replicate (and even expand) the old macro
functionality by hooking an expansion function into
org-export-before-parsing-hook.  Maybe I will write an extension to try.

> Most of the macro code lives in org.el, and will probably be moved into
> a separate library. Again, their limitations come from "org-element.el",
> not "ox.el". Parsing them means "no crazy stuff allowed". For crazy
> stuff, look towards Babel.

One of the great things about working in Emacs is that it is easy(-ish)
to hack together code to work around any design decisions or behavior
that you don't like.  One of the great things about Org is that the
files are plain text, so it is easy(-ish) to hack together code to make
whatever modifications you like.  And one of the great things about the
new exporter is that it provides so many clean and straightforward
interfaces for hacking the output to match whatever you might desire.

You have surely been more helpful than I deserve.  I sincerely
appreciate your time and work on this.

All the best,
Terry
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nicolas Goaziou]

tftorrey@tftorrey.com (T.F. Torrey) writes:

> Perhaps.  We still know of no easy/straightforward way at all to
> replicate using babel the behavior I had (creating '<p
> class="foo">bar</p>' with a macro), let alone in a pair of single
> lines.

Here is an example. Babel specialists can help you further.

--8<---------------cut here---------------start------------->8---
Define your Babel macro.

#+name: html-foo
#+begin_src org :exports none :results raw :var data=""
,#+HTML: <p class="foo">$data</p>
#+end_src

Now, use it anywhere in the code, any number of times.

#+CALL: html-foo(data="bar") :results raw
--8<---------------cut here---------------end--------------->8---

There are many, often shorter, other ways to do it.

> Macros are so straightforward compared to babel, I wonder how much
> trouble they /could/ generate.

Not as much as Babel, obviously. But since blank lines are significant
in Org, allowing blank lines in macros could change structure.

> I wonder if an entry in the (underused) Library of Babel could replicate
> the macro functionality and eliminate the need for macros as separate
> things.  I will have to look into that when I have more time.

You can install the example given above in you Library of Babel.

> I'm pretty sure I could replicate (and even expand) the old macro
> functionality by hooking an expansion function into
> org-export-before-parsing-hook.  Maybe I will write an extension to
> try.

I'd rather have users use macros for short replacements, which is where
they really shine, and Babel for programmable ones. Your extension
wouldn't help at that. But that's your call, obviously.

For the record, I have had to rewrite the macro code two times. This is
to say it is difficult to get them right, if possible at all.


Regards,

-- 
Nicolas Goaziou

Re: Macro expansion in new exporter

[T.F. Torrey]

Hello Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> tftorrey@tftorrey.com (T.F. Torrey) writes:
>
>> Perhaps.  We still know of no easy/straightforward way at all to
>> replicate using babel the behavior I had (creating '<p
>> class="foo">bar</p>' with a macro), let alone in a pair of single
>> lines.
>
> Here is an example. Babel specialists can help you further.
>
> --8<---------------cut here---------------start------------->8---
> Define your Babel macro.
>
> #+name: html-foo
> #+begin_src org :exports none :results raw :var data=""
> ,#+HTML: <p class="foo">$data</p>
> #+end_src
>
> Now, use it anywhere in the code, any number of times.
>
> #+CALL: html-foo(data="bar") :results raw
> --8<---------------cut here---------------end--------------->8---
>
> There are many, often shorter, other ways to do it.

This is interesting, but it is the reverse of the previous
functionality.  What I had was:

#+MACRO: data Snippet of awesome text that changes sometimes.
#HTML: <p class="awesome">{{{data}}}</p>

Your sample code is the equivalent of this, kind of:

#+MACRO: foo <p class="awesome">$1</p>
{{{foo("Snippet of awesome text that changes sometimes.")}}}

The original intent was to avoid changing the snippet of awesome text
everywhere it appears.

Your code actually does not actually work with my Org
(7.9.3e-1032-g791a8d). It asks me: "Evaluate this org code block
(html-foo) on your system? (y or n)" and is not happy with either
answer, exporting nothing.

This almost works, I think ...

#+name: snippet-awesome
#+begin_src org :exports results :results raw
Snippet of awesome text that changes sometimes.
#+end_src

#+HTML: <p class="foo">
#+CALL: snippet-awesome :results raw
#+HTML: </p>

... except that it doesn't actually export anything, and changing
:exports to none doesn't help.  And neither does removing the :results
raw from the call.

If it did work, it would be closer to what I had.  Not perfect, but
closer.

Maybe a babel specialist can say what is wrong with my attempt, or maybe
it's a bug.

I hope you won't spend more time on this.  You've spent way too much
time on my trivial problems already.

Thanks again!

Terry
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nicolas Goaziou]

tftorrey@tftorrey.com (T.F. Torrey) writes:

> This almost works, I think ...
>
> #+name: snippet-awesome
> #+begin_src org :exports results :results raw
> Snippet of awesome text that changes sometimes. #+end_src
>
> #+HTML: <p class="foo">
> #+CALL: snippet-awesome :results raw
> #+HTML: </p>
>
> ... except that it doesn't actually export anything, and changing
> :exports to none doesn't help.  And neither does removing the :results
> raw from the call.

Wild guess: did you activate org in your Babel languages?


Regards,

-- 
Nicolas Goaziou

Re: Macro expansion in new exporter

[T.F. Torrey]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> tftorrey@tftorrey.com (T.F. Torrey) writes:
>
>> This almost works, I think ...
>>
>> #+name: snippet-awesome
>> #+begin_src org :exports results :results raw
>> Snippet of awesome text that changes sometimes. #+end_src
>>
>> #+HTML: <p class="foo">
>> #+CALL: snippet-awesome :results raw
>> #+HTML: </p>
>>
>> ... except that it doesn't actually export anything, and changing
>> :exports to none doesn't help.  And neither does removing the :results
>> raw from the call.
>
> Wild guess: did you activate org in your Babel languages?

Doh!  And that was one of the things I checked, too.

Now it works, but it isn't a solution because the CALL is wrapped in its
own <p>.  I'll figure something out.

Thanks again.

T.
-- 
T.F. Torrey

Re: Macro expansion in new exporter

[Nick Dokos]

T.F. Torrey <tftorrey@tftorrey.com> wrote:

> ... 
> This almost works, I think ...
> 
> #+name: snippet-awesome
> #+begin_src org :exports results :results raw
> Snippet of awesome text that changes sometimes.
> #+end_src
> 
> #+HTML: <p class="foo">
> #+CALL: snippet-awesome :results raw
> #+HTML: </p>
> 
> ... except that it doesn't actually export anything, and changing
> :exports to none doesn't help.  And neither does removing the :results
> raw from the call.
> 
> If it did work, it would be closer to what I had.  Not perfect, but
> closer.
> 

The way I read the doc, you need to say

#+CALL: snippet-awesome[:results raw]()

Nick