Re: Macro expansion in new exporter

[tftorrey@tftorrey.com (T.F. Torrey)]

Hello Nicolas,

Thank your for your thoughtful reply.

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> tftorrey@tftorrey.com (T.F. Torrey) writes:
>
>> 1. You wrote before that macros were scaled back because what they did
>> could be done by babel.  Is that really a good reason for removing
>> functionality?  Everything that Org does could be done in other tools,
>> and yet ... we have Org.
>
> Macros were scaled down because the parser has to be able to know when
> it looks at one of them. This limits the places where a macro can be
> found. For example, it doesn't make much sense to expect a macro to be
> found in an example block.
>
> But scaling down macros made some of its features disappear. Since Babel
> code could provide them anyway, it generated no real regression.

Perhaps.  We still know of no easy/straightforward way at all to
replicate using babel the behavior I had (creating '<p
class="foo">bar</p>' with a macro), let alone in a pair of single lines.

> To sum it up, they were not downgraded because of Babel, but if Babel
> hadn't been there, this wouldn't have happened.
>
>> 2. You wrote to Carsten that macros could no longer contain newlines.
>> That seems like an arbitrary limitation.  Is it?
>
> In the parser, there's a difference between objects and elements.
> Distinction is made clear in the comments lines of "org-element.el". To
> put it simply, objects cannot contain blank lines.

Thanks for the tip.  I've been very impressed with the comments in your
code.  Great work!

> Macros are objects. If I allow a newline in a macro, I allow two and
> therefore a blank line. For example a macro within a paragraph could
> potentially split it into two paragraphs. In this case, there would be
> a mismatch between what the user can see in the buffer, and what will
> happen during export.
>
> Babel code already has got this privilege (of breaking structure of the
> document). During alpha-test phase, bugs were reported because of
> unanticipated discrepancies between Babel code in original buffer and
> results in export buffer.
>
> Allowing macros to do the same would be asking for more trouble.

Macros are so straightforward compared to babel, I wonder how much
trouble they /could/ generate.

>> 3. Is there really a reason why macro expansion is limited to a few
>> keywords rather than all?  Who would that trip up?  Ditto for verbatim
>> and code emphasis.
>
> Most keywords are simple key and values couples. The parser mustn't look
> into the values, as it may find constructs that do not exist.
>
> On the other hand some selected keywords have to be parsed, because
> their value is really Org data. They are defined in "org-element.el".
> See `org-element-document-properties' and `org-element-parsed-keywords'
> for an exhaustive list.

Thanks for this valuable tip.

> Verbatim and code emphasis contents are never parsed, by definition.

I was reading that wrong.  "Code emphasis" is not "emphasis" (italics).
Sorry for the noise.

>> 4. Given that macro values are easy to find in the source document, and
>> unexpanded macros are easy to find in the output document, couldn't I
>> just add a filter to the exporter to find and expand any unexpanded
>> macros (and lingering newline indicators)?  Is there an easy method for
>> adding such a filter?
>
> It may be possible, although very hackish. Functions in filters are, at

I have a lot of code that could be described as "hackish", at best.  I
don't mind because producing beautiful code is pretty far down on my
list of goals.  I suspect, though, it would make you either laugh or
feel nauseous, probably both.

> the moment, called from the export buffer. So you can probably read
> macro definitions there. I think a proper filter to use for that would
> be `org-export-filter-final-output-functions'.

That's one I'm familiar with.  Thanks.

> I still strongly suggest to use Babel instead.

I wonder if an entry in the (underused) Library of Babel could replicate
the macro functionality and eliminate the need for macros as separate
things.  I will have to look into that when I have more time.

>> 5. Actually, why do macros need to be an exporter problem at all?
>> Couldn't the macro functionality be put into a separate package that
>> used hooks and filters to connect itself into the export routine and the
>> various back-ends (if even necessary)?  Then macros could be made to do
>> interesting things without burdening the export engine (and its
>> maintainer) at all.
>
> Macros are an exporter problem, albeit a limited one, because their
> expansion happens during the export process. Also, some macros are very
> export oriented and have to be handled at the export framework level
> (e.g. "{{{author}}}"). Though, an export back-end actually never sees
> any macro, as these are expanded before its translator functions are
> called.

I'm pretty sure I could replicate (and even expand) the old macro
functionality by hooking an expansion function into
org-export-before-parsing-hook.  Maybe I will write an extension to try.

> Most of the macro code lives in org.el, and will probably be moved into
> a separate library. Again, their limitations come from "org-element.el",
> not "ox.el". Parsing them means "no crazy stuff allowed". For crazy
> stuff, look towards Babel.

One of the great things about working in Emacs is that it is easy(-ish)
to hack together code to work around any design decisions or behavior
that you don't like.  One of the great things about Org is that the
files are plain text, so it is easy(-ish) to hack together code to make
whatever modifications you like.  And one of the great things about the
new exporter is that it provides so many clean and straightforward
interfaces for hacking the output to match whatever you might desire.

You have surely been more helpful than I deserve.  I sincerely
appreciate your time and work on this.

All the best,
Terry
-- 
T.F. Torrey