Re: Macro expansion in new exporter

[Nicolas Goaziou <n.goaziou@gmail.com>]

tftorrey@tftorrey.com (T.F. Torrey) writes:

> 1. You wrote before that macros were scaled back because what they did
> could be done by babel.  Is that really a good reason for removing
> functionality?  Everything that Org does could be done in other tools,
> and yet ... we have Org.

Macros were scaled down because the parser has to be able to know when
it looks at one of them. This limits the places where a macro can be
found. For example, it doesn't make much sense to expect a macro to be
found in an example block.

But scaling down macros made some of its features disappear. Since Babel
code could provide them anyway, it generated no real regression.

To sum it up, they were not downgraded because of Babel, but if Babel
hadn't been there, this wouldn't have happened.

> 2. You wrote to Carsten that macros could no longer contain newlines.
> That seems like an arbitrary limitation.  Is it?

In the parser, there's a difference between objects and elements.
Distinction is made clear in the comments lines of "org-element.el". To
put it simply, objects cannot contain blank lines.

Macros are objects. If I allow a newline in a macro, I allow two and
therefore a blank line. For example a macro within a paragraph could
potentially split it into two paragraphs. In this case, there would be
a mismatch between what the user can see in the buffer, and what will
happen during export.

Babel code already has got this privilege (of breaking structure of the
document). During alpha-test phase, bugs were reported because of
unanticipated discrepancies between Babel code in original buffer and
results in export buffer.

Allowing macros to do the same would be asking for more trouble.

> 3. Is there really a reason why macro expansion is limited to a few
> keywords rather than all?  Who would that trip up?  Ditto for verbatim
> and code emphasis.

Most keywords are simple key and values couples. The parser mustn't look
into the values, as it may find constructs that do not exist.

On the other hand some selected keywords have to be parsed, because
their value is really Org data. They are defined in "org-element.el".
See `org-element-document-properties' and `org-element-parsed-keywords'
for an exhaustive list.

Verbatim and code emphasis contents are never parsed, by definition.

> 4. Given that macro values are easy to find in the source document, and
> unexpanded macros are easy to find in the output document, couldn't I
> just add a filter to the exporter to find and expand any unexpanded
> macros (and lingering newline indicators)?  Is there an easy method for
> adding such a filter?

It may be possible, although very hackish. Functions in filters are, at
the moment, called from the export buffer. So you can probably read
macro definitions there. I think a proper filter to use for that would
be `org-export-filter-final-output-functions'.

I still strongly suggest to use Babel instead.

> 5. Actually, why do macros need to be an exporter problem at all?
> Couldn't the macro functionality be put into a separate package that
> used hooks and filters to connect itself into the export routine and the
> various back-ends (if even necessary)?  Then macros could be made to do
> interesting things without burdening the export engine (and its
> maintainer) at all.

Macros are an exporter problem, albeit a limited one, because their
expansion happens during the export process. Also, some macros are very
export oriented and have to be handled at the export framework level
(e.g. "{{{author}}}"). Though, an export back-end actually never sees
any macro, as these are expanded before its translator functions are
called.

Most of the macro code lives in org.el, and will probably be moved into
a separate library. Again, their limitations come from "org-element.el",
not "ox.el". Parsing them means "no crazy stuff allowed". For crazy
stuff, look towards Babel.


Regards,

-- 
Nicolas Goaziou