limitation for macro expansion

limitation for macro expansion

[Yasushi SHOJI]

Hi,

I'm can't expand macro with prefix.  Is there any limitation for macro
expansion listed in the doc?

* test
- It works with chars around like this: Aa{{{author}}}bB.
- It works with numbers around like this: 12{{{author}}}34.
- It works with _ around like this: _{{{author}}}_.
- It works with =__= around like this: __{{{author}}}__.
- But it doesn't work with 'a_' prefixed like this: a_{{{author}}}.
- But it doesn't work with '1_' prefixed like this: 1_{{{author}}}.
- Link doesn't work like this: [[file:{{{input-file}}}][Bad link]]

Thanks,
--
          yashi

Re: limitation for macro expansion

[Nicolas Goaziou]

Hello,

Yasushi SHOJI <yashi@atmark-techno.com> writes:

> I'm can't expand macro with prefix.  Is there any limitation for macro
> expansion listed in the doc?

I don't think so.

> * test
> - It works with chars around like this: Aa{{{author}}}bB.
> - It works with numbers around like this: 12{{{author}}}34.
> - It works with _ around like this: _{{{author}}}_.
> - It works with =__= around like this: __{{{author}}}__.
> - But it doesn't work with 'a_' prefixed like this: a_{{{author}}}.
> - But it doesn't work with '1_' prefixed like this: 1_{{{author}}}.

The two examples above are ambiguous. You should use {{{{author}}}}.

> - Link doesn't work like this: [[file:{{{input-file}}}][Bad link]]

Indeed. This kind of link is not supported as you cannot follow it
(macros are an export-only feature).

Feel free to provide a documentation patch if you think this needs to be
made explicit.

Thank you.


Regards,

-- 
Nicolas Goaziou

Re: limitation for macro expansion

[Yasushi SHOJI]

Hi Nicolas,

On Sun, 05 Mar 2017 19:58:43 +0900,
Nicolas Goaziou wrote:
> 
> Yasushi SHOJI <yashi@atmark-techno.com> writes:
> > * test
> > - It works with chars around like this: Aa{{{author}}}bB.
> > - It works with numbers around like this: 12{{{author}}}34.
> > - It works with _ around like this: _{{{author}}}_.
> > - It works with =__= around like this: __{{{author}}}__.
> > - But it doesn't work with 'a_' prefixed like this: a_{{{author}}}.
> > - But it doesn't work with '1_' prefixed like this: 1_{{{author}}}.
> 
> The two examples above are ambiguous.

Ah, you mean the parser is unable to distinguish the macro and
subscript?  I didn't notice it since I've been disabled sub /
superscript feature all together.  In my configuration it seemed that
Org is silently ignoring the macro expansion.  Thank you for letting
me know!

> You should use {{{{author}}}}.

Doesn't seems to work.  When I export 'a_{{{{author}}}}', it
generates:

  a_{Yasushi SHOJI}

I was expecting to see:

  a_Yasushi SHOJI

> > - Link doesn't work like this: [[file:{{{input-file}}}][Bad link]]
> 
> Indeed. This kind of link is not supported as you cannot follow it
> (macros are an export-only feature).

Hmm, that's true that you can't follow it.

Would it be OK to say that it's a design decision to ignore the macro
expansion in the link field at export time?

> Feel free to provide a documentation patch if you think this needs to be
> made explicit.

Will do once I fully understand.

Now, what I'm trying to achieve with a macro is to generate a
filename-like string with a timestamp in it in my doc.

ie)

  #+MACRO: timestamp {{{date(%Y%m%d)}}}

  Please open log_{{{timestamp}}}.txt

Would it be possible for org to do this?


If not, would it be possible for me to modify the code to achieve this?
My stupid idea is to:

  - disable sub / superscript parser when org-use-sub-superscripts is nil
  - reverse the precedence order of subscript and macro

I know parsers are in org-element.el but not sure how the precedence
is decided.  If someone pointing at where to look, I'm very
appreciated.

Thanks,
-- 
           yashi

Re: limitation for macro expansion

[Nicolas Goaziou]

Hello,

Yasushi SHOJI <yashi@atmark-techno.com> writes:

> Ah, you mean the parser is unable to distinguish the macro and
> subscript?

It's not the parser, but the syntax. Subscript is

  a_{...}

whereas macro is

  {{{macro(...)}}}

so, when you see

  a_{{{something}}}

there are two ways to interpret it: the one you expect and

  a_{CONTENTS} where CONTENTS is {{something}}.

Since subscript syntax started first, it has precedence over the macro.

> Doesn't seems to work.  When I export 'a_{{{{author}}}}', it
> generates:
>
>   a_{Yasushi SHOJI}
>
> I was expecting to see:
>
>   a_Yasushi SHOJI

This is correct if you use `ascii' back-end. How could you tell the
difference between

  a_{Yasushi SHOJI}

and

  a_Yasushi SHOJI

otherwise?

OTOH, `latex' back-end produces

  a\(_{\text{Yasushi SHOJI}}\)

which means the parser and the export process correctly handle it.

>> > - Link doesn't work like this: [[file:{{{input-file}}}][Bad link]]
>> 
>> Indeed. This kind of link is not supported as you cannot follow it
>> (macros are an export-only feature).
>
> Hmm, that's true that you can't follow it.
>
> Would it be OK to say that it's a design decision to ignore the macro
> expansion in the link field at export time?

It is. Actually, macros are not allowed in many places, as a design
decision. See

  (info "(org) Macro replacement")

for more details.

>> Feel free to provide a documentation patch if you think this needs to be
>> made explicit.
>
> Will do once I fully understand.
>
> Now, what I'm trying to achieve with a macro is to generate a
> filename-like string with a timestamp in it in my doc.
>
> ie)
>
>   #+MACRO: timestamp {{{date(%Y%m%d)}}}
>
>   Please open log_{{{timestamp}}}.txt
>
> Would it be possible for org to do this?

You could try

  #+MACRO: timestamp {{{time(%Y%m%d)}}}

  Please open log\under{}{{{timestamp()}}}.txt

> If not, would it be possible for me to modify the code to achieve this?
> My stupid idea is to:
>
>   - disable sub / superscript parser when org-use-sub-superscripts is nil

See `org-export-with-sub-superscripts'.

>   - reverse the precedence order of subscript and macro

This would only move the problem elsewhere.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: limitation for macro expansion

[Yasushi SHOJI]

Hi Nicolas,

On Mon, Mar 6, 2017 at 5:22 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Yasushi SHOJI <yashi@atmark-techno.com> writes:
> there are two ways to interpret it: the one you expect and
>
>   a_{CONTENTS} where CONTENTS is {{something}}.
>
> Since subscript syntax started first, it has precedence over the macro.

Yes.  That's what I understand from your response.  Thanks!

>> Doesn't seems to work.  When I export 'a_{{{{author}}}}', it
>> generates:
>>
>>   a_{Yasushi SHOJI}
>>
>> I was expecting to see:
>>
>>   a_Yasushi SHOJI
>
> This is correct if you use `ascii' back-end.

Sorry for the confusion.
I just thought that you gave me a way to use macro with underscore.

>> Would it be OK to say that it's a design decision to ignore the macro
>> expansion in the link field at export time?
>
> It is. Actually, macros are not allowed in many places, as a design
> decision. See
>
>   (info "(org) Macro replacement")
>
> for more details.

I assume that the key phrase is "anywhere Org markup is recognized".
Link format doesn't allow Org markup, right?

>> ie)
>>
>>   #+MACRO: timestamp {{{date(%Y%m%d)}}}
>>
>>   Please open log_{{{timestamp}}}.txt
>>
>> Would it be possible for org to do this?
>
> You could try
>
>   #+MACRO: timestamp {{{time(%Y%m%d)}}}
>
>   Please open log\under{}{{{timestamp()}}}.txt

Ah!  Great idea.  That's what I'm looking for. Thanks.

# I use file_name_with_underscore.txt more than subscripts
# I'd be nice, at least for me, to have '\sub' and '\super' special keywords
# but leave the underscores alone.

>> If not, would it be possible for me to modify the code to achieve this?
>> My stupid idea is to:
>>
>>   - disable sub / superscript parser when org-use-sub-superscripts is nil
>
> See `org-export-with-sub-superscripts'.

hmm.  just checked the source.  org-use-sub-superscripts is only for display.
org-export-with-sub-superscripts is just for exports.

I was gonna just by-pass or disable subscript parser all together when
org-use-sub-superscripts is nil but it doesn't seems to be a good idea, does it?

>>   - reverse the precedence order of subscript and macro
>
> This would only move the problem elsewhere.

True.
-- 
             yashi

Re: limitation for macro expansion

[Nicolas Goaziou]

Yasushi SHOJI <yashi@atmark-techno.com> writes:

> I assume that the key phrase is "anywhere Org markup is recognized".
> Link format doesn't allow Org markup, right?

Not in the first part indeed. You can, however, use a macro in the
description part of the link.

> # I use file_name_with_underscore.txt more than subscripts
> # I'd be nice, at least for me, to have '\sub' and '\super' special keywords
> # but leave the underscores alone.

I don't understand where you need this. At the export level, you can use
`org-export-with-sub-superscripts' to `{}'. At display level, you can do
the same with `org-use-sub-superscripts'.

> hmm.  just checked the source.  org-use-sub-superscripts is only for display.
> org-export-with-sub-superscripts is just for exports.

See above.

> I was gonna just by-pass or disable subscript parser all together when
> org-use-sub-superscripts is nil but it doesn't seems to be a good
> idea, does it?

This is exactly what a nil `org-export-with-sub-superscripts' does.

Regards,

Re: limitation for macro expansion

[Yasushi SHOJI]

Hi Nicolas,

Thank you for your time.  I really appreciate it.

On Tue, 07 Mar 2017 01:41:34 +0900,
Nicolas Goaziou wrote:
>
> Yasushi SHOJI <yashi@atmark-techno.com> writes:
>
> > I assume that the key phrase is "anywhere Org markup is recognized".
> > Link format doesn't allow Org markup, right?
>
> Not in the first part indeed. You can, however, use a macro in the
> description part of the link.

OK.

> > # I use file_name_with_underscore.txt more than subscripts
> > # I'd be nice, at least for me, to have '\sub' and '\super' special keywords
> > # but leave the underscores alone.
>
> I don't understand where you need this. At the export level, you can use
> `org-export-with-sub-superscripts' to `{}'. At display level, you can do
> the same with `org-use-sub-superscripts'.

I hate to waste your precious time because of my ignorant and I can
live with '\under' but I'm totally lost.

Are you saying that it's possible to generate "a_20170307.txt" from
"a_{{{timestamp}}}.txt" if I set those variables mentioned above
correctly? without using '\under'?

I know you said it's ambiguous.  So let me clarify what I said. What I
said above was to change the syntax (or lex / parser's behavior) when
some variable, let's say 'org-do-not-parse-sub-superscripts', is set
to nil.

So that a) org leaves '_' in "a_{{{somemacro}}}" as-is and the macro
parser will pickup the "{{{somemacro}}}, b) when I want to use
subscript, I can write "a\sub{someword or two}" to make a subscript.

People like me don't use subscript much (to be honest, I've never
wanted to use subscript since I started use Org, seven years?), but
use '_' many times in a doc.  Hence, (setq org-use-sub-superscripts
nil) in my init.el.  But for someone wanting to use subscripts in rare
occasion, '\sub' would serve.

I can understand if you say it's not good idea to change the _syntax_
by a variable.

> > hmm.  just checked the source.  org-use-sub-superscripts is only for display.
> > org-export-with-sub-superscripts is just for exports.
>
> See above.
>
> > I was gonna just by-pass or disable subscript parser all together when
> > org-use-sub-superscripts is nil but it doesn't seems to be a good
> > idea, does it?
>
> This is exactly what a nil `org-export-with-sub-superscripts' does.

If that's the case, why doesn't macro parser pickup the
"{{{timestamp}}}" in "a_{{{timestamp}}}"?

Maybe this is my confusing point?  The component resposible defining
the org "syntax" is different from the parser?  Even if sub/super
parser is by-passed, because the "syntax" is subscript, the macro
parser won't be called on this substring?

Is the syntax defined in org-element--object-regexp and
org-element--object-lex?

Thanks,
--
           yashi

Re: limitation for macro expansion

[Nicolas Goaziou]

Hello,

Yasushi SHOJI <yashi@atmark-techno.com> writes:

> Are you saying that it's possible to generate "a_20170307.txt" from
> "a_{{{timestamp}}}.txt" if I set those variables mentioned above
> correctly? without using '\under'?

Not at all. Sorry about the confusion.

The mentioned variables allow writing, e.g., a_b without worrying about
any subscript, if you don't use it.

> I know you said it's ambiguous.  So let me clarify what I said. What I
> said above was to change the syntax (or lex / parser's behavior) when
> some variable, let's say 'org-do-not-parse-sub-superscripts', is set
> to nil.
>
> So that a) org leaves '_' in "a_{{{somemacro}}}" as-is and the macro
> parser will pickup the "{{{somemacro}}}, b) when I want to use
> subscript, I can write "a\sub{someword or two}" to make a subscript.
>
> People like me don't use subscript much (to be honest, I've never
> wanted to use subscript since I started use Org, seven years?), but
> use '_' many times in a doc.  Hence, (setq org-use-sub-superscripts
> nil) in my init.el.  But for someone wanting to use subscripts in rare
> occasion, '\sub' would serve.
>
> I can understand if you say it's not good idea to change the _syntax_
> by a variable.

Actually, I strictly opposed to any variable controlling syntax. So the
above is not an option.

>> This is exactly what a nil `org-export-with-sub-superscripts' does.
>
> If that's the case, why doesn't macro parser pickup the
> "{{{timestamp}}}" in "a_{{{timestamp}}}"?

My bad. My sentence is inaccurate: it should be "more or less" instead
of "exactly".

Actually `org-export-with-sub-superscripts' doesn't alter the syntax,
and doesn't mess with parsers. The sub/superscript is parsed the usual
way, but then is turned back into plain Org syntax, i.e., a string, in
the parse tree. See `org-export--remove-uninterpreted-data' for details.

In most cases, the result is the same as if the sub/superscript never
had been parsed. Unfortunately, in this case, it differs. This variable
triggers way after macros have been expanded so it cannot help with
macros.

> Maybe this is my confusing point?  The component resposible defining
> the org "syntax" is different from the parser?

No, it isn't.

> Even if sub/super parser is by-passed, because the "syntax" is
> subscript, the macro parser won't be called on this substring?

No parser is by-passed, but its work may be undone, per above. So,
indeed, the macro parser isn't called.

> Is the syntax defined in org-element--object-regexp and
> org-element--object-lex?

The latter controls precedence among parsed objects.

Anyway, I think a simple solution to your problem is to 

1. Write a function turning _{{{...}}} into _{{{{...}}}} and add it to
   `org-export-before-processing-hook'.

2. Depending on the back-end, write a function removing the spurious
   brackets. Note that I pushed a fix in maint that should limit them in
   ASCII back-end.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738