[HELP] Fwd: Org format as a new standard source format for GNU manuals

[HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Dear List,

I am forwarding an official email from RMS changing subject line to more
descriptive. See below.

For some context, in order to support specialized syntax for manuals, we
may first need to implement the discussed special blocks export and
inline special blocks:
1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/

The above links aim to introduce export functionality that we now have
for links to special blocks and new custom markup elements. I am
referring to
1. Ability to create new custom element types programmatically 
2. Ability to define how to :export the custom element types

Similar to `org-link-set-parameters'.

Patches and more concrete ideas are welcome!

-------------------- Start of forwarded message --------------------
From: Richard Stallman <rms@gnu.org>
To: Tim Cross <theophilusx@gmail.com>
Cc: emacs-devel@gnu.org
Subject: Re: Org mode and Emacs
Date: Mon, 26 Sep 2022 08:10:03 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Speaking as the Chief GNUisance, rssponsible for GNU Project
standards, I would be happy to adopt an upgraded Org format as a new
standard source format for GNU manuals, _provided_ Org format has been
extended with the capability to express all the constructions and
distinctions that Texinfo can express, generate all the output formats
Texinfo can generate, and use TeX to make beautiful printed output.

Texinfo can generate these output formats: Info files, HTML, ASCII
text, and DVI and PDF files via TeX.

Texinfo provides numerous subtle distinctions that show up clearly in
each of these output formats.  Compare, for example, @var, @dfn and
@emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I am sure people can extend Org software to handle these semantic
distinctions and generate these output formats.  Since it has been
done once, it can be done again.  But the work is not trivial.

The work has to start by designing what the extended Org format will look
like.  That part is the crucial part; once it has been specified,
people can work independently to implement various parts of handling
that format.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



-------------------- End of forwarded message --------------------

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Samuel Wales]

i really wish i could engage or even post drafts of extensible syntax
stuff that i had replied to many years ago and recently.

On 9/29/22, Ihor Radchenko <yantar92@gmail.com> wrote:
> Dear List,
>
> I am forwarding an official email from RMS changing subject line to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for manuals, we
> may first need to implement the discussed special blocks export and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> -------------------- Start of forwarded message --------------------
> From: Richard Stallman <rms@gnu.org>
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs
> Date: Mon, 26 Sep 2022 08:10:03 -0400
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a new
> standard source format for GNU manuals, _provided_ Org format has been
> extended with the capability to express all the constructions and
> distinctions that Texinfo can express, generate all the output formats
> Texinfo can generate, and use TeX to make beautiful printed output.
>
> Texinfo can generate these output formats: Info files, HTML, ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I am sure people can extend Org software to handle these semantic
> distinctions and generate these output formats.  Since it has been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format will look
> like.  That part is the crucial part; once it has been specified,
> people can work independently to implement various parts of handling
> that format.
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> -------------------- End of forwarded message --------------------
>
> --
> Ihor Radchenko,
> Org mode contributor,
> Learn more about Org mode at https://orgmode.org/.
> Support Org development at https://liberapay.com/org-mode,
> or support my work at https://liberapay.com/yantar92
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Thomas S. Dye]

Good news.  Thanks Ihor.  And thanks to Nicolas for his work on 
ox.

Ihor Radchenko <yantar92@gmail.com> writes:

> Dear List,
>
> I am forwarding an official email from RMS changing subject line 
> to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for 
> manuals, we
> may first need to implement the discussed special blocks export 
> and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we 
> now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically 
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> From: Richard Stallman <rms@gnu.org>
> Subject: Re: Org mode and Emacs
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 08:10:03 -0400 (3 days, 17 hours, 36 
> minutes ago)
> Flags: seen, list
> Maildir: /TSD-ONLINE/INBOX
> List: emacs-orgmode.gnu.org
>
> [[[ To any NSA and FBI agents reading my email: please consider 
> ]]]
> [[[ whether defending the US Constitution against all enemies, 
> ]]]
> [[[ foreign or domestic, requires you to follow Snowden's 
> example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a 
> new
> standard source format for GNU manuals, _provided_ Org format 
> has been
> extended with the capability to express all the constructions 
> and
> distinctions that Texinfo can express, generate all the output 
> formats
> Texinfo can generate, and use TeX to make beautiful printed 
> output.
>
> Texinfo can generate these output formats: Info files, HTML, 
> ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up 
> clearly in
> each of these output formats.  Compare, for example, @var, @dfn 
> and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and 
> @key.
>
> I am sure people can extend Org software to handle these 
> semantic
> distinctions and generate these output formats.  Since it has 
> been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format 
> will look
> like.  That part is the crucial part; once it has been 
> specified,
> people can work independently to implement various parts of 
> handling
> that format.
>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> ----------
All the best,
Tom

-- 
Thomas S. Dye
https://tsdye.online/tsdye

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Christopher M. Miles]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]


Glad to hear this news which Org mode is used in more places.

-- 

[ stardiviner ]
I try to make every word tell the meaning that I want to express without misunderstanding.

Blog: https://stardiviner.github.io/
IRC(libera.chat, freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 30/09/2022 10:31, Ihor Radchenko wrote:
> 
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I have not read the emacs-devel thread, so I may ask about something 
that already has been discussed.

Are there cases when texinfo may use nested formatting commands of the 
same type, something like @samp{a @code{b @samp{c} d} e}? My concern is 
that current org-element parser may be a blocker.

Another point is that most of the mentioned commands a close to 
verbatim, but Org has much more special characters recognized as markup 
and no markup is allowed inside Org verbatim snippets. Escaping (by zero 
width spaces?) of code and samples may be prohibitively inconvenient in 
Org if markup should be recognized inside.

One more point is external tools like pandoc export from Org to other 
formats. When Org extensions are implemented in elisp, such tools become 
hardly usable. Unsure if some kind of declarative style sheets will be 
enough.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Dear List,
>
> I am forwarding an official email from RMS changing subject line to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for manuals, we
> may first need to implement the discussed special blocks export and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically 
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> From: Richard Stallman <rms@gnu.org>
> Subject: Re: Org mode and Emacs
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 08:10:03 -0400 (4 days, 8 hours, 26 minutes ago)
> Flags: seen, list
> Maildir: /gmail/10 - Tech/software/org
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a new
> standard source format for GNU manuals, _provided_ Org format has been
> extended with the capability to express all the constructions and
> distinctions that Texinfo can express, generate all the output formats
> Texinfo can generate, and use TeX to make beautiful printed output.
>
> Texinfo can generate these output formats: Info files, HTML, ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I am sure people can extend Org software to handle these semantic
> distinctions and generate these output formats.  Since it has been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format will look
> like.  That part is the crucial part; once it has been specified,
> people can work independently to implement various parts of handling
> that format.
>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> ----------

I realise this will likely come across as another post from "Debbie
Downer", but I feel it is important to add a warning here and not get
too carried away with the excitement of seeing org mode accepted to the
point it becomes the official documentation format for Emacs. There are
some potential pitfalls here which need to be considered and which could
impact on how we satisfy the remaining 'blocker' to org mode taking on
this important role. A few questions we might want to consider....

What impact will adding all the additional formatting/markup primitives
have to the user experience?

One of the big benefits org has is simplicity in markup. This is one of
the driving themes in the 'markdown' movement. Will adding a lot of
additional syntax and markup tags add to cognitive load and complexity
and losing some of what makes org mode great to use. This could be one
of those situations where less is more.

Will adding a lot of additional markup entities have any impact on the
development of new and maintenance of existing export back ends? i

With all the additional entities, I suspect the demand for nesting of
entities will also increase. This has been an area org has struggled
with in the past. I suspect the big issue is that allowing nesting of
markup entities and maintaining simple syntax is very difficult.

Is there a risk of one aspect of org mode dominating all others and
potentially transforming it from a very flexible and general solution to
a technical documentation focus?

Texifno has a very specific focus. It aims to be an advanced formatting
system for writing software technical documents. As such, it is very
suitable for Emacs documentation. Org mode on the other hand is not a
documentation framework. While it does a fine job in this area, it is
not its prime focus. Org has many other unrelated roles, such as task
management, time tracking, simple spreadsheet and data
management/manipulation, literate programming and live document
generation, data capture etc etc. Will adopting org mode as the default
documentation format for Emacs run the risk of the technical
documentation aspects of org mode taking precedence over other aspects?
Will funcionality in different areas have to be modified in order to
better support technical documentation?

What is the underlying motivation for this very significant change? 

A big question which I've not seen answered is what is the motivation
for this very significant change? Are there problems with texinfo which
are driving this change? If so, are these problems ones we need to be
very careful not to import into org mode. When you look at it, texinfo
already exports to many different formats similar to what org mode
does. It is a system with a very specific and quite narrow focus -
software technical documentation and yet, its use sems to be
declining. If it wasn't the flagship for GNU documentation, would it
even still exist? So the question becomes, why is this system not more
popular within software documentation circles? If part of the reason is
to potentialy increase contributors, will there still be as many people
wanting to use org mode if the underlying syntax is extended and
modified to support all the main texinfo markup set?

What impact on maintenance and future development directions will
becoming the official documentation framework have for org mode?

Will this result in document formatting gaining additional focus over
other areas?  Will it result in interface changes which favor
documentation processes over other areas like babel, data capture etc?

I think it should also be noted that there are some core Emacs
developers who are not supportive of this change and who don't like or
use org mode. Despite what RMS states, this is not a sure thing. Once
org mode is able to meet all the stated requirements, there debate
regarding switching to use org mode will begin and I suspect it will be
pretty full on. We will need to have a very clear picture regarding what
our (the org mode community) motivation is here and know what we are
prepared to compromise and what is non-negotiable.

If we do decide to go down this road, one idea which I feel would be
worth exploring is the extent to which we could have these additional
markup elements as an optional component. In some ways, similar to org
export back ends. If we could add these additional makrujp elements as
an optional add on, perhaps we can maintain the markup simplicity and
simple syntax for those who don't need specialised markup and for when
we do, we could require the 'technical documentation' module? 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 30/09/2022 10:31, Ihor Radchenko wrote:
>> 
>> Texinfo provides numerous subtle distinctions that show up clearly in
>> each of these output formats.  Compare, for example, @var, @dfn and
>> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I have not read the emacs-devel thread, so I may ask about something 
> that already has been discussed.
>
> Are there cases when texinfo may use nested formatting commands of the 
> same type, something like @samp{a @code{b @samp{c} d} e}? My concern is 
> that current org-element parser may be a blocker.

AFAIK, the only nested construct in Texinfo spec is
@acronym{GNU, @acronym{GNU}'s Not Unix}
https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html#g_t_0040acronym

But it should not be a blocker. We can use two different elements for
this instead or employ the requirement of balanced parenthesis inside,
similar to what we do in the HEADERS for inline src blocks
https://orgmode.org/worg/dev/org-syntax.html#Source_Blocks

> Another point is that most of the mentioned commands a close to 
> verbatim, but Org has much more special characters recognized as markup 
> and no markup is allowed inside Org verbatim snippets. Escaping (by zero 
> width spaces?) of code and samples may be prohibitively inconvenient in 
> Org if markup should be recognized inside.

We need a new special object type for markup that does not suffer from
the limitations of our current single-char-style *markup* constructs.
(It is not even solely motivated by this request from RMS; we just need
something to allow whitespace in verbatim boundaries)

> One more point is external tools like pandoc export from Org to other 
> formats. When Org extensions are implemented in elisp, such tools become 
> hardly usable. Unsure if some kind of declarative style sheets will be 
> enough.

We already have Org extensions for links. I envision supporting GNU
manuals in a similar fashion. Only a subset of universally useful
extensions will become the actual part of Org syntax spec. At the end,
Org should remain generic export format without going too far into
manual-specific requirements. Specific exports should be supported
optionally, similar to our ol-*.el libraries.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> What impact will adding all the additional formatting/markup primitives
> have to the user experience?

From my perspective, the first and main goal related to this request is
developing the minimally (!) necessary Org format extensions to allow
Texinfo constructs.

I do not think that we need to add all the variety of Texinfo-specific
constructs to the Org specification. Instead, we should add generic
configurable syntax elements to Org. (The Texinfo-specific part will
come as a separate library, similar to ol-*.el)

In particular, I suggest to (1) extend the existing special block elements
with Elisp-configurable :export settings; (2) add special block
equivalent object.

The latter will also serve as an alternative to our current awkward
approach to escaping the inline markup (bold, italics, verbatim, etc).
Especially wrt requirements to the spaces around the markup symbols.

For user experience, the new markup primitives will be unnecessary,
except specific export requirements or edge cases with our current
markup (e.g. spaces at the beginning/end of verbatim/code).

> One of the big benefits org has is simplicity in markup. This is one of
> the driving themes in the 'markdown' movement. Will adding a lot of
> additional syntax and markup tags add to cognitive load and complexity
> and losing some of what makes org mode great to use. This could be one
> of those situations where less is more.

I agree that adding yet another markup tagging will be an extra
cognitive burden. But I envision it to be used only when necessary. If
our current markup is sufficient for the user needs, there will be no
need to go into the more verbose tagging. But if there are special needs
(like exporting manual), there is no way to maintain Org simplicity yet
allowing the complex @dfn/@code/whatnot distinctions.

Also, see https://yhetil.org/emacs-devel/87v8t3wfgd.fsf@localhost

> Will adding a lot of additional markup entities have any impact on the
> development of new and maintenance of existing export back ends? i

I only suggest adding a single new markup object entity, similar to
special block elements. Just like #+BEGIN_FOO can spawn infinite number
of special block types, the new markup object will allow arbitrary new
markup entities loaded as necessary by the corresponding export
backends/libraries.

> With all the additional entities, I suspect the demand for nesting of
> entities will also increase. This has been an area org has struggled
> with in the past. I suspect the big issue is that allowing nesting of
> markup entities and maintaining simple syntax is very difficult.

Agree. However, it is the demand for nested constructs being one of the
motivators why we should have a new markup object syntax that will allow
nesting when absolutely necessary and when the existing simpler Org
markup does not cut it.

> Is there a risk of one aspect of org mode dominating all others and
> potentially transforming it from a very flexible and general solution to
> a technical documentation focus?

I do not think so. Org functionality only depends on the community
interest and contributions. I see no reason why technical documentation
is going to prevail over other Org uses.

> Texifno has a very specific focus. It aims to be an advanced formatting
> system for writing software technical documents. As such, it is very
> suitable for Emacs documentation. Org mode on the other hand is not a
> documentation framework. While it does a fine job in this area, it is
> not its prime focus. Org has many other unrelated roles, such as task
> management, time tracking, simple spreadsheet and data
> management/manipulation, literate programming and live document
> generation, data capture etc etc. Will adopting org mode as the default
> documentation format for Emacs run the risk of the technical
> documentation aspects of org mode taking precedence over other aspects?
> Will funcionality in different areas have to be modified in order to
> better support technical documentation?

> What impact on maintenance and future development directions will
> becoming the official documentation framework have for org mode?
>
> Will this result in document formatting gaining additional focus over
> other areas?  Will it result in interface changes which favor
> documentation processes over other areas like babel, data capture etc?


I do not think so. We are certainly not going to remove features from
Org just because they will improve support for technical documentation.

I mean, I do not really see why this should be needed. But if it is
needed, then we will not go this road and simply abandon the
documentation idea.
https://bzg.fr/en/the-software-maintainers-pledge/

> What is the underlying motivation for this very significant change? 
>
> A big question which I've not seen answered is what is the motivation
> for this very significant change? Are there problems with texinfo which
> are driving this change? If so, are these problems ones we need to be
> very careful not to import into org mode. When you look at it, texinfo
> already exports to many different formats similar to what org mode
> does. It is a system with a very specific and quite narrow focus -
> software technical documentation and yet, its use sems to be
> declining. If it wasn't the flagship for GNU documentation, would it
> even still exist? So the question becomes, why is this system not more
> popular within software documentation circles? If part of the reason is
> to potentialy increase contributors, will there still be as many people
> wanting to use org mode if the underlying syntax is extended and
> modified to support all the main texinfo markup set?

This is a good question. I'd like to hear from RMS or other people
familiar with Texinfo issues.

As a starter, there was a bit of related discussion in emacs-devel:
- RMS mentioned that one big gotcha with Texinfo is that it inherits
  from plain TeX. Supporting LaTeX features would require a major
  re-design. Org supports LaTeX out of the box.
  https://yhetil.org/emacs-devel/E1o0WEF-0004wJ-Iw@fencepost.gnu.org
- Also, from Christopher Dimech:
  "Because texinfo is just plain tex, and as consequence remained
   limited (e.g. no colouring, no bold mathematical expressions' no
   boxes for formal end of proof).
   https://yhetil.org/emacs-devel/trinity-5129f2bc-bc9d-4f40-a099-ba1c913178fe-1655084866514@3c-app-mailcom-bs16

> I think it should also be noted that there are some core Emacs
> developers who are not supportive of this change and who don't like or
> use org mode. Despite what RMS states, this is not a sure thing. Once
> org mode is able to meet all the stated requirements, there debate
> regarding switching to use org mode will begin and I suspect it will be
> pretty full on. We will need to have a very clear picture regarding what
> our (the org mode community) motivation is here and know what we are
> prepared to compromise and what is non-negotiable.
>
> If we do decide to go down this road, one idea which I feel would be
> worth exploring is the extent to which we could have these additional
> markup elements as an optional component. In some ways, similar to org
> export back ends. If we could add these additional makrujp elements as
> an optional add on, perhaps we can maintain the markup simplicity and
> simple syntax for those who don't need specialised markup and for when
> we do, we could require the 'technical documentation' module? 

Agree. Let's not go too far yet and focus on extending special blocks
and the inline special block element I propose. These topics (especially
for markup element with less edge cases) pop up on the Org ML from time
to time and worth looking into regardless whether Org is going to be
used as technical documentation format.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Agree. Let's not go too far yet and focus on extending special blocks
> and the inline special block element I propose. These topics (especially
> for markup element with less edge cases) pop up on the Org ML from time
> to time and worth looking into regardless whether Org is going to be
> used as technical documentation format.

+1

I'm confident your across the concerns I raised and if I understand your
proposal correctly, adding support for the markup entities RMS
requested should be possible without any significant adverse impact on
existing usability. Greater clarity regarding the underlying drive for
changing from texinfo for Emacs would be good, but that is essentially
and emacs devel question and as you point out, ability to support the
additional elements identified by RMS would potentially address other
issues, such as spaces, quoting and nesting within existing markup.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 10:30, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Another point is that most of the mentioned commands a close to
>> verbatim, but Org has much more special characters recognized as markup
>> and no markup is allowed inside Org verbatim snippets. Escaping (by zero
>> width spaces?) of code and samples may be prohibitively inconvenient in
>> Org if markup should be recognized inside.
> 
> We need a new special object type for markup that does not suffer from
> the limitations of our current single-char-style *markup* constructs.
> (It is not even solely motivated by this request from RMS; we just need
> something to allow whitespace in verbatim boundaries)

I do not remember if the following idea has been discussed. What about 
extending source blocks and inline source snippets to allow :results ast 
header argument that caused executing during export only and expects 
s-expression with AST branch that is included into parsed tree without 
intermediate Org markup representation? Then a special org-babel backend 
may be created to support new markup type. I am unsure however at which 
stage of export source blocks are executed, maybe too early to implement 
the idea. The advantage is that no extension of Org syntax is necessary.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> I do not remember if the following idea has been discussed. What about 
> extending source blocks and inline source snippets to allow :results ast 
> header argument that caused executing during export only and expects 
> s-expression with AST branch that is included into parsed tree without 
> intermediate Org markup representation? Then a special org-babel backend 
> may be created to support new markup type. I am unsure however at which 
> stage of export source blocks are executed, maybe too early to implement 
> the idea. The advantage is that no extension of Org syntax is necessary.

This will not help with new extendable markup elements. No new org
element types can be created this way. We still need generalized inline
markup object.

This can help with escaping syntax and spaces in verbatim.

I do recall you mentioned this idea in one of the earlier threads.
https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
Max Nikulin <manikulin@gmail.com> (2021-12-06)
Subject: Raw Org AST snippets for "impossible" markup

This idea is ok by itself, although we should not use the real AST;
rather some AST-like representation that can be fixed and then
translated to real AST. (I'd rather keep the possibility to change our
real AST; for example, I have plans to change property storage into
vector for faster access---current property list is slower than querying
the binary tree with all elements in buffer).

The downside of this idea is that it will force users into the Elisp
escape hell src_elisp(verbatim "this is \\-n just slash-n and we still
cannot use unmatched } here"}.

I'd rather have something more friendly to users, if possible.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 18:01, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> I do not remember if the following idea has been discussed. What about
>> extending source blocks and inline source snippets to allow :results ast
>> header argument that caused executing during export only and expects
>> s-expression with AST branch that is included into parsed tree without
>> intermediate Org markup representation? Then a special org-babel backend
>> may be created to support new markup type. I am unsure however at which
>> stage of export source blocks are executed, maybe too early to implement
>> the idea. The advantage is that no extension of Org syntax is necessary.
> 
> This will not help with new extendable markup elements. No new org
> element types can be created this way. We still need generalized inline
> markup object.
> 
> This can help with escaping syntax and spaces in verbatim.

It should be enough to create nested "code" or "verbatim" inline objects 
with some attribute like :class file, :class var, :class dfn, etc. 
Export backend may interpret this attribute to create proper texinfo 
commands or more precisely choose HTML element.

> I do recall you mentioned this idea in one of the earlier threads.
> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
> Max Nikulin (2021-12-06)
> Subject: Raw Org AST snippets for "impossible" markup

It was using links, not source blocks. Typing AST directly is a kind of 
last resort. Now I am writing about some higher level markup with a 
parser that allows nested element and uses just a few special characters 
to denote commands. It is a kind of trade-off between brevity of 
lightweight markup and flexibility of more verbose language with ability 
to more precisely express intentions.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 11:08, Ihor Radchenko wrote:
> Tim Cross writes:
> 
> I do not think that we need to add all the variety of Texinfo-specific
> constructs to the Org specification. Instead, we should add generic
> configurable syntax elements to Org. (The Texinfo-specific part will
> come as a separate library, similar to ol-*.el)
> 
> In particular, I suggest to (1) extend the existing special block elements
> with Elisp-configurable :export settings; (2) add special block
> equivalent object.

While I do not mind to have flexible generic inline objects, I feel some 
doubts.

Export is already customizable through creation of derived backend. For 
links :export property is merely an alternative way supposed to be more 
convenient. In some sense it is a way to dispatch proper handler, a kind 
of virtual function table, etc. I see a couple of limitations with link 
export implementation.

1. Interface is rather different from the derived backend property. 
Instead of org-element object only selected properties (and backend 
communication channel is available).

2. Unsure if there is a robust way to extend implementation of the 
backend handler without replacing it completely. I mean a function that 
modifies or sets some attributes and delegate real export to the default 
handler.

Mentioned in this thread texinfo commands are not convincing reason for 
special blocks from my point of view. They are different flavors of 
verbatim (or code) object. If they are verbatim objects with some 
additional property then they may be just exported by a backend that is 
unaware of their kinds. It can be considered as graceful degradation. 
For special blocks export handlers must be implemented for each backend 
unless type hierarchy is someway declared.

Earlier we discussed assigning attributes for inline objects. While 
nesting is not involved, it may be a way to implement necessary texinfo 
commands as verbatim with class or type attribute. I am unsure if 
different types of special blocks is the best way to resolve nesting 
problem. Verbatim custom objects require different rules of parsing.

Actually I simplified things when wrote that a backend may be unaware of 
verbatim type. When nesting is involved it should be ready at least to 
nested verbatim object. E.g. markdown backend can not blindly wrap text 
into backticks, it has to check if parent element is not a verbatim 
snippet or a verbatim block.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> This can help with escaping syntax and spaces in verbatim.
>
> It should be enough to create nested "code" or "verbatim" inline objects 
> with some attribute like :class file, :class var, :class dfn, etc. 
> Export backend may interpret this attribute to create proper texinfo 
> commands or more precisely choose HTML element.
>
>> I do recall you mentioned this idea in one of the earlier threads.
>> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
>> Max Nikulin (2021-12-06)
>> Subject: Raw Org AST snippets for "impossible" markup
>
> It was using links, not source blocks. Typing AST directly is a kind of 
> last resort. Now I am writing about some higher level markup with a 
> parser that allows nested element and uses just a few special characters 
> to denote commands. It is a kind of trade-off between brevity of 
> lightweight markup and flexibility of more verbose language with ability 
> to more precisely express intentions.

I am not sure if I understand the details of what you are referring to.

Can you please illustrate how to use the described AST markup for the
following Texinfo snippet:

    By convention, the dynamic library for @var{language} is
    @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
    system-specific extension for dynamic libraries.


-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Fraga, Eric]

Hello all,

I like the idea of inline special blocks.  Given that we have recently
introduced [cite:...] as new syntax, could we generalise this and allow
any xxx in [xxx:...]?

With this, the example Max gave from texinfo:

>     By convention, the dynamic library for @var{language} is
>     @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>     system-specific extension for dynamic libraries.

could look like

   By convention, the dynamic library for [var:language] is
   [code:libtree-{sitter}-[var:"language"].[var:ext]], where [var:ext]
   is the system-specific extension for dynamic libraries.

noting the recursive embedded syntax.  (and not knowing texinfo, I've
assumed that @{ and @} are escapes for the braces but could be something
else.)

Obviously, this would be a breaking change for any documents that
actually had anything along the lines of [xxx:...] in their text.

Just musing out loud. ;-) And procrastinating from preparing my lectures
for tomorrow... Feel free to ignore of course.

eric
-- 
: Eric S Fraga, with org release_9.5.5-851-ge9781f in Emacs 29.0.50

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> I like the idea of inline special blocks.  Given that we have recently
> introduced [cite:...] as new syntax, could we generalise this and allow
> any xxx in [xxx:...]?

This syntax will make it difficult to pass optional arguments, like in
@abbr{FSF, Free Software Foundation}. Also, escaping "]" will be tricky.

> With this, the example Max gave from texinfo:
>
>>     By convention, the dynamic library for @var{language} is
>>     @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>>     system-specific extension for dynamic libraries.
>
> could look like
>
>    By convention, the dynamic library for [var:language] is
>    [code:libtree-{sitter}-[var:"language"].[var:ext]], where [var:ext]
>    is the system-specific extension for dynamic libraries.

I am thinking about something like

_name{<contents>}
_name{{<contents>}}  <--- extra {} is added as needed for escaping
_name[:key value ...]{<contents>}

The syntax idea is to follow the relevance between [[links]] and [cite:citations]
but here we have src_name[...]{...} and _name[...]{<contents>} instead.

    By convention, the dynamic library for _var{language} is
    _code{{libtree-{sitter}-_var{"language"}._var{ext}}}, where _var{ext}
    is the system-specific extension for dynamic libraries.

We may even follow Max's idea about AST and make it so that
_bold{contents} will be parsed just like *contents*.

> noting the recursive embedded syntax.  (and not knowing texinfo, I've
> assumed that @{ and @} are escapes for the braces but could be something
> else.)

Yep, @{ and @} are the escaped { and } in Texinfo.

-- 
Ihor Radchenko aka yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Fraga, Eric]

On Sunday,  2 Oct 2022 at 21:02, Ihor Radchenko wrote:
>
> _name{<contents>}
> _name{{<contents>}}  <--- extra {} is added as needed for escaping
> _name[:key value ...]{<contents>}

I'd be okay with this as well.


-- 
: Eric S Fraga, with org release_9.5.5-851-ge9781f in Emacs 29.0.50

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Juan Manuel Macías]

Ihor Radchenko writes:

> _name{<contents>}
> _name{{<contents>}}  <--- extra {} is added as needed for escaping
> _name[:key value ...]{<contents>}
>
> The syntax idea is to follow the relevance between [[links]] and [cite:citations]
> but here we have src_name[...]{...} and _name[...]{<contents>} instead.

And how about this:

%_name{<contents>}
%_name{{<contents>}}  <--- extra {} is added as needed for escaping
%_name[:key value ...]{<contents>}

or any other symbol instead of "%" ?

N.B.: I must admit this is more for an "aesthetic" reason. Although
perhaps it can be useful to search in the documents.

Best regards,

Juan Manuel

--
--
------------------------------------------------------
Juan Manuel Macías

https://juanmanuelmacias.com

https://lunotipia.juanmanuelmacias.com

https://gnutas.juanmanuelmacias.com

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 02/10/2022 11:59, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>> This can help with escaping syntax and spaces in verbatim.
>>
>> It should be enough to create nested "code" or "verbatim" inline objects
>> with some attribute like :class file, :class var, :class dfn, etc.
>> Export backend may interpret this attribute to create proper texinfo
>> commands or more precisely choose HTML element.
>>
>>> I do recall you mentioned this idea in one of the earlier threads.
>>> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
>>> Max Nikulin (2021-12-06)
>>> Subject: Raw Org AST snippets for "impossible" markup
>>
>> It was using links, not source blocks. Typing AST directly is a kind of
>> last resort. Now I am writing about some higher level markup with a
>> parser that allows nested element and uses just a few special characters
>> to denote commands. It is a kind of trade-off between brevity of
>> lightweight markup and flexibility of more verbose language with ability
>> to more precisely express intentions.
> 
> I am not sure if I understand the details of what you are referring to.
> 
> Can you please illustrate how to use the described AST markup for the
> following Texinfo snippet:
> 
>      By convention, the dynamic library for @var{language} is
>      @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>      system-specific extension for dynamic libraries.

If you are asking how to represent such construct without introducing 
custom elements then (it may be e.g. :type, not :class) parsed AST 
should be like

     (code nil ("libtree-{sitter}-"
                (code (:class var) "\"language\"")
                "."
                (code (:class var) "ext")))

If there was some syntax for object attributes then simple cases would 
be like

     [[attr:(:class var)]]~language~

I have no idea concerning particular markup that can be used inside 
source blocks. It might be LaTeX-like commands as discussed in the 
sibling subthread or HTML (XML) based syntax that is more verbose than 
TeX-like notation.

     By convention, the dynamic library
     for src_alt{\code[class=var]{language}} is
 
src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
     where src_alt{\code[class=var]{ext}} is the
     system-specific extension for dynamic libraries.

or

     By convention, the dynamic library for
     src_alt{<code class="var">language</code>} is
     src_alt{<code>libtree-{sitter}-<code 
class="var">"language"</code>.<code class="var">ext</code></code>},
     where src_alt{<code class="var">ext</code>} is the
     system-specific extension for dynamic libraries.

Hypothetical "alt" babel language has default :results ast :export 
results header arguments to inject AST bypassing Org markup stage.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 01/10/2022 11:08, Ihor Radchenko wrote:
>> In particular, I suggest to (1) extend the existing special block elements
>> with Elisp-configurable :export settings; (2) add special block
>> equivalent object.
>
> While I do not mind to have flexible generic inline objects, I feel some 
> doubts.
>
> Export is already customizable through creation of derived backend. For 
> links :export property is merely an alternative way supposed to be more 
> convenient. In some sense it is a way to dispatch proper handler, a kind 
> of virtual function table, etc. I see a couple of limitations with link 
> export implementation.

Creating derived backend will force users to use that non-standard
backend for export - inconvenient. Especially for third-party backends.

:export property, among other things, can also provide a reasonable
fallback for arbitrary backend not considered in advance.

> 1. Interface is rather different from the derived backend property. 
> Instead of org-element object only selected properties (and backend 
> communication channel is available).

Is it? :export function for links is taking similar parameters with the
other export transcoders.

> 2. Unsure if there is a robust way to extend implementation of the 
> backend handler without replacing it completely. I mean a function that 
> modifies or sets some attributes and delegate real export to the default 
> handler.

We may provide something like :export-filter-object that will act
similar to `:filter-parse-tree' and replace the original link object
with arbitrary Org AST.

> Mentioned in this thread texinfo commands are not convincing reason for 
> special blocks from my point of view. They are different flavors of 
> verbatim (or code) object. If they are verbatim objects with some 
> additional property then they may be just exported by a backend that is 
> unaware of their kinds. It can be considered as graceful degradation. 
> For special blocks export handlers must be implemented for each backend 
> unless type hierarchy is someway declared.

No. There is no need to consider every possible backend. There could be
an export handler that will provide a fallback for unknown backend, if
needed.

> Earlier we discussed assigning attributes for inline objects. While 
> nesting is not involved, it may be a way to implement necessary texinfo 
> commands as verbatim with class or type attribute. I am unsure if 
> different types of special blocks is the best way to resolve nesting 
> problem. Verbatim custom objects require different rules of parsing.

Please do remember that texinfo commands, are _not_ verbatim. They can
contain other markup inside. I'd rather look at them as extended
emphasis. Their contents must be parsed as well.

> Actually I simplified things when wrote that a backend may be unaware of 
> verbatim type. When nesting is involved it should be ready at least to 
> nested verbatim object. E.g. markdown backend can not blindly wrap text 
> into backticks, it has to check if parent element is not a verbatim 
> snippet or a verbatim block.

Agree. See export filter idea.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> And how about this:
>
> %_name{<contents>}
> %_name{{<contents>}}  <--- extra {} is added as needed for escaping
> %_name[:key value ...]{<contents>}
>
> or any other symbol instead of "%" ?
>
> N.B.: I must admit this is more for an "aesthetic" reason. Although
> perhaps it can be useful to search in the documents.

I have to admit that I am not a big fan of underscore in _name myself.
Just wanted to keep some resemblance of src_name{<contents>} yet not
making things too verbose (like block_name{<contents>}).

If I were to choose an alternative symbol other than "_", I'd choose
"@":

@name{<contents>}
@name{{<contents}}
@name[:key value ...]{<contents>}

1. It is similar to Texinfo
2. It does not clash with TeX
3. We already use @ in the inline export snippets.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 02/10/2022 11:59, Ihor Radchenko wrote:
>> Can you please illustrate how to use the described AST markup for the
>> following Texinfo snippet:
>> 
>>      By convention, the dynamic library for @var{language} is
>>      @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>>      system-specific extension for dynamic libraries.
>
> If you are asking how to represent such construct without introducing 
> custom elements then (it may be e.g. :type, not :class) parsed AST 
> should be like
>
>      (code nil ("libtree-{sitter}-"
>                 (code (:class var) "\"language\"")
>                 "."
>                 (code (:class var) "ext")))

This is not much different from @name[nil]{<contents>} idea, but
more verbose.

Also, more importantly, I strongly dislike the need to wrap the text
into "". You will have to escape \". And it will force third-party
parsers to re-implement Elisp sexp reader.

> If there was some syntax for object attributes then simple cases would 
> be like
>
>      [[attr:(:class var)]]~language~

I do not like this idea. It will require non-trivial changes in Org
parser and fontification.

Using dedicated object properties or at least inheriting properties from
:parent is the style we employ more commonly across the code:

@var{language}
or
@code[:class var]{language}
or
@attr[:class var]{~language~}

> I have no idea concerning particular markup that can be used inside 
> source blocks. It might be LaTeX-like commands as discussed in the 
> sibling subthread or HTML (XML) based syntax that is more verbose than 
> TeX-like notation.
>
>      By convention, the dynamic library
>      for src_alt{\code[class=var]{language}} is
>  
> src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
>      where src_alt{\code[class=var]{ext}} is the
>      system-specific extension for dynamic libraries.

I am against the idea of LaTeX-like commands. It will clash with
latex-fragment object type.
https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Fragments

> or
>
>      By convention, the dynamic library for
>      src_alt{<code class="var">language</code>} is
>      src_alt{<code>libtree-{sitter}-<code 
> class="var">"language"</code>.<code class="var">ext</code></code>},
>      where src_alt{<code class="var">ext</code>} is the
>      system-specific extension for dynamic libraries.

This style will indeed make things easier for the parser. But I find it
too verbose for practical usage. This is why I instead proposed the idea
with variable number of brackets: @code{{can have } inside}}.

> Hypothetical "alt" babel language has default :results ast :export 
> results header arguments to inject AST bypassing Org markup stage.

The problem with src block emitting AST is clashing with the way src
blocks work during export. What `org-export-as' does is replacing/adding
src block output into the actual Org buffer text before the parsing is
done.

Handling direct AST sexps will require a rewrite on how babel
integration with export works.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

It seems I completely failed trying to express my idea.

Instead of extending Org grammar (syntax), I suggest to change behavior 
of source blocks during export. In addition to current :results options, 
"ast" may be added. Its effect is that instead of adding text to export 
buffer that is parsed as Org markup, it causes insertion of a branch of 
syntax tree into original parse results. I admit, during export it may 
be necessary to iterate over source blocks one more time at a later stage.

Such source blocks should return "Org Syntax Tree", a simplified variant 
of org-element. It allows to change implementation details and e.g. to 
use vectors instead of lists for attributes in org-element. A converter 
from Org Syntax Tree to org-element should be implemented.

Certainly such format may be used directly as src_ost{(code (:class var) 
"language")} inline snippets or as

#+begin_src ost
   (code nil ("libtree-{sitter}-"
              (code (:class var) "\"language\"")
              "."
              (code (:class var) "ext")))
#+end_src

block-level elements. However I expect that it is the last resort option 
when there is no way to express desired construct in some other way.

I think, more convenient org-babel backends may be created to parse 
TeX-like (texinfo-like) or SGML-like (XML-like) syntax into Org Syntax 
Tree hierarchy. The essential idea is that outside of source blocks 
usual lightweight markup is used. Source blocks however have just a few 
special characters ([\{}], [@{}], or [&<>], etc.) to reduce issues with 
escaping for regular text or verbatim-like commands.

Some comments are inline.

On 03/10/2022 11:36, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> On 02/10/2022 11:59, Ihor Radchenko wrote:
>>
>> If you are asking how to represent such construct without introducing
>> custom elements then (it may be e.g. :type, not :class) parsed AST
>> should be like
>>
>>       (code nil ("libtree-{sitter}-"
>>                  (code (:class var) "\"language\"")
>>                  "."
>>                  (code (:class var) "ext")))
> 
> This is not much different from @name[nil]{<contents>} idea, but
> more verbose.
>
 > Also, more importantly, I strongly dislike the need to wrap the text
 > into "". You will have to escape \". And it will force third-party
 > parsers to re-implement Elisp sexp reader.

By this example I was trying to show how to express @var, @samp, @file 
without introducing of new custom objects. I do not see any problem with 
verbosity of such format, it may be used for really special cases only, 
while some more convenient markup is used for more simple cases.

>> If there was some syntax for object attributes then simple cases would
>> be like
>>
>>       [[attr:(:class var)]]~language~
> 
> I do not like this idea. It will require non-trivial changes in Org
> parser and fontification.
> 
> Using dedicated object properties or at least inheriting properties from
> :parent is the style we employ more commonly across the code:
> 
> @var{language}
> or
> @code[:class var]{language}
> or
> @attr[:class var]{~language~}

I do not mind to have some "span" object to assign attributes to its 
direct children. I used link-like prefix object just because a proof of 
concept may be tried with no changes in Org. It does not require support 
of nested objects. There is no existing syntax for such "span" objects, 
but perhaps it is not necessary and source blocks should be used instead 
for special needs.

>> I have no idea concerning particular markup that can be used inside
>> source blocks. It might be LaTeX-like commands as discussed in the
>> sibling subthread or HTML (XML) based syntax that is more verbose than
>> TeX-like notation.
>>
>>       By convention, the dynamic library
>>       for src_alt{\code[class=var]{language}} is
>>   
>> src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
>>       where src_alt{\code[class=var]{ext}} is the
>>       system-specific extension for dynamic libraries.
> 
> I am against the idea of LaTeX-like commands. It will clash with
> latex-fragment object type.
> https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Fragments
> 
>> or
>>
>>       By convention, the dynamic library for
>>       src_alt{<code class="var">language</code>} is
>>       src_alt{<code>libtree-{sitter}-<code
>> class="var">"language"</code>.<code class="var">ext</code></code>},
>>       where src_alt{<code class="var">ext</code>} is the
>>       system-specific extension for dynamic libraries.
> 
> This style will indeed make things easier for the parser. But I find it
> too verbose for practical usage. This is why I instead proposed the idea
> with variable number of brackets: @code{{can have } inside}}.

Texinfo is TeX with \ replaced by @. Just another character has the 
category starting command. The important point is that while Org markup 
uses a lot of special characters (*/_+[]...) this flexible markup should 
use just a few ones. I do not see any obstacles to try texinfo-like 
markup. Source blocks allow to have several languages.

>> Hypothetical "alt" babel language has default :results ast :export
>> results header arguments to inject AST bypassing Org markup stage.
> 
> The problem with src block emitting AST is clashing with the way src
> blocks work during export. What `org-export-as' does is replacing/adding
> src block output into the actual Org buffer text before the parsing is
> done.
> 
> Handling direct AST sexps will require a rewrite on how babel
> integration with export works.

Yes, it will. I am evaluating feasibility of such change instead of 
extending of Org syntax for custom elements.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Juan Manuel Macías]

Ihor Radchenko writes:

> If I were to choose an alternative symbol other than "_", I'd choose
> "@":
>
> @name{<contents>}
> @name{{<contents}}
> @name[:key value ...]{<contents>}
>
> 1. It is similar to Texinfo
> 2. It does not clash with TeX
> 3. We already use @ in the inline export snippets.

I like the "@" alternative a lot. And I agree with all three points. It is
also compact without losing clarity, and does not give the feeling of a
blank space before, as in the case of "_".

Best regards,

Juan Manuel 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Rick Lupton]

Hi Ihor and all,

I wonder if you have seen Pollen’s approach to this? https://docs.racket-lang.org/pollen/pollen-command-syntax.html

There are two separate ideas used there which seemed related to this discussion. I’m not sure if they are useful in the org context.  

1. The use of a special character (◊ by default) which introduces a command/inline special block. I don’t know if this would be worth considering as an alternative to @ (which also seems reasonable) to avoid ambiguity with other syntax. As the link above discusses it’s harder to type but there are solutions. 

2. Making it easy to define custom functions that produce org syntax. A bit like perhaps Max's suggestion to use source blocks, but instead of writing AST-like structure directly in the document where you want it, you can call a previously defined function to build it. This is similar to org macros but I’m not sure if they are so flexible as a lisp function. There is also the option to choose between passing arguments as lisp (in [ ]) or as markup (in { })

On Tue, 4 Oct 2022, at 9:28 PM, Juan Manuel Macías wrote:
> Ihor Radchenko writes:
>
>> If I were to choose an alternative symbol other than "_", I'd choose
>> "@":
>>
>> @name{<contents>}
>> @name{{<contents}}
>> @name[:key value ...]{<contents>}
>>
>> 1. It is similar to Texinfo
>> 2. It does not clash with TeX
>> 3. We already use @ in the inline export snippets.
>
> I like the "@" alternative a lot. And I agree with all three points. It is
> also compact without losing clarity, and does not give the feeling of a
> blank space before, as in the case of "_".
>
> Best regards,
>
> Juan Manuel

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

"Rick Lupton" <rclsub@fastmail.com> writes:

> I wonder if you have seen Pollen’s approach to this? https://docs.racket-lang.org/pollen/pollen-command-syntax.html

Thanks for the interesting reference!
After skimming through the syntax described in the link, I feel like
most of the features are already present in Org syntax or being
discussed here.

> There are two separate ideas used there which seemed related to this discussion. I’m not sure if they are useful in the org context.  
>
> 1. The use of a special character (◊ by default) which introduces a command/inline special block. I don’t know if this would be worth considering as an alternative to @ (which also seems reasonable) to avoid ambiguity with other syntax. As the link above discusses it’s harder to type but there are solutions. 

I do understand the idea behind ◊, but it is the design decision we
would need to make before creating Org syntax. At this point, we are not
going to change the Org syntax concepts just for having the nice
feature. There is no point to solve the escaping problem just for a
single new Org syntax element and not changing the rest of the syntax. 

So, I'd prefer to keep "@" or other symbol we already use in the
existing Org elements/objects.

For escaping @ inside, we already have zero-width, an alternative idea
with special \-- entity, and we can introduce \atsymbol entity that
exports and displays as "@".

> 2. Making it easy to define custom functions that produce org syntax. A bit like perhaps Max's suggestion to use source blocks, but instead of writing AST-like structure directly in the document where you want it, you can call a previously defined function to build it. This is similar to org macros but I’m not sure if they are so flexible as a lisp function. There is also the option to choose between passing arguments as lisp (in [ ]) or as markup (in { })

We already have named code blocks to call arbitrary code in arbitrary
programming language and produce arbitrary results, including Org
markup. On export. I do not think that we should allow runtime markup
calculations on-the-fly due to performance reasons.

Also, Org macros do allow arbitrary elisp:

       As a special case, Org parses any replacement text starting with
    ‘(eval’ as an Emacs Lisp expression and evaluates it accordingly.
    Within such templates, arguments become strings.  Thus, the following
    macro


         #+MACRO: gnustamp (eval (concat "GNU/" (capitalize $1)))
    
    turns ‘{{{gnustamp(linux)}}}’ into ‘GNU/Linux’ during export.

Max's idea about AST structure is just an extra fallback to produce
really tricky cases. And since it is going to be an src block result,
all the features for src blocks will apply, including calling arbitrary
code (not just Elisp). I do not see this idea as being a part of normal
usage, just really weird border cases. (At least, it is my
understanding. Max may chime in with more clarifications).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> It seems I completely failed trying to express my idea.
>
> Instead of extending Org grammar (syntax), I suggest to change behavior 
> of source blocks during export. In addition to current :results options, 
> "ast" may be added. Its effect is that instead of adding text to export 
> buffer that is parsed as Org markup, it causes insertion of a branch of 
> syntax tree into original parse results. I admit, during export it may 
> be necessary to iterate over source blocks one more time at a later stage.

So, do I understand it correctly that you are _not_ suggesting the AST
format _instead_ of the discussed inline blocks?

> Such source blocks should return "Org Syntax Tree", a simplified variant 
> of org-element. It allows to change implementation details and e.g. to 
> use vectors instead of lists for attributes in org-element. A converter 
> from Org Syntax Tree to org-element should be implemented.

Doesn't it effectively mean that we need to introduce yet another Org
element syntax---"Simplified AST"?

> Certainly such format may be used directly as src_ost{(code (:class var) 
> "language")} inline snippets or as
>
> #+begin_src ost
>    (code nil ("libtree-{sitter}-"
>               (code (:class var) "\"language\"")
>               "."
>               (code (:class var) "ext")))
> #+end_src
>
> block-level elements. However I expect that it is the last resort option 
> when there is no way to express desired construct in some other way.

I can foresee issues when we insert, say, code object in place of
paragraph element. The consequences might be drastic. Unless we just
allow users to shoot their own leg.

> I think, more convenient org-babel backends may be created to parse 
> TeX-like (texinfo-like) or SGML-like (XML-like) syntax into Org Syntax 
> Tree hierarchy. The essential idea is that outside of source blocks 
> usual lightweight markup is used. Source blocks however have just a few 
> special characters ([\{}], [@{}], or [&<>], etc.) to reduce issues with 
> escaping for regular text or verbatim-like commands.

This is not a bad idea, but I feel like it is getting a bit too far from
the syntax discussion herein. You might open a new thread about
importing foreign syntax into Org.

>>>       (code nil ("libtree-{sitter}-"
>>>                  (code (:class var) "\"language\"")
>>>                  "."
>>>                  (code (:class var) "ext")))
>> 
>> This is not much different from @name[nil]{<contents>} idea, but
>> more verbose.
>>
>  > Also, more importantly, I strongly dislike the need to wrap the text
>  > into "". You will have to escape \". And it will force third-party
>  > parsers to re-implement Elisp sexp reader.
>
> By this example I was trying to show how to express @var, @samp, @file 
> without introducing of new custom objects. I do not see any problem with 
> verbosity of such format, it may be used for really special cases only, 
> while some more convenient markup is used for more simple cases.

I was comparing the inline block vs. your AST proposal. If the AST idea
is complementary, I see no issue.

>>> If there was some syntax for object attributes then simple cases would
>>> be like
>>>
>>>       [[attr:(:class var)]]~language~
>> 
>> I do not like this idea. It will require non-trivial changes in Org
>> parser and fontification.
>> 
>> Using dedicated object properties or at least inheriting properties from
>> :parent is the style we employ more commonly across the code:
>> 
>> @var{language}
>> or
>> @code[:class var]{language}
>> or
>> @attr[:class var]{~language~}
>
> I do not mind to have some "span" object to assign attributes to its 
> direct children. I used link-like prefix object just because a proof of 
> concept may be tried with no changes in Org. It does not require support 
> of nested objects. There is no existing syntax for such "span" objects, 
> but perhaps it is not necessary and source blocks should be used instead 
> for special needs.

The problem is that instead of supporting nested objects we will have to
support "previous object changing the meaning of subsequent", which is
more fundamental change.

I envision the new inline blocks to allow assigning attributes to
children. These new inline blocks must not have issues with nesting by
design.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For some context, in order to support specialized syntax for manuals, we
  > may first need to implement the discussed special blocks export and
  > inline special blocks:
  > 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
  > 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/

  > The above links aim to introduce export functionality that we now have
  > for links to special blocks and new custom markup elements. I am
  > referring to
  > 1. Ability to create new custom element types programmatically 
  > 2. Ability to define how to :export the custom element types

I'm not sure of the meaning of some of those words.  Does this mean
that people are implementing those distinctions in Org mode?  Or does
it mean someone has proposed to implement them in Org mode but not yet
started actually writing it?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > For some context, in order to support specialized syntax for manuals, we
>   > may first need to implement the discussed special blocks export and
>   > inline special blocks:
>   > 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
>   > 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
>   > The above links aim to introduce export functionality that we now have
>   > for links to special blocks and new custom markup elements. I am
>   > referring to
>   > 1. Ability to create new custom element types programmatically 
>   > 2. Ability to define how to :export the custom element types
>
> I'm not sure of the meaning of some of those words.  Does this mean
> that people are implementing those distinctions in Org mode?  Or does
> it mean someone has proposed to implement them in Org mode but not yet
> started actually writing it?

The latter. We currently need a new syntax element that will allow all
the things available in Texinfo markup syntax.

We have decided that we do need such new syntax. We are discussing how
the new syntax will look like. We will implement the syntax in future.
Once implemented, the new syntax will open the road to add Texinfo
markup structures into Org export backends.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The latter. We currently need a new syntax element that will allow all
  > the things available in Texinfo markup syntax.

  > We have decided that we do need such new syntax. We are discussing how
  > the new syntax will look like. We will implement the syntax in future.
  > Once implemented, the new syntax will open the road to add Texinfo
  > markup structures into Org export backends.

I'm glad you are working on this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Robert Weiner]

I would just like to point out that anyone familiar with writing a Texinfo-format manual who wants to combine this with Org mode would likely just want to embed Org constructs, like Org tables in the manual; not to use Org as a formatter that exports individual source blocks to form a Texinfo manual (literate programming style).  This is because their focus while writing will be on the manual and its formatting, not on Org markup that exports to another markup.

I hope this is considered in your strategy.

-- rsw

> On Oct 8, 2022, at 6:38 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> The latter. We currently need a new syntax element that will allow all
>> the things available in Texinfo markup syntax.
> 
>> We have decided that we do need such new syntax. We are discussing how
>> the new syntax will look like. We will implement the syntax in future.
>> Once implemented, the new syntax will open the road to add Texinfo
>> markup structures into Org export backends.
> 
> I'm glad you are working on this.
> 
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
> 
> 
> 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Robert Weiner <rswgnu@gmail.com> writes:

> I would just like to point out that anyone familiar with writing a Texinfo-format manual who wants to combine this with Org mode would likely just want to embed Org constructs, like Org tables in the manual; not to use Org as a formatter that exports individual source blocks to form a Texinfo manual (literate programming style).  This is because their focus while writing will be on the manual and its formatting, not on Org markup that exports to another markup.

Could you please elaborate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I would just like to point out that anyone familiar with writing a
  > Texinfo-format manual who wants to combine this with Org mode
  > would likely just want to embed Org constructs, like Org tables in
  > the manual; not to use Org as a formatter that exports individual
  > source blocks to form a Texinfo manual (literate programming
  > style).

The proposal I thought I was responding to is different from both of
those.  It was to make Org format an option for the source format for
a whole GNU manual.  One that could generate the same output formats
that we now generate from Texinfo, and would be able to produce a
well-formatted printed manual via TeX.

This does not inherently require imply generating Texinfo format from
Org format.  Texinfo format is a source format, not an output format.

However, generating Texinfo format as an intermediate format might be
advantageous as a way to feed the Org-format manual through TeX to
get a well-formatted printed manual.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)