Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

I am CCing Org ML from now signifying that this branch of the thread is
directly relevent to Org mode and might be of interest for other Org
contributors.

Richard Stallman <rms@gnu.org> writes:

>   > So, now we have our manual written in Org mode and we never had reasons
>   > to come back to texi.
>
> I suspect that it doesn't fully follow the markup conventions
> for GNU manuals.  That's because we designed Texinfo to have markup commands
> to make all the proper semantic distinctions.  If the manual source
> is written in a language which doesn't have the full gamut of markup
> distinctions, there is no way to do the markup correctly.
>
> It would be useful for someone who understands these conventions
> to check the Org manual and see.

Yes, it would certainly help!
The work on manual has been done a long time ago and we also extended
our texinfo exported to suit the manual at that time. Assumingly, just
enough to handle the Org manual use-cases.

Note that we have doc/Documentation_Standards.org explaining some of the
conventions.

Here is a possibly relevant note inside it:

 - Only two of the standard Texinfo indexes are used; those for
   concepts and keys.  This has some implications:

   + The preference is to document commands by key rather than by name

   + Texinfo commands such as @var and @defoption are not used.  The
     preference for this type of thing is that the user browses the
     customize groups.  If you want or need to refer to, say, a
     variable then document it as "the variable
     @code{org-startup-folded}"
 
   + Entries in the concept index are normally all lower case unless
     some other rule dictates otherwise.

Without knowing texinfo, the above paragraphs do not make a whole lot of
sense for me. So, if someone points out any omissions, it would be
helpful for future Org contributors.

> I don't think the slowness of processing nowadays is the crucial issue
> here.  Computers are much faster now than in the 1990s.  It used
> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> Now it is perhaps 10 times as fast.

Generally, there is no way Org export to .info gets any faster than
texinfo. Org is only able to export to other text formats: org->texi;
org->tex; org->html; etc. Convertion to more low-level formats is left
to the external tools like texinfo and pdflatex.

As for reports on the slow performance, they are still useful as long as
they reveal some bottlenecks in Org exporter. Org is being used to
export large documents and whole websites. Hence, improving performance
in this area is generally helpful, even if it is not strictly a critical
blocking issue.

What I want to say is: do not expect Org export to be faster than native
binary tools, but do not hesitate to report performance issues either.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Forwarding a relevant message from emacs-devel on examples of markup
nesting in texinfo:

Eli Zaretskii <eliz@gnu.org> writes:

> From: Ihor Radchenko <yantar92@gmail.com>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can someone please prepare a small example texi file containing examples
> >> of various texi features that are commonly needed and possibly also
> >> demonstrating some concrete instances of problems that exist in texinfo?
> >
> > Given that you already produced a list of markups, is such an example
> > still wanted?
> 
> It would certainly help with development. For example, I did not look
> close into allowed markup nesting. File with tricky examples will be
> helpful as a benchmark for any attempted implementation.

 I'm afraid that would take someone who knows more than I do about Org,
 because I don't have a clear idea what examples to provide to be
 useful for this purpose.  In general the Texinfo manual has examples
 of every Texinfo feature, of course.  But it is a large manual.

 Specifically wrt nested markups, the most frequent ones I can think
 off the top of my head are:

   . @var inside @code
   . @key inside @kbd
   . any in-line markup inside @w
   . @r, which is "escape to regular typeface" and can be used inside
     any markup that produces other typefaces

 HTH

Cusom special block export, similar org org-link :export parameter (was: Org mode and Emacs)

[Ihor Radchenko]

Robert Pluim <rpluim@gmail.com> writes:

>     Ihor> However, we have special blocks for this purpose. They are extendable,
>     Ihor> as I descibed above. Though some improvements in Org core might be
>     Ihor> needed if we have to use this extensibility more frequently.
>
> My first thought here was Org macros, but as far as I can tell they canʼt
> be multi-line, which would make them a bit cumbersome to use. Tell us
> more about special blocks, the documentation on them is a bit sparse.

Macros are meant to configure per-document level of export. Not global,
usually.

Special blocks are Org blocks with the following syntax:

#+begin_environment [optional parameters]
#+end_environment

With "environment" being any word except "center", "quote", "comment",
"example", "export", "src", and "verse".

Currently, latex export directly translates such blocks into
\begin{environment}
\end{environment}

plain text export supports special handling of
environment=justifyleft, justifyright.

html export supports
'("article" "aside" "audio" "canvas" "details" "figcaption"
    "figure" "footer" "header" "menu" "meter" "nav" "output"
    "progress" "section" "summary" "video")
special blocks

texinfo export translates special blocks into
@environment [options]
@end

odt exprot supports "annotation" and "textbox" blocks.

etc.

Because "environment" can be pretty much anything, we can have a
pre-processor in org-export-filter-parse-tree-functions that transforms
different special blocks into appropriate form for a given export
backend (texinfo, latex, ascii, html, etc). This way, we will not have
to rely on backend-specific handling.

Such pre-processor currently does not exist, but it would be fairly
trivial to implement. We may even introduce an interface similar to
org-link-set-parameters for easier configuration of various types of
special blocks.

WDYT?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tom Gillespie]

Having read the whole thread now: oof. Thank you Ihor for shepherding
that and for the performance improvements!

With regard to the key-bindings straw man. I guess I'm a bit of an
outsider on this one, because I started writing org documents by just
typing them in and only over time learning some of the bindings. Maybe
having an org-markup-mode or something like that would be a way to
provide a sandbox for the +recalcitrants+ newcomers? It might also be
a nice way to a/b test them on whether the Emacs editing commands
really are as good as they think they are (said the evil-mode user).

With regard to ... everything else. I guess at this point it is
unsurprising that (for lack of a better term) the uninitiated in the
dark corners of org syntax frequently think that syntactic extensions
are advisable, skipping over the consideration of possible.

Given the opportunities that seem to be lurking in the thread, it
seems like it would be good to have some examples of how the
e.g. texinfo semantic markup could (or could not) be implemented using
existing org syntax. The suggestion to use custom link types seems
very practical. It requires no new syntax, and is basically fully
extensible for semantic markup needs.

I say this having recently spent time reworking the paragraph grammar
and the lexer needed to enable it in laundry for the 3rd (or is it
4th?) time. Say it with me: No new syntactic forms! We have more than
enough syntax to enable all the extensibility that pretty much anyone
will ever need (we just have to document how to use it).

In-document extensibility of link types might be possible if we get my
regularized keyword syntax implemented, if that were done then all the
configuration could in-principle live in a setup file (I have a
response on the syntax thread drafted, will try to get back to it).

Nesting markup inside code or verbatim seems more difficult because
they are intentionally terminal. I am also unfamiliar with texinfo so
will be of no help with the examples, but I do look forward to them.

Best!
Tom

Org syntax compatibility with texinfo syntax (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))

[Ihor Radchenko]

I am forwarding yet another email from the emacs-devel thread to here.
This thread includes both emacs-devel and Org ML.
I will still keep emacs-devel in the loop because people from there can
provide a useful insight about texinfo capabilities.

Richard Stallman wrote:

> I don't know for certain that every possible nesting "does the right
> thing".  I do know that @var{} is used inside many other constructs.
> By contrast, @dfn{} would not be nested inside or around other
> contructs very much.  @key can be nested inside @kbd, and it behaves
> a little differently when nested.

signifying (in addition to previous message), that
1. Texinfo has some software documentation-specific markup elements
2. Those elements can be nested in non-trivial ways, similar to Org's
org-element-object-restrictions, the that nesting can affect formatted
export (again, similar to Org).

With regards to Org syntax, I do not think that we must include all the
possible texinfo elements into Org standard.
@dfn, @key, and @var constructs are extremely specific to software
documentation and have no sense to maintain as a part of general-purpose
markup syntax.

However, what we can do is to allow extending Org syntax to support such
elements. We do support custom syntax elements already (or are
discussing such support):

We have the following, potentially customizeable syntax elements:
- Org links (extensible via org-link-set-parameters)
- Special blocks (de-facto extensible in export backends; also,
  https://github.com/alhassy/org-special-block-extras/);
  https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af
- [FR] Inline special blocks idea being discussed in
  https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net

The links are somewhat limited wrt nesting: links cannot be nested
inside links thus limiting their usefulness as custom markup. Hence, we
may need to look more closely into the idea of inline special blocks
(discuss it in https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net).

Special blocks already support custom :type, but lack convenient Elisp
interface like org-link-set-parameters. (let's keep this part in
https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af)

I'd like to keep discussion on the specifics of customizeable Org syntax
in the relevant threads.

In here, I'd like to hear feedback on possible additional
incompatibilities between texinfo and Org. Is there anything (not
covered by the above list) that is present in texinfo, but impossible in
Org?

For starters, Org does not have a full support for generating index
(apart from ox-texinfo and ox-publish). Though see
https://github.com/tecosaur/org-glossary

Best,
Ihor

Simplified Org mode for newcomer Emacs veterans (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))

[Ihor Radchenko]

Tom Gillespie <tgbugs@gmail.com> writes:

> With regard to the key-bindings straw man. I guess I'm a bit of an
> outsider on this one, because I started writing org documents by just
> typing them in and only over time learning some of the bindings. Maybe
> having an org-markup-mode or something like that would be a way to
> provide a sandbox for the +recalcitrants+ newcomers? It might also be
> a nice way to a/b test them on whether the Emacs editing commands
> really are as good as they think they are (said the evil-mode user).

It can be either a simplified org-markup-mode or a series of minor-modes
that enable different sets of bindings. Something like:
org-structure-edit-mode, org-table-edit-mode, org-babel-edit-mode, etc

However, the tricky question is: What should be the default?

If we have org-markup-mode (disabled by default), how would the
newcomers know to enable it?

If we have the alternative set of minor modes and disable them by
default, will the existing Org mode users accept the need to adjust the
configuration?

Can we address the above concerns without dissatisfying neither the
existing Org users nor the newcomers?

Best,
Ihor

Re: Simplified Org mode for newcomer Emacs veterans (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tom Gillespie <tgbugs@gmail.com> writes:
>
>> With regard to the key-bindings straw man. I guess I'm a bit of an
>> outsider on this one, because I started writing org documents by just
>> typing them in and only over time learning some of the bindings. Maybe
>> having an org-markup-mode or something like that would be a way to
>> provide a sandbox for the +recalcitrants+ newcomers? It might also be
>> a nice way to a/b test them on whether the Emacs editing commands
>> really are as good as they think they are (said the evil-mode user).
>
> It can be either a simplified org-markup-mode or a series of minor-modes
> that enable different sets of bindings. Something like:
> org-structure-edit-mode, org-table-edit-mode, org-babel-edit-mode, etc
>
> However, the tricky question is: What should be the default?
>
> If we have org-markup-mode (disabled by default), how would the
> newcomers know to enable it?
>
> If we have the alternative set of minor modes and disable them by
> default, will the existing Org mode users accept the need to adjust the
> configuration?
>
> Can we address the above concerns without dissatisfying neither the
> existing Org users nor the newcomers?
>

I'm not convinced we actually need to do anything (yet). Most of the
'issues' raised by Eli were IMO theoretical rather than real. WE see
few, if any, issues or bug reports relating to most of the points he
raised. I'm also not convinced regarding some of the arguments regarding
casual or 'seldom' users. For me, many of the issues felt somewhat
contrived and actively looking for reasons why increased use of org in
Emacs was a "bad thing". 

This is not to say some of his points don't warrant some consideration.
However, they do seem largely general 'theoretical' and based on a
preconception of what an emacs mode is. In many ways, Org is not a
'normal' emacs mode. In some ways, it is a collection of modes with glue
to make them interoperate a little better. It is therefore possible,
many of the normal 'best practices', especially with respect to key
bindings, may not apply. 

I'm not fond of the 'magic' approach whereby special modes get activated
because some specific data is 'seen' in the buffer. For example, only
loading table editing mode because a table was seen or when the user
enters a line which looks like the start of a table. I much prefer a
system which allows me to enable the modes I want - similar I guess to
how we handle babel languages. However, that could just be me. It would
be good though that if we do support some form of 'dynamic' loading if
Emacs first asks i.e. "It looks like your editing a table. Shal I load
org-table-minor-mode?" sort of thing. 

So my approach would be to break things up into their own minor modes,
but by default, load them all. This will deal with the issue of not
impacting existing users. Typically, those who will care about not
loading additional unwanted bindings or features will also be the same
set of people who will be willing to customise their setup and they can
easily remove/turn off the modes they don't want. 

The one big area which does concern me slightly with introducing this
sort of modularity is with debugging and support. For example, to
reproduce the environment where an issue is encountered, we may need to
also know more about exactly which set of minor modes as been enabled
and possibly the order they are enabled. Even just basic testing will
become more complex as you may need to test with different minor mode
permutations. We may need to add some additional debugging and reporting
functionality to assist in this area. 

Likewise, how does org deal with an org file which includes some feature
the user has turned off. Consider a babel minor mode. Do we allow the
user to edit the babel blocks without loading that mode? Doe we warn
them the mode is not being loaded due to personal configuration? Do we
just disable all babel support if they don't load babel minor mode? 

One area which might be worth starting with would be to create an org
minor mode which only provides basic org navigation, folding and
font-locking support - no babel, no export, no agenda. reduced task
management key bindings. Essentially a minor mode which would render org
files in a 'clean' readable format, allow basic navigation and editing
and some basic/simple todo management. This would be the preferred mode
for seldom/casual users not interested in the full org experience. 

Re: Simplified Org mode for newcomer Emacs veterans (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I'm not convinced we actually need to do anything (yet). Most of the
> 'issues' raised by Eli were IMO theoretical rather than real. WE see
> few, if any, issues or bug reports relating to most of the points he
> raised. I'm also not convinced regarding some of the arguments regarding
> casual or 'seldom' users. For me, many of the issues felt somewhat
> contrived and actively looking for reasons why increased use of org in
> Emacs was a "bad thing". 

This is a similar wording to my previous reply to Eli (I think it was to
Eli). The answer was that the emacs-devel discussion _was the bug
report_. Eli sometimes uses Org, but finds many things too much for him.
Other complaints are from Org non-users who are not interested to give
bug reports because they stopped short at the first try of using Org.

Not to say that I agree with all the complaints, but they should be
discussed here at least.

> This is not to say some of his points don't warrant some consideration.
> However, they do seem largely general 'theoretical' and based on a
> preconception of what an emacs mode is. In many ways, Org is not a
> 'normal' emacs mode. In some ways, it is a collection of modes with glue
> to make them interoperate a little better. It is therefore possible,
> many of the normal 'best practices', especially with respect to key
> bindings, may not apply. 

Surely, Org is a collection of at least org-agenda-mode and org-mode :)
The irony is that 'best practices' are not implemented by Emacs itself
(look at the number of default Emacs bindings!).
In any case, we should still try to extract something useful out of the
received feedback.

> I'm not fond of the 'magic' approach whereby special modes get activated
> because some specific data is 'seen' in the buffer. For example, only
> loading table editing mode because a table was seen or when the user
> enters a line which looks like the start of a table. I much prefer a
> system which allows me to enable the modes I want - similar I guess to
> how we handle babel languages. However, that could just be me. It would
> be good though that if we do support some form of 'dynamic' loading if
> Emacs first asks i.e. "It looks like your editing a table. Shal I load
> org-table-minor-mode?" sort of thing. 

I also dislike that idea. I proposed it as a part of brainstorming,
hoping other ideas to be proposed by Eli. Alas...

> So my approach would be to break things up into their own minor modes,
> but by default, load them all. This will deal with the issue of not
> impacting existing users. Typically, those who will care about not
> loading additional unwanted bindings or features will also be the same
> set of people who will be willing to customise their setup and they can
> easily remove/turn off the modes they don't want. 
>
> ...
> One area which might be worth starting with would be to create an org
> minor mode which only provides basic org navigation, folding and
> font-locking support - no babel, no export, no agenda. reduced task
> management key bindings. Essentially a minor mode which would render org
> files in a 'clean' readable format, allow basic navigation and editing
> and some basic/simple todo management. This would be the preferred mode
> for seldom/casual users not interested in the full org experience.

I tend to agree:
1. We modularize some of the key bindings and overriding defaults (like
   org-special-ctrl-o in org-open-line) into minor modes. They will be
   enabled by default.

2. We create an org-clean-mode (aka clean-mode in Emacs master) that
   disables/toggles those minor modes.

> Likewise, how does org deal with an org file which includes some feature
> the user has turned off. Consider a babel minor mode. Do we allow the
> user to edit the babel blocks without loading that mode? Doe we warn
> them the mode is not being loaded due to personal configuration? Do we
> just disable all babel support if they don't load babel minor mode? 

I think I was not very clear in my previous message. There is no way we
disable parts of Org syntax. That will require refactoring of
org-element and many other functions. Not acceptable.

What we can disable/change is some of the default bindings + certain
customizations. The minor modes I propose will simply bind/unbind groups
of Org bindings and toggle certain custom variables between
Org-preferred and Emacs-default-ish.

> The one big area which does concern me slightly with introducing this
> sort of modularity is with debugging and support. For example, to
> reproduce the environment where an issue is encountered, we may need to
> also know more about exactly which set of minor modes as been enabled
> and possibly the order they are enabled. Even just basic testing will
> become more complex as you may need to test with different minor mode
> permutations. We may need to add some additional debugging and reporting
> functionality to assist in this area. 

I do not think that we should really disable babel support hard. I do
not see any problem leaving M-x org-babel-... working all the time.
We can just disable key bindings and change certain settings. This will
not be any different compared to dealing with tailored user configs. No
extra effort will be required on our side in this regard.

Best,
Ihor