Re: [RFC] Org Minor Mode?

[Richard Lawrence <richard.lawrence@berkeley.edu>]

Hi Thorsten,

Bastien <bzg@gnu.org> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> What do you think - is there any chance that Org-mode switches from
>>> static hardcoded regexp strings (all over the place) to dynamic
>>> regexps calculated at runtime (using libraries like drx.el or rx.el)?
>>
>> I hope not. The syntax should stabilize, not drift away.
>
> Agreed.  Maybe there are some hardcoded regexps that we can factor
> out, but dynamically building those fundamental regexp is a deadend.

I agree with what Nicolas and Bastien have said, but I wanted to say
that I think there is an interesting idea in Thorsten's post that
shouldn't be dismissed out of hand.

Org provides a set of UI concepts (tree-like structure, visibility
cycling, tree filtering, task state tracking, building an agenda from
multiple sources, ...)  that map nicely onto a lot of other situations,
and would be really handy to have access to even when the syntax of the
underlying file is incompatible with Org's syntax.

There are two ways to think about Org syntax, which I think should be
distinguished here.  One is as the grammar of a .org file: basically, a
set of rules that allow a sequence of characters to be parsed into an
AST.  The other way to think about Org syntax is the way Lisp
programmers sometimes talk about syntax: as the AST itself, the
collection of Lisp data types and their interrelationships that define a
valid Org document.

If Org were to evolve to the point where the UI concepts were
implemented purely as transformations on an AST -- on Org syntax in the
second sense -- then the way would be clear for making those concepts
available in editing modes where the grammar of the underlying file is
incompatible with Org syntax in the first sense.  A programming mode
could, say, parse comments into an Org AST, then expose that AST to the
functions implementing Org's UI concepts.  Et voila: you get visibility
cycling, task state tracking, agendas...in your source code comments.

One sort of use case where I think this idea could really shine is in
dealing with email.  Obviously, the grammar of the underlying mail files
(say, in a Maildir) will never be compatible with Org syntax in the
first sense.  But Org handles so many of the concepts that apply to
email (threading messages into hierarchies, visibility cycling, tagging,
sorting by date or priority, thinking of messages as tasks to be dealt
with, dealing with attachments) in such a nice way that I find myself
sorely missing Org whenever I read mail in a client that doesn't
implement them as nicely -- which is all of them.  If it were possible
to build a parser for message files that transformed them into an Org
AST, the mail client of my dreams would be in reach.

I have no idea if evolving Org in this direction is feasible or even
really desireable.  It may be that the two notions of Org syntax are
tightly coupled in principle, so that the idea of producing an Org AST
from an alternative underlying file format will never make sense.  But I
think that would be surprising.  

This evolution would clearly require more work than just abstracting out
the regular expressions that implement much of Org's syntax in the first
sense, and I think Bastien and Nicolas are right that we don't want
either notion of Org syntax to become less stable.  Still, I think
there's a lot of interesting possibilities we could explore if Org's
implementations of the two notions of syntax were to become less tightly
coupled.

Best,
Richard