Re: Orgdown: negative feedback & attempt of a root-cause analysis (was: "Orgdown", the new name for the syntax of Org-mode)

[Tim Cross <theophilusx@gmail.com>]

Tom Gillespie <tgbugs@gmail.com> writes:

> Karl,
>    The exact naming of a thing is nearly always the most contentious
> step in trying to promulgate it. In my own field we can easily get all
> parties to agree on a definition, but they refuse to budge on a name.
> As others have said, I wouldn't worry about kibitizing over the name.
>
> I would however worry about the larger negative reaction. From my
> perspective I think the issue is that there are many efforts working
> toward a formalized specification for Org syntax and Org mode
> functionality, and some of those stakeholders who have invested
> significant effort may feel blindsided by a public declaration
> announcing Orgdown because they were not consulted and not
> made aware that you were working on it.
>
> I appreciate the amount of work that you have put in, I have devoted
> hundreds of hours to working on an alternate implementation of org
> in Racket that uses a formal ebfn in hopes that others will be able
> to use it as a guide and as a way to talk formally about how Org
> parsers and implementations should behave.
>
> It would thus be easy for me to say that your approach has put the
> cart before the horse, because there are countless nuances in the
> specification for Org syntax which must be addressed before any
> levels of org compliance can be specified, otherwise the behavior
> between levels will be inconsistent.
>
> If I were to say this, it would not be fair to you at all. The ideas
> and motivation for Orgdown are vital and important. You have put
> in enormous thought and effort, all because you care about Org
> and want to see it succeed.
>
> The issue is that any shared specification for Org syntax is
> fundamentally about how to coordinate as a community.
> The way that Orgdown was presented to the community feels
> (to me) like it is being imposed top down or coming from an
> individual source, not from an open and visible community
> process (the subject of your original email reads as a declaration
> in english, and thus can be quite off putting, though I know that
> was not the intention).
>
> I personally haven't bothered with promulgation because I think
> that we are not technically ready as a community to approach
> outreach to other developers in a way that we can succeed.
>
> The good news is that all of this can co-exist if we want it to,
> but we need to be clear about our objectives as a community.
>
> To me these objectives are as follows (and I would love
> to hear from others about additional or alternate objectives).
>
> 1. To never fracture Org syntax so as to avoid the nightmare
> of markdown flavors. (This means being able to say clearly
> as a community that a parser is out of compliance and that
> it is up to the user to fix their files. The ruby org parser used
> by Github is a major issue here.)
> 2. To provide a clear specification for what graceful degradation
> looks like when parsing Org syntax if a parser does not support
> some portion of that syntax (e.g. should property drawer lines
> be excluded or rendered as plain text?).
> 3. Provide a solid basis on which further formal specification
> can be built. (My interests in particular are around providing
> consistent semantics for org-babel blocks across languages
> so that babel implementations can clearly communicate what
> runtime features they support.)
>
> The approach for Orgdown can absolutely meet all three of
> these objectives, however in its current form Orgdown1 is not
> sufficiently well specified to avoid fracturing the syntax.
> This is because Org syntax is extremely complex (even the
> elisp implementation of Org mode is internally inconsistent)
> and there are edge cases where behavior will diverge if parsing
> of even the simplest elements is not fully specified.
>
> There are many ways to remedy this, however they require
> a more formal approach. A number of us are working to build
> technical foundations for such a formal approach, but I do not
> think that any of those projects are ready to be used to
> specify discrete levels of Org syntax parsing compliance.
>
> If I may, I would suggest that an Orgdown0 is something that
> could be well specified, but it would avoid parsing of markup
> altogether and only deal with the major element types. Parsing
> paragraphs and all the org objects is not something that can
> be done piecemeal. There are too many interactions between
> different parts of the syntax, and in some cases the existing
> specification desperately needs to be revisited due to the
> complexity that it induces or because it is underspecified.
> Of course this would make Orgdown0 fairly useless as a
> replacement for markdown, but at least it would be a start.
>

Hi Tom,

I pretty much agree with everything you wrote. I also feel it is
unfortunate that Karl received so much negative focus on the name and
not on the underlying idea - but I'm not surprised. As you say, naming
is extremely hard and getting consensus is even harder. In hind-sight it
would probably have been better to separate the two things - finding a
name and defining a way to specify compliance. however, I also agree
that I'm not sure we are yet ready to specify compliance because we need
more formalism regarding what we are dealing with.

One of the most important and I think potentially positive threads
recently has been that of Ibor's regarding using the element parser to
drive fontification and formatting. Inconsistencies between what an org
file looks like and how it is parsed, exported etc seem to be the source
for considerable confusion and bugs. Having the 'engine' be based around
an element parser which all other components leverages from would at
least provide increased consistency. Things will be easier to manage,
fix and extend if they are consistent. I also think this is the key to
being able to improve system performance and could provide a useful
'validator' which could be used by 3rd party tools to verify their
implementations. It would also make extending org-mode easier. I also
think less reliance on regular expressions scattered across different
layers will make maintenance much easier. While regular expressions can
be very powerful, they are hard to get right, maintain and avoid
performance problems. I also find regexp in elisp one of the more
difficult regexp formats/engines to work with.