Re: [Out-of-Thread] Re: [RFC] Org syntax (draft)

[Carsten Dominik <carsten.dominik@gmail.com>]

Hi everyone,

first a disclaimer:  Nicolas has thought about all things parser a lot more than I have, so he might disagree.  But here is my take on the issue.

First of all, we should not see Org as just another plain text markup language (no offense meant, I am sure, and none taken).  Because of its unique treatment of source code inclusion, source code markup, and executability, it is very much unique, I think.  Therefore it deserved a clean definition and a well defined parser, with a goal to make files function correctly in different peoples setup.  This is what Nicolas is trying to give to us - aside from turning an incredible parser/exporter mess into a sane system.  We used to have as many parsers as exporters, all slightly different, bugs having to be fixed in different ways in each parser.  That is gone, with a very limited amount of pain, and I am entirely amazed that this was even possible, let alone that someone had the stamina to do it and the patience it takes to put this, with the small limitations it brings, onto the community.

Now, I also agree that it is desirable that Org remains to be a hackable system with as much flexibility as we can allow, and we need to carefully consider where the line is.

For example, every users setup has some dependency of parser behavior on external variables: todo keywords.  The way this is fixed in the sense that we can guarantee a stable parsing of such a system is to tell people that including the TODO keyword definitions with a #+TODO line into the file.  So you can have a global setting in a global customization variable to make things easy for yourself and get good behavior in every new file you open, but you can stabilize a file for the parser with in-buffer settings when you need compatibility for other users.

There are a few exceptions that Nicolas has pointed out in this thread.  The COMMENT keyword is one.  We could define an in-buffer setting for it, or we could just fix it, the pain here would be minimal.

Another example is the emphasis stuff.  There are no in-buffer settings for it, and they would be pretty hard to make.

The reason why the emphasis regexp components were made configurable in the first place is because when the feature was introduced, I had no idea what would work, and I redesigned this part several times over.  Emphasis is a very heuristic system, the character that are allowed before and after the markers are necessarily a compromise, and we will always find people for who the chosen selection will not work.  That is why I would like to argue for keeping this part hackable, even if I agree that the official definition should be fixed.  Keeping this variable a customize variable invites changes also by people who do not really know what they are doing.  Turning it into a defvar or defconst and somewhere document how to hack around the restriction if you really need to sounds like a good solution for me.

Now to the discussion with Z about additional emphasis definitions which he/she uses for custom highlighting of stuff.  Right now this relies on modifying the emph-alist variable.  However, for the purpose of in-buffer only highlighting, it is not necessary to go through parser-sensitive code.  You can do this simply with additions to font-lock, for example using font-lock-add-keywords or something like this, see also Thorsten's post.  If someone wants, I can provide an example for Z's case, and we could encapsulate such behavior into a little library in contrib, to make it easy to configure such behavior.  Compromising the parser for this application is not necessary.

Nicolas, I would assume that your wish to disallow customization is focused on the regexp components, and the marker characters for emphasis, but not on the possibility to change the in-buffer face that is being used to highlight the stuff in the buffer?

- Carsten


On 18.3.2013, at 16:21, W. Greenhouse <wgreenhouse@riseup.net> wrote:

> zeltak <zeltak@gmail.com> writes:
> 
>> Dear Carsten,
>> 
>> Thank you for your quick reply. Let me start by first thanking you
>> for your great work on orgmode, I only recently discovered it
>> (someone referred me to your great talk on youtube) and it made me
>> have the courage to start learning emacs and use orgmode.
> 
> [snip]
> 
>>  '(org-emphasis-alist (quote (("@" (:foreground "#B40000" :background
>> "#FFDDDD" :weight bold) "" "") ("$" (:foreground "#FF0000") "" "")
>> ("*" bold "<b>" "</b>") ("/" italic "<i>" "</i>") ("_" underline "
>> <span style=\"text-decoration:underline;\">" "</span>") ("=" org-code
>> "<code>" "</code>" verbatim) ("~" org-verbatim "<code>" "</code>"
>> verbatim) ("+" (:strike-through t) "<del>" "</del>")))) 
>> 
>> That would have worked for me but i understood that there are plans
>> to actually disable these customization's in the next version to
>> allow better portability. 
>> 
>> If its not to hard It would be great to have a method similar to the 
>> customizable emphasis that lets a user define custom colors of FG/BG
>> for inline viewing. I am less concerned about exporting since at
>> least for me i plan to do all the editing/viewing inline inside emacs
>> (though it would be nice of course to be able to use it with mobile
>> org and/or other mobile solutions for when we do field work).
>> 
>> I will hold on with starting with the mammoth task of converting my
>> Notecase notes into orgmode until the issues is resolved, i assume i
>> should follow the mailing list to check on this?
>> 
>> Sorry for the long email and thank you so much again for all your
>> work, its truly fantastic
>> 
>> Z.
> 
> I want to add, as one of the people that helped Z with this on IRC--and
> as another person that made the leap into Emacs largely because of Org,
> about five years ago--that I think there are a lot of users like this:
> people who value Org as a tool that is tightly integrated with the power
> and flexibility of Emacs and Emacs Lisp, who aren't necessarily closely
> following Org's upstream development or this list (I didn't follow it
> closely until recently, either), and who are more concerned with keeping
> Org flexible and customizable enough to exactly fit their needs within
> Emacs than they are about making it available as yet another plain-text
> markup language outside of Emacs.  Much as my Gnus is heavily customized
> to my needs at this point, with Elisp-based features such as adaptive
> scoring and virtual groups that other news and mail readers simply don't
> have, I would never really dream of reproducing Org outside of Org.  And
> there are plenty of things that I would never expect to work in an
> external application or parser that speaks the Org format (dynamic
> blocks that run Elisp, for example), which everyone nonetheless wants to
> have.
> 
> Perhaps a compromise could be reached on variables such as
> `org-emphasis-alist' and others possibly slated for the defconst
> treatment: instead of doing that, let's consider keeping them
> customizable but include the default values in the Org format
> specification.  Org users who are never using Org outside of Emacs will
> never see a problem using custom emphasis marks inside Emacs, unless Org
> drops the feature.  For those who know that they want to use their Org
> files with some external parser, we could have an `org-rfc-check'
> function that warns about non-standard values of things like
> org-emphasis-alist and offers to revert them to the defaults (which
> would be the same as the values in the spec).
> 
> What do you think?  Is this a crazy scheme?
> 
> -- 
> Regards,
> WGG
> 
>