Proposal for new document-level syntax

Proposal for new document-level syntax

[Gustav Wikström]

                ________________________________________

                 PROPOSAL FOR NEW DOCUMENT-LEVEL SYNTAX

                            Gustav Wikström
                ________________________________________


Table of Contents
_________________

1. Summary
2. Background, details etc.
3. Proposal details
4. Code?


1 Summary
=========

  I propose a "document" element in org-element, a property-drawer on
  document-level, a setting-drawer on document-level and
  property-keywords (slightly different than what already exist). And
  would like your comments regarding that!

  More details below.


2 Background, details etc.
==========================

  I'm amazed by how well org-mode fullfills my needs for writing,
  thinking, organizing and structuring information. Having a text-based
  outliner with rich formatting, searching and inter-linking options is
  amazing! There are a few things that's bugged me for a while though.
  One of those things is the fact that org-mode is lacking when you're
  outside of a headline. Basically, org-mode lacks the proper notion of
  a document. One use case for a more clear notion of a document is for
  large libraries of short notes separated by files instead of separated
  by headlines. Separating notes by files has drawbacks today. You can't
  easily define todo's, tags, ID's, attachments etc. when your not
  within a headline!

  You might think: "But just create a headline if you want those
  things..!"

  To which I'd answer: "Yes, that works. As a workaround. But it's not
  good enough! What do I name my headline? I already have a name for the
  file, having to define a name again for a headline inside that file is
  redundant for short notes."

  Look for example at org-brain which had to introduce separate concepts
  to be able to deal with files and subtrees ([headlines and files]).
  Some org-brain issues that relates to this lack of a document-concept:
  ([#118], [#91], [#48]). This suggestion is not about org-brain. But I
  just wanted to give an example of how this is something that /can/ be
  improved.

  To get started on this road towards a clearer definition and syntax
  for things above the headline-concept, I have a couple of suggestions
  I hope the community will support!


[headlines and files]
<https://github.com/Kungsgeten/org-brain#headline-and-file-entries>

[#118] <https://github.com/Kungsgeten/org-brain/issues/118>

[#91] <https://github.com/Kungsgeten/org-brain/issues/91>

[#48] <https://github.com/Kungsgeten/org-brain/issues/48>


3 Proposal details
==================

  I propose to introduce two org-mode drawers to be used in the top of
  org-mode documents:
  1) A setting drawer. Applicable only if positioned at the top of a
     buffer, below any potential comment-line or document-level keywords
     (more on those later).
  2) A property drawer. Applicable with the same positional condition as
     the settings-block, and positioned after the settings-block if both
     blocks are available.

  I also propose to allow for whatever property to be defined as a
  keyword if also defined at the top of the document.

  Details for these proposals follows:

  The setting drawer makes the following keywords redundant:
  - #+STARTUP
  - #+TODO
  - #+SEQ_TODO
  - #+TYP_TODO
  - #+PRIORITIES
  - #+TAGS
  - #+LINK
  - #+CONSTANTS
  - #+SETUPFILE
  - #+MACRO

  Initially I propose to just use the setting drawer as an additional
  syntax allowed for the above keywords, preferably with a new
  helper-method for setting them. Settings defined in this drawer should
  have precedence over settings defined by the keywords above. Long term
  I'd like to depricate the above keywords in favour of the setting
  drawer.

  The property drawer is used for properties defined at node-level 0.
  Properties defined inside this drawer are applied in exactly the same
  way as properties on headline-entries are applied for it's subtree.
  I.e. the same inheritance-rules apply.

  The property drawer makes the following keywords redundant: (since all
  of them have property drawer equivalences)
  - #+PROPERTY
  - #+OPTIONS
  - #+CATEGORY
  - #+FILETAGS
  - #+COLUMNS
  - #+ARCHIVE

  Defining properties on the document-level should work using the same
  commands as today when the point is before the first headline.
  Properties defined in the document property drawer should have
  precedence over properties defined by the (then redundant) keywords
  above.

  I propose to allow properties to be defined also as document property
  keywords. All keywords in the top of a buffer, before any non-comment
  line, are document-level keywords. In effect, they are properties that
  apply in exactly the same way as properties defined in the property
  drawer. The only reason for using a document keyword instead of
  defining it inside the property drawer is to make it more visible. One
  example would be the title-keword (#+TITLE: ...).


4 Code?
=======

  Work in progress can be found here:
  <https://github.com/Whil-/org-mode>

  With separate branches in an attempt to isolate things from each
  other:
  New document element
        <https://github.com/Whil-/org-mode/tree/feature/org-document-formalization>
  Document property drawer
        <https://github.com/Whil-/org-mode/tree/feature/org-document-property-drawer>
  Document property keywords
        <https://github.com/Whil-/org-mode/tree/feature/org-document-property-keywords>
  Document setting drawer
        <https://github.com/Whil-/org-mode/tree/feature/org-document-setting-drawer>

  The first two branches are working already (though lacking
  documentation) while the two last ones mostly exist as ideas. Note
  that of these branches most likely will get their history rewritten
  from time to time. I.e they're public... but dangerous!

Re: Proposal for new document-level syntax

[Fraga, Eric]

On Saturday,  1 Jun 2019 at 10:15, Gustav Wikström wrote:
>   I propose a "document" element in org-element, a property-drawer on
>   document-level, a setting-drawer on document-level and
>   property-keywords (slightly different than what already exist). And
>   would like your comments regarding that!

Dear Gustav,

thanks for this proposal.  I do understand your motivation for this and
can see a document level entity being useful.  

For me, however, your proposed structure would clash strongly with my
usual working practice.  Specifically, I put all customizations and
settings at the end of my documents, along with, for instance, Emacs
file local variables.  These are things I do not change often and
usually don't care to see.  At the start of the document, however, is
the real content.  So, I don't particularly like the idea of having
settings etc. *before* the content in my files.

Just by 2¢.

-- 
Eric S Fraga via Emacs 27.0.50, Org release_9.2.3-379-gff2bf2

Re: Proposal for new document-level syntax

[Achim Gratz]

Fraga, Eric writes:
> For me, however, your proposed structure would clash strongly with my
> usual working practice.  Specifically, I put all customizations and
> settings at the end of my documents, along with, for instance, Emacs
> file local variables.  These are things I do not change often and
> usually don't care to see.  At the start of the document, however, is
> the real content.  So, I don't particularly like the idea of having
> settings etc. *before* the content in my files.

If the customization data isn't visible by default, does that address
your concerns?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Proposal for new document-level syntax

[Fraga, Eric]

On Saturday,  1 Jun 2019 at 15:45, Achim Gratz wrote:
> If the customization data isn't visible by default, does that address
> your concerns?

It might indeed.  Depends on how intrusive it is and how easy it is to
manage.

But having said this, and my previous email, don't take my views too
much into account in your development of this proposal.  My use cases
for org generally are not requiring the type of changes you are
suggesting.  I do understand your motivation but I tend to work with
small numbers of large files instead of the other way around.  A given
project will typically be contained in a single file that that file
will, for instance, have an "* [/] TODO Actions" headline etc.

-- 
Eric S Fraga via Emacs 27.0.50, Org release_9.2.3-379-gff2bf2

Re: Proposal for new document-level syntax

[Gustav Wikström]

Hi Eric!

> -----Original Message-----
> From: Fraga, Eric <e.fraga@ucl.ac.uk>
> Sent: den 1 juni 2019 15:18
> To: Gustav Wikström <gustav@whil.se>
> Cc: emacs-orgmode@gnu.org
> Subject: Re: [O] Proposal for new document-level syntax
> 
> For me, however, your proposed structure would clash strongly with my
> usual working practice.  Specifically, I put all customizations and
> settings at the end of my documents, along with, for instance, Emacs
> file local variables.  These are things I do not change often and
> usually don't care to see.  At the start of the document, however, is
> the real content.  So, I don't particularly like the idea of having
> settings etc. *before* the content in my files.

Point taken. File local variables potitions would not be affected of
course. And to be honest, I don't see any existing keyword going away
any time soon either. So your workflow probably won't be affected for
a long time. Unless you choose to!

With that said, I'd still like to argue a bit more for the structure
I've presented. The idea with having drawers is exactly the point
you're also making; that it makes configuration go out of the way for
real content. I question the disctraction 1-2 rows provide. Take for
example the way current property drawer and planning-properties work
on outline-nodes. They have fixed positions at the top of the node and
take up maximum two rows when declared. Not really an issue in my book
and the benefit of fixed positions is easy parsing for both humans and
the machine. The same would apply for document settings and
properties. Take the example below:

#+begin_example
  :SETTINGS:...
  :PROPERTIES:...
  ,* Heading 1
  DEADLINE: <2019-06-02 Sun> SCHEDULED: <2019-06-01 Sat>
  :PROPERTIES:...
  lorem

  ,* Heading 2
  Ipsum

  ,# -*- mode: org -*-
#+end_example

To me it makes perfect sense to have the drawers at the top at fixed
positions. It follows current convention for headlines. At most two
rows before the real content is hopefully a compromize that can be
made for syntactic clarity and simplicity. And personally it would be
a great improvement over the current way document-level keywords work.
> 
> Just by 2¢.

Thanks!

Re: Proposal for new document-level syntax

[Nicolas Goaziou]

Hello,

Gustav Wikström <gustav@whil.se> writes:

>   I propose to allow properties to be defined also as document property
>   keywords. All keywords in the top of a buffer, before any non-comment
>   line, are document-level keywords. In effect, they are properties that
>   apply in exactly the same way as properties defined in the property
>   drawer. The only reason for using a document keyword instead of
>   defining it inside the property drawer is to make it more visible. One
>   example would be the title-keword (#+TITLE: ...).

I only skimmed over your proposal, as I don't have much time. I'm sorry
if you answered this already, but I don't understand how what you
suggest would differ from regular keywords.

As a reminder:
- you can create, and support, as many keywords as you want;
- keywords all operate at the "document-level";
- keywords can be located anywhere in the document.

I think Org keywords already provide everything you need. If they don't,
I would suggest to improve them instead of creating something else.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: Proposal for new document-level syntax

[Gustav Wikström]

Hi Nicolas,

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Sent: den 1 juni 2019 23:01
> To: Gustav Wikström <gustav@whil.se>
> Cc: emacs-orgmode@gnu.org
> Subject: Re: Proposal for new document-level syntax
> 
> I only skimmed over your proposal, as I don't have much time. I'm sorry
> if you answered this already, but I don't understand how what you
> suggest would differ from regular keywords.

No worries. I think I explained but it can be further detailed. What I
mean is that any property you can think of should be possible to add
to a document as a keyword with the syntax:

#+{PROPERTY}: {Value}

But only at the beginning of a file, before any other content. The
only reason for defining properties like that is for them to be
visible outside of the property drawer. I'm thinking mostly of
=#+TITLE= and similar keywords.

I'd like to depricate =#+PROPERTY:= since it breaks the outline
hierarchy and doens't follow the convention for how properties are
defined inside headlines. Removing the "old" way of defining
properties for the whole buffer will make property-syntax defined the
same for documents and headlines. With the slight extention of
allowing arbitrary keywords to stand for properties at the beginning
of the buffer. Note that we already have "document property keywords"
in org-mode. Less limited since they're not positionally contained.
And only for a limited set of keywords; the "export keywords". (See
[[info:org#Export Settings]]) 

The proposal has a side-effect of reducing the complexity for
keywords.[fn:1] I did an evaluation of available keywords in org-mode
today by going through the documentation. And there are *lots* of uses
for keywords today. Making keywords easier to use, by removing the
need for them in many cases, hopefully will help future org-moders.

[fn:1] All keywords that have outline property equivalences can be
deprecated and later removed since we're introducing a property drawer
on the document level. All keywords that are meant for customizations
that don't have an outline property equivalent can be *moved* into the
setting drawer. That only leaves keywords that have real meaning in
the outline. =#+NAME=, =#+TOC=, =#+INCLUDE=, for example.

> 
> As a reminder:
> - you can create, and support, as many keywords as you want;
> - keywords all operate at the "document-level";
> - keywords can be located anywhere in the document.
> 
> I think Org keywords already provide everything you need. If they
> don't, I would suggest to improve them instead of creating something
> else.

In my opinion property drawers is the improvement which in time will 
make the existing property-keyword redundant.

With regards,
Gustav

Re: Proposal for new document-level syntax

[Nicolas Goaziou]

Hello,

Gustav Wikström <gustav@whil.se> writes:

> No worries. I think I explained but it can be further detailed. What I
> mean is that any property you can think of should be possible to add
> to a document as a keyword with the syntax:
>
> #+{PROPERTY}: {Value}

IIRC, Org uses

  #+PROPERTY: {key} {value}

Why "should" it be possible to use a different syntax?

> But only at the beginning of a file, before any other content. The
> only reason for defining properties like that is for them to be
> visible outside of the property drawer. I'm thinking mostly of
> =#+TITLE= and similar keywords.
>
> I'd like to depricate =#+PROPERTY:= since it breaks the outline
> hierarchy and doens't follow the convention for how properties are
> defined inside headlines.

So the reason for this change is that keywords break the outline
hierarchy? Well, keywords do not belong to the outline hierarchy in the
first place. But syntax is not very different, either.

> Removing the "old" way of defining properties for the whole buffer
> will make property-syntax defined the same for documents and
> headlines. With the slight extention of allowing arbitrary keywords to
> stand for properties at the beginning of the buffer. Note that we
> already have "document property keywords" in org-mode. Less limited
> since they're not positionally contained. And only for a limited set
> of keywords; the "export keywords". (See [[info:org#Export Settings]])

"Document property keyword" has no syntactical meaning. It is used for
fontification.

> In my opinion property drawers is the improvement which in time will 
> make the existing property-keyword redundant.

I still don't get how this is an improvement. What would you be able to
do with properties drawers that you cannot do currently with regular
keywords? This is a genuine question: I don't want to turn down your
suggestion, but I think it entails a lot of changes, and I want to be
sure there is a real benefit to it.


Regards,

-- 
Nicolas Goaziou

Re: Proposal for new document-level syntax

[Gustav Wikström]

Hi again,

This became a bit long, so the most important thing is repeated here
in the top:

I think there is merit to the idea of a file/document/buffer being
seen as a level-0 node. I think we'd benefit by adhering to the
principles for outline nodes also for the document-node (level-0 node)
as much as possible.

I'd like the specification for org-mode to be clear and succinct and
it's features to be as congruent as possible.

Comments to your concerns and questions below!

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Sent: den 3 juni 2019 22:40
> To: Gustav Wikström <gustav@whil.se>
> Cc: emacs-orgmode@gnu.org
> Subject: Re: Proposal for new document-level syntax
> 
> Hello,
> 
> Gustav Wikström <gustav@whil.se> writes:
> 
> > No worries. I think I explained but it can be further detailed. What I
> > mean is that any property you can think of should be possible to add
> > to a document as a keyword with the syntax:
> >
> > #+{PROPERTY}: {Value}
> 
> IIRC, Org uses
> 
>   #+PROPERTY: {key} {value}
> 
> Why "should" it be possible to use a different syntax?

The main change I want to introduce is the property-drawer. The new
property-syntax is secondary but makes 100% sense when you start
thinking about it. If you read through my initial mail you'll see that
I'm trying to consolidate functionality, moving things from keywords
possibly scattered throughout a document into drawers. That
consolidation is to simplify org-mode, even though it for a while will
make the /actual implementation/ more complex to maintain backwards
compatability. There is a longer investigation behind this that I
probably should share. But given the already long text and what I
understand is a lack of time to read it I hesitate to wall-of-text you
guys more than I'm already doing! :) So I'm taking things
incrementally when questions arise!

To the point; the =#+{PROPERTY}: {Value}= syntax idea started with
thinking about the purpose of existing export keywords. [fn:1] They're
not /only/ valuable for exporting. Defining a title for a document
(TITLE is an export-keyword) makes sense even if you don't export a
document. And what is that title-keyword really? It's a property! You
can define it in an outline as well! [fn:2]

Thinking of TITLE as a property, and thinking of depricating the
export keywords (i.e. remove that concept but maintain functionality
with properties) lead to the problem of TITLE in the future being
hidden away in a drawer. Being a user of the short and easy syntax for
export keywords today, my idea was to generalize them to work for all
properties. But contain them to a fixed position above the document
property drawer since the only use of that kind of property-definition
is to visually highlight it while letting all other document
properties be hidden away in the drawer.

What I've done is to try to not get stuck in existing syntax and think
more freely what would benefit us long term. I hope you can appreciate
that. It will indeed create another way of defining properties on
document-level . But I claim it makes sense! And I claim that the
current way of defining document properties with the

  #+PROPERTY: {key} {value}

doesn't make that much sense when we have document property drawers in
place! I do appreciate the concern of making breaking changes, which
is the reason for not proposing the removal of existing document
property syntax. That maybe could be saved for a major version
sometime in the future, when the drawer has been tried and tested
more, and hopefully accepted more! (I honestly didn't think I'd get
resistance for trying to introduce it...)

> > But only at the beginning of a file, before any other content. The
> > only reason for defining properties like that is for them to be
> > visible outside of the property drawer. I'm thinking mostly of
> > =#+TITLE= and similar keywords.
> >
> > I'd like to depricate =#+PROPERTY:= since it breaks the outline
> > hierarchy and doens't follow the convention for how properties are
> > defined inside headlines.
> 
> So the reason for this change is that keywords break the outline
> hierarchy? Well, keywords do not belong to the outline hierarchy in the
> first place. But syntax is not very different, either.

No no no, you clearly have not read my first mail. Please do. Keywords
breaking the outline hierarchy is only one of the reasons. What you're
saying is actually exactly what I mean with them breaking the outline.
I'm just a bit harsher and more strict - If keywords don't belong to
the outline, then don't put them inside headlines in the outline! They
don't belong there! Also, as I'm sure you're aware since you are the
most knowledgable org-mode'er out there, the following keywords *do*
belong in the outline: 
  #+NAME 
  #+TOC 
  #+ASCII
  #+INCLUDE

What I'm saying is basically that all keywords that "break the
outline" (or in your words: don't belong to the outline hierarchy),
can - and should! - be put into drawers before the first headline.
That way we don't even have to argue about what it means to belong or
not belong to the outline. One can claim that they belong to level-0
in the outline or one could claim that they have nothing to do with
the outline. Both perspectives are fine.

> > Removing the "old" way of defining properties for the whole buffer
> > will make property-syntax defined the same for documents and
> > headlines. With the slight extention of allowing arbitrary keywords to
> > stand for properties at the beginning of the buffer. Note that we
> > already have "document property keywords" in org-mode. Less limited
> > since they're not positionally contained. And only for a limited set
> > of keywords; the "export keywords". (See [[info:org#Export Settings]])
> 
> "Document property keyword" has no syntactical meaning. It is used for
> fontification.

"Document property keywords" of today you mean? The export keywords?
It's a bit harsh to say it has no syntactical meaning. Ofc. it has -
its keywords that set properties on the document level! I agree that
it doesn't apply 100% correctly today for export keywords. But the
reason it doesn't seems to me to be an oversight, since we have the
export_{export-keyword} properties that do set properties. It's just a
bit convoluted. And /one of the reasons/ for proposing this change is
to try to clean things up, this included.

> I still don't get how this is an improvement. What would you be able to
> do with properties drawers that you cannot do currently with regular
> keywords? This is a genuine question: I don't want to turn down your
> suggestion, but I think it entails a lot of changes, and I want to be
> sure there is a real benefit to it.

My text above and the first mail do give arguments on why... But to
try to give some more:

- I think there is merit to the idea of a document being seen as a
  level-0 node. I think we'd benefit by adhering to the principles for
  outline nodes also for the document-node (level-0 node) as much as
  possible.

- I'd like the specification for org-mode to be clear and succinct and
  it's features to be as congruent as possible.

As a final note, I want to say that code already exist for
document-properties. It works and it's not terribly complicated,
although I'm sure you'll have comments on my code!

[fn:1] [[info:org#Export Settings]]

[fn:2] Unfortunately for export-keywords, the property-syntax differs
a bit from the keywords and they need the prefix =export_= to work.
That's surely something that can be fixed to align things though.


Kind regards, and respectfully
Gustav Wikström

Re: Proposal for new document-level syntax

[Norman Walsh]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Gustav Wikström <gustav@whil.se> writes:
>   I propose a "document" element in org-element, a property-drawer on
>   document-level, a setting-drawer on document-level and
>   property-keywords (slightly different than what already exist). And
>   would like your comments regarding that!

When I started searching for ways to put arbitrary metadata on org
documents, I was very surprised to find that there didn’t appear to be
support for a top-level properties drawer. This seems like a good idea
to me.

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | If you are losing your leisure, look
http://nwalsh.com/            | out! You may be losing your
                              | soul.--Logan Pearsall Smith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Proposal for new document-level syntax

[Samuel Wales]

if the question is properties and settings, and not anything else,
then perhaps a solution could be to put them on a dedicated header,
and then the user can move them to top or bottom as desired.  the name
of this meta or level 0 header could be set in a variable.

perhaps alraedy considered or suggested.


On 6/2/19, Norman Walsh <ndw@nwalsh.com> wrote:
> Gustav Wikström <gustav@whil.se> writes:
>>   I propose a "document" element in org-element, a property-drawer on
>>   document-level, a setting-drawer on document-level and
>>   property-keywords (slightly different than what already exist). And
>>   would like your comments regarding that!
>
> When I started searching for ways to put arbitrary metadata on org
> documents, I was very surprised to find that there didn’t appear to be
> support for a top-level properties drawer. This seems like a good idea
> to me.
>
>                                         Be seeing you,
>                                           norm
>
> --
> Norman Walsh <ndw@nwalsh.com> | If you are losing your leisure, look
> http://nwalsh.com/            | out! You may be losing your
>                               | soul.--Logan Pearsall Smith
>


-- 
The Kafka Pandemic

What is misopathy?
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

Re: Proposal for new document-level syntax

[Marco Wahl]

Norman Walsh <ndw@nwalsh.com> writes:

> Gustav Wikström <gustav@whil.se> writes:
>>   I propose a "document" element in org-element, a property-drawer on
>>   document-level, a setting-drawer on document-level and
>>   property-keywords (slightly different than what already exist). And
>>   would like your comments regarding that!
>
> When I started searching for ways to put arbitrary metadata on org
> documents, I was very surprised to find that there didn’t appear to be
> support for a top-level properties drawer. This seems like a good idea
> to me.

+1

BTW being able to write lines like 

    #title: foo title

*anywhere* in the file to produce that title always astonished me.


Ciao,  Marco