emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [RFC] Org document concept + document property drawers
@ 2019-08-31 18:49 Gustav Wikström
  2019-09-01  5:20 ` Adam Porter
                   ` (2 more replies)
  0 siblings, 3 replies; 13+ messages in thread
From: Gustav Wikström @ 2019-08-31 18:49 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org; +Cc: Carsten Dominik, Nicolas Goaziou

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Hi!

I'm continuing on my proposal to introduce a "document" element in
org-mode and the idea of seeing everything before the first headline
as the base level 0 outline for a file. I've attached two patches that
I'd like some public review of before pushing to master.

Patch 0001 introduces the document element into org-element.el, and
some restructuring related to that.

Patch 0002 makes it possible to use property drawers at the document
level. I've hopefully covered all related commands to make this work.
And I've added a bunch of tests to guard against future regressions.

Waiting for your comments!

Kind regards Gustav

[-- Attachment #2: 0001-New-top-level-document-concept-and-minor-restructure.patch --]
[-- Type: application/octet-stream, Size: 26795 bytes --]

From 2c28183f8803d813970fcbf030c9001300544480 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gustav=20Wikstr=C3=B6m?= <gustav@whil.se>
Date: Sun, 26 May 2019 21:46:31 +0200
Subject: [PATCH 1/2] New top level document concept and minor restructure for
 keywords

Top-level document element and accompanying modifications through the
system to make it work and fit together nicely.

Also some restructuring in code that works with keywords, since most
keywords relate to the document and it helps to make it explicit in
code which keywords do that and which keywords don't.

* lisp/org.el

** (new) org-keyword-regexp
Addition of regexp for keywords.  Let's try to use parameters and
constants instead of hardcoding these things all over the place...

** (removed) org-options-keywords.
This naming of this list is incongruent with the manual and the
concepts used throughout org-mode.  It also fits better in
org-element. So I'm removing this list in favour of new lists for
keywords in that file.

** (renamed, modified) org--setup-collect-keywords -> org-collect-keywords
Renamed and generalized org--setup-collect-keywords to make it work
for multiple purposes.  Is not limited to a fixed set of keywords any
longer.  New name: org-collect-keywords.

** (renamed, modified) org-make-options-regexp -> org-make-keyword-regexp
The EXTRA argument is never used so it is removed.  No need for
potential (or old?) functionatlity that adds complexity.  Can be used
together with org-collect-keywords to speed up processing.

** (renamed) org-set-regexps-and-options -> org-set-regexps-and-keywords
Renamed to more propertly match what it is doing.

* lisp/org-element.el
Addition of document as a concept and as a top-level node in the
content of org-data.  The content of a document element is everything
that previously was the content of org-data.

** org-element-*-keywords
Instead of org-options-keywords, a couple of lists are added to
org-element for keywords.  Multiple lists are used to indicate
different use-cases for particular sets of keywords.

*** (new) org-element-document-keywords
Keywords in this constant are those that apply for the whole document.

*** (new) org-element-export-keywords
List of keywords branded as export keywords.  These keywords are
supposed to be supported by all export backends.

*** (new) org-element-inline-keywords
List of keywords that provide content at a certain position in the
outline.

*** (new) org-element--get-document-keywords
Parses a document for keywords that are not dependent on their
position and should be regarded as applying to the whole document.

Return document and document export keywords based on reworked
function in org.el; org-collect-keywords.

** (modified) org-element-keyword-parser
Uses (new) org-keyword-regexp instead of hardcoding it's own regexp.

* lisp/ox.el
Fixing code so that it respects the new document element and uses
org-element functions instead of parsing org-element-contents itself.

* lisp/ob-core.el
Unused declaration for function, org-make-options-regexp, is removed.
It is removed since the function was renamed in lisp/org.el and found
here due to a dependency check.

* lisp/org-pcomplete.el
Cascading fixes due to cleanup in org and org-element.

* testing/lisp/test-org-element.el
Fix existing tests so they care for the document element.

* org.el
---
 lisp/ob-core.el                  |   1 -
 lisp/org-element.el              | 122 +++++++++++++++++++++++++------
 lisp/org-pcomplete.el            |  13 ++--
 lisp/org.el                      |  59 ++++++++-------
 lisp/ox.el                       |  33 +++++----
 testing/lisp/test-org-element.el |   4 +-
 testing/lisp/test-org.el         |   2 +-
 7 files changed, 159 insertions(+), 75 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 97ec18fd1..a8366ee33 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -66,7 +66,6 @@
 (declare-function org-list-to-generic "org-list" (LIST PARAMS))
 (declare-function org-list-to-lisp "org-list" (&optional delete))
 (declare-function org-macro-escape-arguments "org-macro" (&rest args))
-(declare-function org-make-options-regexp "org" (kwds &optional extra))
 (declare-function org-mark-ring-push "org" (&optional pos buffer))
 (declare-function org-narrow-to-subtree "org" ())
 (declare-function org-next-block "org" (arg &optional backward block-regexp))
diff --git a/lisp/org-element.el b/lisp/org-element.el
index 56b3cc413..cdd3eb67f 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -41,7 +41,10 @@
 ;; The next part implements a parser and an interpreter for each
 ;; element and object type in Org syntax.
 ;;
-;; The following part creates a fully recursive buffer parser.  It
+;; Org-element can parse org-mode documents.  The top-node in the
+;; parse-tree will always have TYPE `org-data' and PROPERTIES nil.
+;;
+;; The following part creates a fully recursive org-mode parser.  It
 ;; also provides a tool to map a function to elements or objects
 ;; matching some criteria in the parse tree.  Functions of interest
 ;; are `org-element-parse-buffer', `org-element-map' and, to a lesser
@@ -92,6 +95,7 @@
 (defvar org-edit-src-content-indentation)
 (defvar org-emph-re)
 (defvar org-emphasis-regexp-components)
+(defvar org-keyword-regexp)
 (defvar org-keyword-time-not-clock-regexp)
 (defvar org-match-substring-regexp)
 (defvar org-odd-levels-only)
@@ -220,8 +224,8 @@ specially in `org-element--object-lex'.")
   (org-element-cache-reset 'all))
 
 (defconst org-element-all-elements
-  '(babel-call center-block clock comment comment-block diary-sexp drawer
-	       dynamic-block example-block export-block fixed-width
+  '(babel-call center-block clock comment comment-block diary-sexp document
+	       drawer dynamic-block example-block export-block fixed-width
 	       footnote-definition headline horizontal-rule inlinetask item
 	       keyword latex-environment node-property paragraph plain-list
 	       planning property-drawer quote-block section
@@ -229,9 +233,9 @@ specially in `org-element--object-lex'.")
   "Complete list of element types.")
 
 (defconst org-element-greater-elements
-  '(center-block drawer dynamic-block footnote-definition headline inlinetask
-		 item plain-list property-drawer quote-block section
-		 special-block table)
+  '(center-block document drawer dynamic-block footnote-definition headline
+		 inlinetask item plain-list property-drawer quote-block
+		 section special-block table)
   "List of recursive element types aka Greater Elements.")
 
 (defconst org-element-all-objects
@@ -250,6 +254,27 @@ specially in `org-element--object-lex'.")
   (append org-element-recursive-objects '(paragraph table-row verse-block))
   "List of object or element types that can directly contain objects.")
 
+(defconst org-element-document-keywords
+  '("ARCHIVE" "BIND" "CATEGORY" "COLUMNS" "CONSTANTS" "FILETAGS" "LINK"
+    "MACRO" "OPTIONS" "PRIORITIES" "PROPERTY" "SEQ_TODO" "SETUPFILE"
+    "STARTUP" "TAGS" "TODO" "TYP_TODO")
+  "List of supported keywords that are used to configure document
+  global options and properties.")
+
+(defconst org-element-export-keywords
+  '("AUTHOR" "CREATOR" "DATE" "EMAIL" "EXCLUDE_TAGS" "LANGUAGE"
+    "SELECT_TAGS" "TITLE" "EXPORT_FILE_NAME")
+  "List of keywords that are mentioned in the manual as something
+  that should be supported by all export backends.  More keywords
+  might exist that are specific for certain backends.")
+
+(defconst org-element-inline-keywords
+  '("INCLUDE" "TOC" "INDEX" "ASCII" "HTML" "LATEX" "ODT" "TEXINFO" "BEAMER")
+  "List of keywords that provide content at certain positions in
+  the outline.  These keywords are considered as a part of the
+  document outline rather than as properties of the document
+  element.")
+
 (defconst org-element-affiliated-keywords
   '("CAPTION" "DATA" "HEADER" "HEADERS" "LABEL" "NAME" "PLOT" "RESNAME" "RESULT"
     "RESULTS" "SOURCE" "SRCNAME" "TBLNAME")
@@ -726,6 +751,51 @@ Assume point is at the beginning of the block."
 CONTENTS is the contents of the element."
   (format "#+begin_center\n%s#+end_center" contents))
 
+;;;; Document
+
+(defun org-element--get-document-keywords ()
+  "Return keywords that associate with the document.
+Returns a plist with one key, `:document-keywords', and an alist
+of all keywords related to the whole document with their
+corresponding values.  Does not expand setupfile keywords to get
+inherited properties."
+  (let ((document-keywords (org-collect-keywords
+			    (org-make-keyword-regexp
+			     org-element-document-keywords)
+			    nil nil t))
+	(export-keywords (org-collect-keywords
+			  (org-make-keyword-regexp
+			   org-element-export-keywords)
+			  nil nil t)))
+    (list :document-keywords document-keywords
+	  :export-keywords export-keywords)))
+
+(defun org-element-document-parser ()
+  "Parse a document for it's settings and properties.
+Return a list whose CAR is `document' and CDR is a plist
+containing `:buffer', `:file', `:level', `:contents-begin',
+`:contents-end' and `:post-blank'.
+
+In addition to the above, the plist also contains configurations
+and properties that applies for the whole document, coming from
+document keywords."
+  (save-excursion
+    (list 'document
+	  (nconc
+	   (list :buffer (current-buffer)
+		 :file buffer-file-name
+		 :level 0
+		 :contents-begin (point-min)
+		 :contents-end (point-max)
+		 :begin (point-min)
+		 :end (point-max)
+		 :post-blank 0)
+	   (org-element--get-document-keywords)))))
+
+(defun org-element-document-interpreter (_ contents)
+  "Interpret document element as Org syntax.
+CONTENTS is the contents of the element."
+  contents)
 
 ;;;; Drawer
 
@@ -2195,10 +2265,9 @@ containing `:key', `:value', `:begin', `:end', `:post-blank' and
     ;; this corner case.
     (let ((begin (or (car affiliated) (point)))
 	  (post-affiliated (point))
-	  (key (progn (looking-at "[ \t]*#\\+\\(\\S-*\\):")
+	  (key (progn (looking-at org-keyword-regexp)
 		      (upcase (match-string-no-properties 1))))
-	  (value (org-trim (buffer-substring-no-properties
-			    (match-end 0) (point-at-eol))))
+	  (value (org-trim (match-string-no-properties 2)))
 	  (pos-before-blank (progn (forward-line) (point)))
 	  (end (progn (skip-chars-forward " \r\t\n" limit)
 		      (if (eobp) (point) (line-beginning-position)))))
@@ -3847,7 +3916,7 @@ recursion.  Allowed values are `headline', `greater-element',
 nil), secondary values will not be parsed, since they only
 contain objects.
 
-Optional argument MODE, when non-nil, can be either
+Optional argument MODE, when non-nil, can be either `document',
 `first-section', `section', `planning', `item', `node-property'
 and `table-row'.
 
@@ -3863,6 +3932,9 @@ element it has to parse."
 	  ;; `org-element-secondary-value-alist'.
 	  (raw-secondary-p (and granularity (not (eq granularity 'object)))))
       (cond
+       ;; Document
+       ((eq mode 'document)
+	(org-element-document-parser))
        ;; Item.
        ((eq mode 'item)
 	(org-element-item-parser limit structure raw-secondary-p))
@@ -3875,6 +3947,8 @@ element it has to parse."
         (org-element-headline-parser limit raw-secondary-p))
        ;; Sections (must be checked after headline).
        ((eq mode 'section) (org-element-section-parser limit))
+       ;; First-section.  Special mode for dealing with the first section
+       ;; in a document, which might or might not be before first headline.
        ((eq mode 'first-section)
 	(org-element-section-parser
 	 (or (save-excursion (org-with-limited-levels (outline-next-heading)))
@@ -4121,9 +4195,10 @@ This function assumes that current major mode is `org-mode'."
     (org-skip-whitespace)
     (org-element--parse-elements
      (point-at-bol) (point-max)
-     ;; Start in `first-section' mode so text before the first
-     ;; headline belongs to a section.
-     'first-section nil granularity visible-only (list 'org-data nil))))
+     ;; Start in `document' mode so propeties and potential other
+     ;; content before the first headline belongs to the document
+     ;; node.
+     'document nil granularity visible-only (list 'org-data nil))))
 
 (defun org-element-parse-secondary-string (string restriction &optional parent)
   "Recursively parse objects in STRING and return structure.
@@ -4324,11 +4399,12 @@ looking into captions:
   "Return next special mode according to TYPE, or nil.
 TYPE is a symbol representing the type of an element or object
 containing next element if PARENTP is non-nil, or before it
-otherwise.  Modes can be either `first-section', `item',
-`node-property', `planning', `property-drawer', `section',
-`table-row' or nil."
+otherwise.  Modes can be either `document', `first-section',
+`item', `node-property', `planning', `property-drawer',
+`section', `table-row' or nil."
   (if parentp
       (pcase type
+	(`document 'first-section)
 	(`headline 'section)
 	(`inlinetask 'planning)
 	(`plain-list 'item)
@@ -4346,8 +4422,8 @@ otherwise.  Modes can be either `first-section', `item',
   "Parse elements between BEG and END positions.
 
 MODE prioritizes some elements over the others.  It can be set to
-`first-section', `section', `planning', `item', `node-property'
-or `table-row'.
+`document', `first-section', `section', `planning', `item',
+`node-property' or `table-row'.
 
 When value is `item', STRUCTURE will be used as the current list
 structure.
@@ -4375,7 +4451,8 @@ Elements are accumulated into ACC."
 	(let* ((element (org-element--current-element
 			 end granularity mode structure))
 	       (type (org-element-type element))
-	       (cbeg (org-element-property :contents-begin element)))
+	       (cbeg (org-element-property :contents-begin element))
+	       (cend (org-element-property :contents-end element)))
 	  (goto-char (org-element-property :end element))
 	  ;; Visible only: skip invisible parts between siblings.
 	  (when (and visible-only (org-invisible-p2))
@@ -4392,9 +4469,10 @@ Elements are accumulated into ACC."
 		 (or (memq granularity '(element object nil))
 		     (and (eq granularity 'greater-element)
 			  (eq type 'section))
-		     (eq type 'headline)))
+		     (eq type 'headline)
+		     (eq type 'document)))
 	    (org-element--parse-elements
-	     cbeg (org-element-property :contents-end element)
+	     cbeg cend
 	     ;; Possibly switch to a special mode.
 	     (org-element--next-mode type t)
 	     (and (memq type '(item plain-list))
@@ -4404,7 +4482,7 @@ Elements are accumulated into ACC."
 	   ;; GRANULARITY allows it.
 	   ((memq granularity '(object nil))
 	    (org-element--parse-objects
-	     cbeg (org-element-property :contents-end element) element
+	     cbeg cend element
 	     (org-element-restriction type))))
 	  (push (org-element-put-property element :parent acc) elements)
 	  ;; Update mode.
diff --git a/lisp/org-pcomplete.el b/lisp/org-pcomplete.el
index 5a7fb57c9..8ba9c0714 100644
--- a/lisp/org-pcomplete.el
+++ b/lisp/org-pcomplete.el
@@ -51,6 +51,9 @@
 (defvar org-default-priority)
 (defvar org-drawer-regexp)
 (defvar org-element-affiliated-keywords)
+(defvar org-element-document-keywords)
+(defvar org-element-export-keywords)
+(defvar org-element-inline-keywords)
 (defvar org-entities)
 (defvar org-export-default-language)
 (defvar org-export-exclude-tags)
@@ -60,7 +63,6 @@
 (defvar org-link-abbrev-alist)
 (defvar org-link-abbrev-alist-local)
 (defvar org-lowest-priority)
-(defvar org-options-keywords)
 (defvar org-outline-regexp)
 (defvar org-property-re)
 (defvar org-startup-options)
@@ -197,10 +199,11 @@ When completing for #+STARTUP, for example, this function returns
   (require 'org-element)
   (pcomplete-here
    (org-pcomplete-case-double
-    (append (mapcar (lambda (keyword) (concat keyword " "))
-		    org-options-keywords)
-	    (mapcar (lambda (keyword) (concat keyword ": "))
-		    org-element-affiliated-keywords)
+    (append (mapcar (lambda (keyword) (concat keyword ": "))
+		    (append org-element-document-keywords
+			    org-element-export-keywords
+			    org-element-inline-keywords
+			    org-element-affiliated-keywords))
 	    (let (block-names)
 	      (dolist (name
 		       '("CENTER" "COMMENT" "EXAMPLE" "EXPORT" "QUOTE" "SRC"
diff --git a/lisp/org.el b/lisp/org.el
index db957a112..358989070 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -350,6 +350,9 @@ FULL is given."
 
 \f
 ;;; Syntax Constants
+;;;; Keyword
+(defconst org-keyword-regexp "^[ \t]*#\\+\\(\\S-+?\\):[ \t]*\\(.*\\)$"
+  "Regular expression for keyword-lines")
 
 ;;;; Block
 
@@ -4304,13 +4307,13 @@ See `org-tag-alist' for their structure."
       ;; Preserve order of ALIST1.
       (append (nreverse to-add) alist2)))))
 
-(defun org-set-regexps-and-options (&optional tags-only)
-  "Precompute regular expressions used in the current buffer.
+(defun org-set-regexps-and-keywords (&optional tags-only)
+  "Precompute regular expressions and set variables based on keywords.
 When optional argument TAGS-ONLY is non-nil, only compute tags
 related expressions."
   (when (derived-mode-p 'org-mode)
-    (let ((alist (org--setup-collect-keywords
-		  (org-make-options-regexp
+    (let ((alist (org-collect-keywords
+		  (org-make-keyword-regexp
 		   (append '("FILETAGS" "TAGS" "SETUPFILE")
 			   (and (not tags-only)
 				'("ARCHIVE" "CATEGORY" "COLUMNS" "CONSTANTS"
@@ -4465,16 +4468,17 @@ related expressions."
 		      "[ \t]*$"))
 	(org-compute-latex-and-related-regexp)))))
 
-(defun org--setup-collect-keywords (regexp &optional files alist)
-  "Return setup keywords values as an alist.
-
-REGEXP matches a subset of setup keywords.  FILES is a list of
-file names already visited.  It is used to avoid circular setup
-files.  ALIST, when non-nil, is the alist computed so far.
-
-Return value contains the following keys: `archive', `category',
-`columns', `constants', `filetags', `link', `priorities',
-`property', `scripts', `startup', `tags' and `todo'."
+(defun org-collect-keywords (&optional regexp files alist no-setupfile-expansion)
+  "Return keywords values as an alist.
+
+REGEXP can be provided to override default keyword regexp if only
+a subset of keywords are interesting, for performance purposes.
+FILES is a list of file names already visited.  It is used to
+avoid circular setup files.  ALIST, when non-nil, is the alist
+computed so far.  NO-SETUPFILE-EXPANSION can be used to not
+recurse into setupfiles.  In that case the `setupfile' keyword
+and value will be returned instead, if it exist."
+  (setq regexp (or regexp org-keyword-regexp))
   (org-with-wide-buffer
    (goto-char (point-min))
    (let ((case-fold-search t))
@@ -4554,7 +4558,9 @@ Return value contains the following keys: `archive', `category',
 		 (if todo (push value (cdr todo))
 		   (push (list 'todo value) alist))))
 	      ((equal key "SETUPFILE")
-	       (unless buffer-read-only ; Do not check in Gnus messages.
+	       (if (or buffer-read-only ; Do not check in Gnus messages.
+		       no-setupfile-expansion)
+		   (push (cons 'setupfile value) alist)
 		 (let ((f (and (org-string-nw-p value)
 			       (expand-file-name (org-strip-quotes value)))))
 		   (when (and f (file-readable-p f) (not (member f files)))
@@ -4565,8 +4571,10 @@ Return value contains the following keys: `archive', `category',
 			     ;; Fake Org mode to benefit from cache
 			     ;; without recurring needlessly.
 			     (let ((major-mode 'org-mode))
-			       (org--setup-collect-keywords
-				regexp (cons f files) alist)))))))))))))))
+			       (org-collect-keywords
+				regexp (cons f files) alist))))))))
+	      (t
+	       (push (cons (make-symbol (downcase key)) value) alist)))))))))
   alist)
 
 (defun org-tag-string-to-alist (s)
@@ -4843,7 +4851,7 @@ The following commands are available:
      (vconcat (mapcar (lambda (c) (make-glyph-code c 'org-ellipsis))
 		      org-ellipsis)))
     (setq buffer-display-table org-display-table))
-  (org-set-regexps-and-options)
+  (org-set-regexps-and-keywords)
   (org-set-font-lock-defaults)
   (when (and org-tag-faces (not org-tags-special-faces-re))
     ;; tag faces set outside customize.... force initialization.
@@ -9722,13 +9730,6 @@ keywords relative to each registered export back-end."
 	;; Back-end options.
 	(push (nth 1 option-entry) keywords)))))
 
-(defconst org-options-keywords
-  '("ARCHIVE:" "AUTHOR:" "BIND:" "CATEGORY:" "COLUMNS:" "CREATOR:" "DATE:"
-    "DESCRIPTION:" "DRAWERS:" "EMAIL:" "EXCLUDE_TAGS:" "FILETAGS:" "INCLUDE:"
-    "INDEX:" "KEYWORDS:" "LANGUAGE:" "MACRO:" "OPTIONS:" "PROPERTY:"
-    "PRIORITIES:" "SELECT_TAGS:" "SEQ_TODO:" "SETUPFILE:" "STARTUP:" "TAGS:"
-    "TITLE:" "TODO:" "TYP_TODO:" "SELECT_TAGS:" "EXCLUDE_TAGS:"))
-
 (defcustom org-structure-template-alist
   '(("a" . "export ascii")
     ("c" . "center")
@@ -15870,7 +15871,7 @@ When a buffer is unmodified, it is just killed.  When modified, it is saved
 	      (org-check-agenda-file file)
 	      (set-buffer (org-get-agenda-file-buffer file)))
 	    (widen)
-	    (org-set-regexps-and-options 'tags-only)
+	    (org-set-regexps-and-keywords 'tags-only)
 	    (setq pos (point))
 	    (or (memq 'category org-agenda-ignore-properties)
 		(org-refresh-category-properties))
@@ -21156,13 +21157,11 @@ modified."
 		      (org-do-remove-indentation))))))))
     (funcall unindent-tree (org-element-contents parse-tree))))
 
-(defun org-make-options-regexp (kwds &optional extra)
+(defun org-make-keyword-regexp (kwds)
   "Make a regular expression for keyword lines.
-KWDS is a list of keywords, as strings.  Optional argument EXTRA,
-when non-nil, is a regexp matching keywords names."
+KWDS is a list of keywords, as strings."
   (concat "^[ \t]*#\\+\\("
 	  (regexp-opt kwds)
-	  (and extra (concat (and kwds "\\|") extra))
 	  "\\):[ \t]*\\(.*\\)"))
 
 \f
diff --git a/lisp/ox.el b/lisp/ox.el
index 1b579f50e..592cb9100 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1741,13 +1741,15 @@ Return updated plist."
 DATA is parsed tree as returned by `org-element-parse-buffer'.
 OPTIONS is a plist holding export options."
   (catch 'exit
-    (let ((min-level 10000))
-      (dolist (datum (org-element-contents data))
-	(when (and (eq (org-element-type datum) 'headline)
-		   (not (org-element-property :footnote-section-p datum))
-		   (not (memq datum (plist-get options :ignore-list))))
-	  (setq min-level (min (org-element-property :level datum) min-level))
-	  (when (= min-level 1) (throw 'exit 1))))
+    (let ((ignore-list (plist-get options :ignore-list))
+          (min-level 10000))
+      (org-element-map data 'headline
+        (lambda (hl)
+          (when (and
+                 (not (org-element-property :footnote-section-p hl))
+                 (not (memq hl ignore-list)))
+            (setq min-level (min (org-element-property :level hl) min-level))
+            (when (= min-level 1) (throw 'exit 1)))))
       ;; If no headline was found, for the sake of consistency, set
       ;; minimum level to 1 nonetheless.
       (if (= min-level 10000) 1 min-level))))
@@ -1934,7 +1936,8 @@ not exported."
 INFO is a plist containing export directives."
   (let ((type (org-element-type blob)))
     ;; Return contents only for complete parse trees.
-    (if (eq type 'org-data) (lambda (_datum contents _info) contents)
+    (if (member type '(org-data document))
+	(lambda (_datum contents _info) contents)
       (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
 	(and (functionp transcoder) transcoder)))))
 
@@ -2755,9 +2758,11 @@ from tree."
     ;; If a select tag is active, also ignore the section before the
     ;; first headline, if any.
     (when selected
-      (let ((first-element (car (org-element-contents data))))
-	(when (eq (org-element-type first-element) 'section)
-	  (org-element-extract-element first-element))))
+      (org-element-map data 'document
+	(lambda (document)
+	  (let ((first-element (car (org-element-contents document))))
+	    (when (eq (org-element-type first-element) 'section)
+	      (org-element-extract-element first-element))))))
     ;; Prune tree and communication channel.
     (funcall walk-data data)
     (dolist (entry (append
@@ -3040,7 +3045,7 @@ Return code as a string."
 				parsed-keywords)
 	 ;; Refresh buffer properties and radio targets after previous
 	 ;; potentially invasive changes.
-	 (org-set-regexps-and-options)
+	 (org-set-regexps-and-keywords)
 	 (org-update-radio-target-regexp)
 	 ;;  Possibly execute Babel code.  Re-run a macro expansion
 	 ;;  specifically for {{{results}}} since inline source blocks
@@ -3049,7 +3054,7 @@ Return code as a string."
 	 (when org-export-use-babel
 	   (org-babel-exp-process-buffer)
 	   (org-macro-replace-all '(("results" . "$1")) parsed-keywords)
-	   (org-set-regexps-and-options)
+	   (org-set-regexps-and-keywords)
 	   (org-update-radio-target-regexp))
 	 ;; Run last hook with current back-end's name as argument.
 	 ;; Update buffer properties and radio targets one last time
@@ -3058,7 +3063,7 @@ Return code as a string."
 	 (save-excursion
 	   (run-hook-with-args 'org-export-before-parsing-hook
 			       (org-export-backend-name backend)))
-	 (org-set-regexps-and-options)
+	 (org-set-regexps-and-keywords)
 	 (org-update-radio-target-regexp)
 	 ;; Update communication channel with environment.
 	 (setq info
diff --git a/testing/lisp/test-org-element.el b/testing/lisp/test-org-element.el
index f2ab38031..2a1716134 100644
--- a/testing/lisp/test-org-element.el
+++ b/testing/lisp/test-org-element.el
@@ -198,7 +198,7 @@ Some other text
    (equal '(org-data nil)
 	  (org-test-with-temp-text "* Headline"
 	    (let* ((tree (org-element-parse-buffer))
-		   (element (org-element-map tree 'headline 'identity nil t)))
+		   (element (org-element-map tree 'document 'identity nil t)))
 	      (org-element-extract-element element)
 	      tree))))
   ;; Extract an element.
@@ -3653,7 +3653,7 @@ Text
 	      "* H1\n** H2\n#+BEGIN_CENTER\n*bold<point>*\n#+END_CENTER"
 	    (mapcar #'car (org-element-lineage (org-element-context))))))
   (should
-   (equal '(paragraph center-block section headline headline org-data)
+   (equal '(paragraph center-block section headline headline document org-data)
 	  (org-test-with-temp-text
 	      "* H1\n** H2\n#+BEGIN_CENTER\n*bold<point>*\n#+END_CENTER"
 	    (mapcar #'car
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 215d8fdb0..01e965ea7 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2095,7 +2095,7 @@ SCHEDULED: <2014-03-04 tue.>"
 ;;; Keywords
 
 (ert-deftest test-org/set-regexps-and-options ()
-  "Test `org-set-regexps-and-options' specifications."
+  "Test `org-set-regexps-and-keywords' specifications."
   ;; TAGS keyword.
   (should
    (equal '(("A"))
-- 
2.17.1


[-- Attachment #3: 0002-Org-document-property-drawers.patch --]
[-- Type: application/octet-stream, Size: 50647 bytes --]

From 59fa97efff2b8ee50bda55e63903cf7db1f4af2b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gustav=20Wikstr=C3=B6m?= <gustav@whil.se>
Date: Sun, 26 May 2019 22:13:09 +0200
Subject: [PATCH 2/2] Org document property-drawers

* doc/org-manual.org

Describe the new possibility to define property-blocks on document
level, in addition to at headline level.

* etc/ORG-NEWS

Mention the change

* lisp/org.el

Code is modified to work with the concept of a node-level of 0 for
all text before the first headline.  Property drawers on
document-level, i.e node-level 0, work exactly the same as property
drawers on headlines.

Property drawers on document level can (only) be defined at the top of
a document.

** (Renamed) org-file-properties -> org-keyword-properties
Due to the fact that properties can be defined for the whole document
using property drawers this local variable needs a rename to make its
name less ambigous.

** (Modified) org-refresh-properties
Made to work also before first headline.

** (Modified) org-refresh-property
Made to work also before first headline.

** (Modified) org-refresh-category-properties
Made to work before first headline due to property drawers being
available there now!

** (Modified) org-get-property-block
Make it work also before first headline.

** (New) org-at-property-block-p
New function to validate if point is at the start of a property
block.

** (Modified) org-entry-properties
Make it work before first headline

** (Renamed) org-property-global-value -> org-property-global-or-keyword-value
Renamed to make its name match its functionality.

** (Modified) org-entry-get-with-inheritance
Make it work also before first headline.

** (Modified) org-entry-put
Make it work also before first headline.

** (Modified) org-insert-property-drawer
Make it work also before first headline.

** (New) org-back-to-heading-or-point-min
New function to make a document work as a level 0 node in the outline.

** (New) org-at-keyword-p
Predicate function to answer to if we're currently at a keyword line
or not.

** (New) org-up-heading-or-point-min
New function to make a document work as a level 0 node in the outline.

** (Modified) org-end-of-subtree
Make it work for the whole buffer, not just inside headlines. The end
of the buffer "subtree" is the end of the file.

* lisp/org-element.el

Add properties both from property drawer and property keywords to
document element. Drawer has precedence if properties are defined in
both.

Add post-blank property to document element  post-element is according
to ducumentation a property that all elements should have.

** (new) org-element--get-document-properties
Collect properties for whole documents from the property keywords and
from the document property drawer.

** (modified) org-element-document-parser
Add document properties to the document parser.

** (renamed) org-element--get-node-properties ->
   org-element--get-headline-properties
With the introduction of document property drawers while still
maintaining the concept of `node-property' for a property within a
property drawer, the function to get properties from headlines
property drawers need to be renamed to not be confusing.

(I suppose the concept of `node-property' could be removed in favour
of the more general `property' concept.  I couldn't calculate the
side-effects of doing that though.  And it would mean making changes in
far more files)

** (modified) org-element--current-element
Can now detect property-blocks before first headline according to it's
positional rules.

* lisp/org-attach.el

** (Modified) org-attach
Make it possible to call the attachment dispatcher (org-attach) also
before the first headline, since document property drawers make
attachments possible for the whole document now.

* lisp/org-capture.el

Modified only due to rename of function in org.el.

* testing/lisp/test-org.el

Add tests to make sure the specification of document-level
property-blocks can be maintained over time!

Additions to existing:
- /insert-property-drawer
- /set-property
- /delete-property
- /delete-property-globally

Added:
- /at-property-p
- /at-property-block-p
- /get-property-block
- /entry-get
- /refresh-properties

* testing/examples/property-inheritance.org

Switch from property-keywords to a property-drawer in the testfile.
Functionality should be the same, but now using a document drawer
instead of property-keywords.

Reason for switching is that I'd like us to slowly depricate
property-keywords.

* testing/lisp/test-org-element.el

Addition to existing test definitions:
- /node-property
- /property-drawer-parser

* contrib/lisp/ox-taskjuggler.el

A comment is modified only due to rename of function in org.el.
---
 contrib/lisp/ox-taskjuggler.el            |   2 +-
 doc/org-manual.org                        |  27 +-
 etc/ORG-NEWS                              |   8 +
 lisp/org-attach.el                        |   2 +-
 lisp/org-capture.el                       |   4 +-
 lisp/org-element.el                       |  55 +++-
 lisp/org.el                               | 267 ++++++++++++-------
 testing/examples/property-inheritance.org |   6 +-
 testing/lisp/test-org-element.el          |  21 ++
 testing/lisp/test-org.el                  | 306 +++++++++++++++++++++-
 10 files changed, 576 insertions(+), 122 deletions(-)

diff --git a/contrib/lisp/ox-taskjuggler.el b/contrib/lisp/ox-taskjuggler.el
index 16a6a3a74..bc90b93a4 100644
--- a/contrib/lisp/ox-taskjuggler.el
+++ b/contrib/lisp/ox-taskjuggler.el
@@ -137,7 +137,7 @@
 ;;   :END:
 ;;
 ;;;; * TODO
-;;   - Look at org-file-properties, org-global-properties and
+;;   - Look at org-keyword-properties, org-global-properties and
 ;;     org-global-properties-fixed
 ;;   - What about property inheritance and org-property-inherit-p?
 ;;   - Use TYPE_TODO as an way to assign resources
diff --git a/doc/org-manual.org b/doc/org-manual.org
index eb0a1e939..3855b9543 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -4951,7 +4951,7 @@ with many examples, see [[*Matching tags and properties]].
 
 A property is a key-value pair associated with an entry.  Properties
 can be set so they are associated with a single entry, with every
-entry in a tree, or with every entry in an Org file.
+entry in a tree, or with the whole buffer.
 
 There are two main applications for properties in Org mode.  First,
 properties are like tags, but with a value.  Imagine maintaining
@@ -5015,8 +5015,13 @@ disks in a box like this:
   :END:
 #+end_example
 
-If you want to set properties that can be inherited by any entry in
-a file, use a line like:
+Properties can be inserted on buffer level.  That means they apply
+before the first headline and can be inherited by all entries in a
+file.  Property blocks defined before first headline needs to be
+located at the top of the buffer, allowing only comments and keywords
+above.
+
+Properties can also be defined using lines like:
 
 #+cindex: @samp{_ALL} suffix, in properties
 #+cindex: @samp{PROPERTY}, keyword
@@ -5081,7 +5086,9 @@ The following commands help to work with properties:
   #+findex: org-insert-drawer
   Insert a property drawer into the current entry.  The drawer is
   inserted early in the entry, but after the lines with planning
-  information like deadlines.
+  information like deadlines.  If before first headline the drawer is
+  inserted at the top of the drawer after any potential comments or
+  keywords.
 
 - {{{kbd(C-c C-c)}}} (~org-property-action~) ::
 
@@ -5304,11 +5311,6 @@ done by defining a column format line.
 :DESCRIPTION: Where defined, where valid?
 :END:
 
-To define a column format for an entire file, use a line like:
-
-#+cindex: @samp{COLUMNS}, keyword
-: #+COLUMNS: %25ITEM %TAGS %PRIORITY %TODO
-
 To specify a format that only applies to a specific tree, add
 a =COLUMNS= property to the top node of that tree, for example:
 
@@ -5319,6 +5321,13 @@ a =COLUMNS= property to the top node of that tree, for example:
    :END:
 #+end_example
 
+A =COLUMNS= property within a property drawer before first headline
+will apply to the entire file.  As an addition to property drawers,
+keywords can also be defined for an entire file using a line like:
+
+#+cindex: @samp{COLUMNS}, keyword
+: #+COLUMNS: %25ITEM %TAGS %PRIORITY %TODO
+
 If a =COLUMNS= property is present in an entry, it defines columns for
 the entry itself, and for the entire subtree below it.  Since the
 column definition is part of the hierarchical structure of the
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 089f62995..86fba544f 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -123,6 +123,14 @@ auto-commit attachments to git:
   one need to require the =org-attach-git= module in the startup.
 
 ** New features
+*** Property drawers before first headline, outline  level 0
+Property drawers will now work before first headline and org-mode is
+moving more towards making things before the first headline behave
+just as if it was at outline level 0.  Inheritance for properties will
+work also for this level.  In other words; defining things in a
+property drawer before the first headline will make them "inheritable"
+for all headlines.
+
 *** Add split-window-right option for src block edit window placement
 Given the increasing popularity of wide screen monitors, splitting
 horizontally may make more sense than splitting vertically.  An
diff --git a/lisp/org-attach.el b/lisp/org-attach.el
index 726085014..f1c9d69d8 100644
--- a/lisp/org-attach.el
+++ b/lisp/org-attach.el
@@ -243,7 +243,7 @@ Shows a list of commands and prompts for another key to execute a command."
       (when marker
 	(set-buffer (marker-buffer marker))
 	(goto-char marker))
-      (org-back-to-heading t)
+      (org-back-to-heading-or-point-min t)
       (save-excursion
 	(save-window-excursion
 	  (unless org-attach-expert
diff --git a/lisp/org-capture.el b/lisp/org-capture.el
index 4f97e17ea..67343e02a 100644
--- a/lisp/org-capture.el
+++ b/lisp/org-capture.el
@@ -1736,11 +1736,11 @@ The template may still contain \"%?\" for cursor positioning."
 			 (_ (error "Invalid `org-capture--clipboards' value: %S"
 				   org-capture--clipboards)))))
 		    ("p"
-		     ;; We remove file properties inherited from
+		     ;; We remove keyword properties inherited from
 		     ;; target buffer so `org-read-property-value' has
 		     ;; a chance to find allowed values in sub-trees
 		     ;; from the target buffer.
-		     (setq-local org-file-properties nil)
+		     (setq-local org-keyword-properties nil)
 		     (let* ((origin (set-marker (make-marker)
 						(org-capture-get :pos)
 						(org-capture-get :buffer)))
diff --git a/lisp/org-element.el b/lisp/org-element.el
index cdd3eb67f..ee5c7a735 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -73,6 +73,9 @@
 (require 'org-table)
 
 (declare-function org-at-heading-p "org" (&optional _))
+(declare-function org-before-first-heading-p "org")
+(declare-function org-at-comment-p "org" ())
+(declare-function org-at-keyword-p "org" ())
 (declare-function org-end-of-subtree "org" (&optional invisible-ok to-heading))
 (declare-function org-escape-code-in-string "org-src" (s))
 (declare-function org-find-visible "org" ())
@@ -770,6 +773,42 @@ inherited properties."
     (list :document-keywords document-keywords
 	  :export-keywords export-keywords)))
 
+(defun org-element--get-document-properties ()
+  "Return document properties associated to the whole document.
+Upcase property names to avoid confusion between properties
+obtained through property drawer and default properties from the
+parser (e.g. `:end' and :END:).  Return value is a plist."
+  (let ((keyword-properties
+	 (alist-get 'property
+		    (org-collect-keywords
+		     (org-make-keyword-regexp
+		      '("PROPERTY")))))
+	properties)
+    ;; Parse properties from keywords first.
+    (dolist (cons keyword-properties)
+      (setq properties
+	    (plist-put properties
+		       (intern (concat ":" (car cons)))
+		       (cdr cons))))
+    ;; If the document property drawer exist, let properties
+    ;; set there override properties from keywords.
+    (save-excursion
+      (goto-char (point-min))
+      (when (org-before-first-heading-p)
+	(while (and (org-at-comment-p) (bolp)) (forward-line))
+	(while (and (org-at-keyword-p) (bolp)) (forward-line)))
+      (when (looking-at org-property-drawer-re)
+	(forward-line)
+	(let ((end (match-end 0)))
+	  (while (< (line-end-position) end)
+	    (looking-at org-property-re)
+	    (setq properties
+		  (plist-put properties
+			     (intern (concat ":" (upcase (match-string 2))))
+			     (match-string-no-properties 3)))
+	    (forward-line)))))
+    properties))
+
 (defun org-element-document-parser ()
   "Parse a document for it's settings and properties.
 Return a list whose CAR is `document' and CDR is a plist
@@ -778,7 +817,7 @@ containing `:buffer', `:file', `:level', `:contents-begin',
 
 In addition to the above, the plist also contains configurations
 and properties that applies for the whole document, coming from
-document keywords."
+document keywords and the document property drawer."
   (save-excursion
     (list 'document
 	  (nconc
@@ -790,6 +829,7 @@ document keywords."
 		 :begin (point-min)
 		 :end (point-max)
 		 :post-blank 0)
+	   (org-element--get-document-properties)
 	   (org-element--get-document-keywords)))))
 
 (defun org-element-document-interpreter (_ contents)
@@ -1001,8 +1041,8 @@ CONTENTS is the contents of the footnote-definition."
 
 ;;;; Headline
 
-(defun org-element--get-node-properties ()
-  "Return node properties associated to headline at point.
+(defun org-element--get-headline-properties ()
+  "Return properties associated to headline at point.
 Upcase property names.  It avoids confusion between properties
 obtained through property drawer and default properties from the
 parser (e.g. `:end' and :END:).  Return value is a plist."
@@ -1088,7 +1128,7 @@ Assume point is at beginning of the headline."
 	   (archivedp (member org-archive-tag tags))
 	   (footnote-section-p (and org-footnote-section
 				    (string= org-footnote-section raw-value)))
-	   (standard-props (org-element--get-node-properties))
+	   (standard-props (org-element--get-headline-properties))
 	   (time-props (org-element--get-time-properties))
 	   (end (min (save-excursion (org-end-of-subtree t t)) limit))
 	   (contents-begin (save-excursion
@@ -1233,7 +1273,7 @@ Assume point is at beginning of the inline task."
 		       (and (re-search-forward org-outline-regexp-bol limit t)
 			    (looking-at-p "[ \t]*END[ \t]*$")
 			    (line-beginning-position))))
-	   (standard-props (and task-end (org-element--get-node-properties)))
+	   (standard-props (and task-end (org-element--get-headline-properties)))
 	   (time-props (and task-end (org-element--get-time-properties)))
 	   (contents-begin (and task-end
 				(< (point) task-end)
@@ -3984,7 +4024,10 @@ element it has to parse."
 	     ;; LaTeX Environment.
 	     ((looking-at org-element--latex-begin-environment)
 	      (org-element-latex-environment-parser limit affiliated))
-	     ;; Drawer and Property Drawer.
+	     ;; Property drawer (before first headline, else it's catched above).
+	     ((org-at-property-block-p)
+	      (org-element-property-drawer-parser limit))
+	     ;; Drawer.
 	     ((looking-at org-drawer-regexp)
 	      (org-element-drawer-parser limit affiliated))
 	     ;; Fixed Width
diff --git a/lisp/org.el b/lisp/org.el
index 358989070..5c21e130c 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -3185,8 +3185,13 @@ This list will be combined with the constant `org-global-properties-fixed'.
 The entries in this list are cons cells where the car is a property
 name and cdr is a string with the value.
 
-You can set buffer-local values for the same purpose in the variable
-`org-file-properties' this by adding lines like
+Buffer local properties are added either by a document property drawer
+
+:PROPERTIES:
+:NAME: VALUE
+:END:
+
+or by adding lines like
 
 #+PROPERTY: NAME VALUE"
   :group 'org-properties
@@ -3194,10 +3199,14 @@ You can set buffer-local values for the same purpose in the variable
 	  (cons (string :tag "Property")
 		(string :tag "Value"))))
 
-(defvar-local org-file-properties nil
+(defvar-local org-keyword-properties nil
   "List of property/value pairs that can be inherited by any entry.
-Valid for the current buffer.
-This variable is populated from #+PROPERTY lines.")
+Valid for the current buffer.  This variable is populated from
+#+PROPERTY lines.
+
+Note that properties are defined also in property drawers.
+Properties defined there will take precedence over properties
+defined as keywords.")
 
 (defgroup org-agenda nil
   "Options concerning agenda views in Org mode."
@@ -3206,11 +3215,18 @@ This variable is populated from #+PROPERTY lines.")
 
 (defvar-local org-category nil
   "Variable used by Org files to set a category for agenda display.
-Such files should use a file variable to set it, for example
+There are multiple ways to set the category.  One way is to set
+it in the document property drawer.  For example:
+
+:PROPERTIES:
+:CATEGORY: ELisp
+:END:
+
+Other ways to define it is as an emacs file variable, for example
 
 #   -*- mode: org; org-category: \"ELisp\"
 
-or contain a special line
+or for the file to contain a special line:
 
 #+CATEGORY: ELisp
 
@@ -4343,8 +4359,8 @@ related expressions."
       (setq org-tag-groups-alist
 	    (org-tag-alist-to-groups org-current-tag-alist))
       (unless tags-only
-	;; File properties.
-	(setq-local org-file-properties (cdr (assq 'property alist)))
+	;; Properties.
+	(setq-local org-keyword-properties (cdr (assq 'property alist)))
 	;; Archive location.
 	(let ((archive (cdr (assq 'archive alist))))
 	  (when archive (setq-local org-archive-location archive)))
@@ -4352,9 +4368,9 @@ related expressions."
 	(let ((cat (org-string-nw-p (cdr (assq 'category alist)))))
 	  (when cat
 	    (setq-local org-category (intern cat))
-	    (setq-local org-file-properties
+	    (setq-local org-keyword-properties
 			(org--update-property-plist
-			 "CATEGORY" cat org-file-properties))))
+			 "CATEGORY" cat org-keyword-properties))))
 	;; Columns.
 	(let ((column (cdr (assq 'columns alist))))
 	  (when column (setq-local org-columns-default-format column)))
@@ -8276,13 +8292,14 @@ the value of the drawer property."
 	 (inhibit-read-only t)
 	 (inherit? (org-property-inherit-p dprop))
 	 (property-re (org-re-property (concat (regexp-quote dprop) "\\+?") t))
-	 (global (and inherit? (org--property-global-value dprop nil))))
+	 (global-or-keyword (and inherit?
+				 (org--property-global-or-keyword-value dprop nil))))
     (with-silent-modifications
       (org-with-point-at 1
-	;; Set global values (e.g., values defined through
-	;; "#+PROPERTY:" keywords) to the whole buffer.
-	(when global (put-text-property (point-min) (point-max) tprop global))
-	;; Set local values.
+	;; Set global and keyword based values to the whole buffer.
+	(when global-or-keyword
+	  (put-text-property (point-min) (point-max) tprop global-or-keyword))
+	;; Set values based on property-drawers throughout the document.
 	(while (re-search-forward property-re nil t)
 	  (when (org-at-property-p)
 	    (org-refresh-property tprop (org-entry-get (point) dprop) inherit?))
@@ -8290,21 +8307,29 @@ the value of the drawer property."
 
 (defun org-refresh-property (tprop p &optional inherit)
   "Refresh the buffer text property TPROP from the drawer property P.
-The refresh happens only for the current headline, or the whole
-sub-tree if optional argument INHERIT is non-nil."
-  (unless (org-before-first-heading-p)
-    (save-excursion
-      (org-back-to-heading t)
-      (let ((start (point))
-	    (end (save-excursion
-		   (if inherit (org-end-of-subtree t t)
-		     (or (outline-next-heading) (point-max))))))
-	(if (symbolp tprop)
-	    ;; TPROP is a text property symbol.
-	    (put-text-property start end tprop p)
-	  ;; TPROP is an alist with (property . function) elements.
-	  (pcase-dolist (`(,prop . ,f) tprop)
-	    (put-text-property start end prop (funcall f p))))))))
+The refresh happens only for the current entry, or the whole
+sub-tree if optional argument INHERIT is non-nil.
+
+If point is before first headline, the function applies to the
+part before the first headline.  In that particular case, when
+optional argument INHERIT is non-nil, it refreshes properties for
+the whole buffer."
+  (save-excursion
+    (org-back-to-heading-or-point-min t)
+    (let ((start (point))
+	  (end (save-excursion
+		 (cond ((and inherit (org-before-first-heading-p))
+			(point-max))
+		       (inherit
+			(org-end-of-subtree t t))
+		       ((outline-next-heading))
+		       ((point-max))))))
+      (if (symbolp tprop)
+	  ;; TPROP is a text property symbol.
+	  (put-text-property start end tprop p)
+	;; TPROP is an alist with (property . function) elements.
+	(pcase-dolist (`(,prop . ,f) tprop)
+	  (put-text-property start end prop (funcall f p)))))))
 
 (defun org-refresh-category-properties ()
   "Refresh category text properties in the buffer."
@@ -8320,9 +8345,9 @@ sub-tree if optional argument INHERIT is non-nil."
 		(t org-category))))
     (with-silent-modifications
       (org-with-wide-buffer
-       ;; Set buffer-wide category.  Search last #+CATEGORY keyword.
-       ;; This is the default category for the buffer.  If none is
-       ;; found, fall-back to `org-category' or buffer file name.
+       ;; Set buffer-wide property from keyword.  Search last #+CATEGORY
+       ;; keyword.  If none is found, fall-back to `org-category' or
+       ;; buffer file name, or set it by the document property drawer.
        (put-text-property
 	(point-min) (point-max)
 	'org-category
@@ -8334,15 +8359,20 @@ sub-tree if optional argument INHERIT is non-nil."
 		(throw 'buffer-category
 		       (org-element-property :value element)))))
 	  default-category))
-       ;; Set sub-tree specific categories.
+       ;; Set categories from the document property drawer or
+       ;; property drawers in the outline.  If category is found in
+       ;; the property drawer for the whole buffer that value
+       ;; overrides the keyword-based value set above.
        (goto-char (point-min))
        (let ((regexp (org-re-property "CATEGORY")))
 	 (while (re-search-forward regexp nil t)
 	   (let ((value (match-string-no-properties 3)))
 	     (when (org-at-property-p)
 	       (put-text-property
-		(save-excursion (org-back-to-heading t) (point))
-		(save-excursion (org-end-of-subtree t t) (point))
+		(save-excursion (org-back-to-heading-or-point-min t))
+		(save-excursion (if (org-before-first-heading-p)
+				    (point-max)
+				  (org-end-of-subtree t t)))
 		'org-category
 		value)))))))))
 
@@ -10646,7 +10676,7 @@ prefer a state in the current sequence over on in another sequence."
 	    (setq tg (org-add-props tg nil 'face
 				    (org-get-todo-face tg)))
 	    (when (and (= cnt 0) (not ingroup)) (insert "  "))
-	    (setq prompt (concat prompt "[" (char-to-string c) "] " tg " ")) 
+	    (setq prompt (concat prompt "[" (char-to-string c) "] " tg " "))
 	    (insert "[" c "] " tg (make-string
 				   (- fwidth 4 (length tg)) ?\ ))
 	    (when (and (= (setq cnt (1+ cnt)) ncol)
@@ -12921,30 +12951,45 @@ Modifications are made by side-effect.  Return new alist."
 
 (defun org-get-property-block (&optional beg force)
   "Return the (beg . end) range of the body of the property drawer.
-BEG is the beginning of the current subtree, or of the part
-before the first headline.  If it is not given, it will be found.
-If the drawer does not exist, create it if FORCE is non-nil, or
-return nil."
+BEG is the beginning of the current subtree or the beginning of
+the document if before the first headline.  If it is not given,
+it will be found.  If the drawer does not exist, create it if
+FORCE is non-nil, or return nil."
   (org-with-wide-buffer
-   (when beg (goto-char beg))
-   (unless (org-before-first-heading-p)
-     (let ((beg (cond (beg)
+   (let ((beg (cond (beg (goto-char beg))
 		      ((or (not (featurep 'org-inlinetask))
 			   (org-inlinetask-in-task-p))
-		       (org-back-to-heading t))
-		      (t (org-with-limited-levels (org-back-to-heading t))))))
-       (forward-line)
-       (when (looking-at-p org-planning-line-re) (forward-line))
-       (cond ((looking-at org-property-drawer-re)
-	      (forward-line)
-	      (cons (point) (progn (goto-char (match-end 0))
-				   (line-beginning-position))))
-	     (force
-	      (goto-char beg)
-	      (org-insert-property-drawer)
-	      (let ((pos (save-excursion (search-forward ":END:")
-					 (line-beginning-position))))
-		(cons pos pos))))))))
+		       (org-back-to-heading-or-point-min t) (point))
+		      (t (org-with-limited-levels
+			  (org-back-to-heading-or-point-min t))
+			 (point)))))
+     ;; Move point to its position according to its positional rules.
+     (cond ((org-before-first-heading-p)
+	    (while (and (bolp) (or (org-at-keyword-p) (org-at-comment-p))) (forward-line)))
+	   (t (forward-line)
+	      (when (looking-at-p org-planning-line-re) (forward-line))))
+     (cond ((looking-at org-property-drawer-re)
+	    (forward-line)
+	    (cons (point) (progn (goto-char (match-end 0))
+				 (line-beginning-position))))
+	   (force
+	    (goto-char beg)
+	    (org-insert-property-drawer)
+	    (let ((pos (save-excursion (re-search-forward org-property-drawer-re)
+				       (line-beginning-position))))
+	      (cons pos pos)))))))
+
+(defun org-at-property-block-p ()
+  "Non-nil when point is at the first line of a property drawer.
+The property drawer is validated according to its positional
+rules using `org-get-property-block'."
+  (save-excursion
+    (beginning-of-line)
+    (and (looking-at org-property-start-re)
+	 (forward-line)
+	 (let ((property-drawer (org-get-property-block)))
+	   (and property-drawer
+		(= (point) (car property-drawer)))))))
 
 (defun org-at-property-p ()
   "Non-nil when point is inside a property drawer.
@@ -13029,7 +13074,7 @@ Return value is an alist.  Keys are properties, as upcased
 strings."
   (org-with-point-at pom
     (when (and (derived-mode-p 'org-mode)
-	       (ignore-errors (org-back-to-heading t)))
+	       (org-back-to-heading-or-point-min t))
       (catch 'exit
 	(let* ((beg (point))
 	       (specific (and (stringp which) (upcase which)))
@@ -13238,13 +13283,13 @@ unless LITERAL-NIL is non-nil."
 	;; Return final values.
 	(and (not (equal value '(nil))) (nreverse value))))))
 
-(defun org--property-global-value (property literal-nil)
-  "Return value for PROPERTY in current buffer.
+(defun org--property-global-or-keyword-value (property literal-nil)
+  "Return value for PROPERTY as defined by global properties or by keyword.
 Return value is a string.  Return nil if property is not set
-globally.  Also return nil when PROPERTY is set to \"nil\",
-unless LITERAL-NIL is non-nil."
+globally or by keyword.  Also return nil when PROPERTY is set to
+\"nil\", unless LITERAL-NIL is non-nil."
   (let ((global
-	 (cdr (or (assoc-string property org-file-properties t)
+	 (cdr (or (assoc-string property org-keyword-properties t)
 		  (assoc-string property org-global-properties t)
 		  (assoc-string property org-global-properties-fixed t)))))
     (if literal-nil global (org-not-nil global))))
@@ -13393,12 +13438,12 @@ However, if LITERAL-NIL is set, return the string value \"nil\" instead."
 			   value)))
 	   (cond
 	    ((car v)
-	     (org-back-to-heading t)
+	     (org-back-to-heading-or-point-min t)
 	     (move-marker org-entry-property-inherited-from (point))
 	     (throw 'exit nil))
-	    ((org-up-heading-safe))
+	    ((org-up-heading-or-point-min))
 	    (t
-	     (let ((global (org--property-global-value property literal-nil)))
+	     (let ((global (org--property-global-or-keyword-value property literal-nil)))
 	       (cond ((not global))
 		     (value (setq value (concat global " " value)))
 		     (t (setq value global))))
@@ -13430,8 +13475,8 @@ decreases scheduled or deadline date by one day."
 	 (user-error "Invalid property name: \"%s\"" property)))
   (org-with-point-at pom
     (if (or (not (featurep 'org-inlinetask)) (org-inlinetask-in-task-p))
-	(org-back-to-heading t)
-      (org-with-limited-levels (org-back-to-heading t)))
+	(org-back-to-heading-or-point-min t)
+      (org-with-limited-levels (org-back-to-heading-or-point-min t)))
     (let ((beg (point)))
       (cond
        ((equal property "TODO")
@@ -13567,19 +13612,26 @@ COLUMN formats in the current buffer."
 Do nothing if the drawer already exists.  The newly created
 drawer is immediately hidden."
   (org-with-wide-buffer
+   ;; Set point to the position where the drawer should be inserted.
    (if (or (not (featurep 'org-inlinetask)) (org-inlinetask-in-task-p))
-       (org-back-to-heading t)
-     (org-with-limited-levels (org-back-to-heading t)))
-   (forward-line)
-   (when (looking-at-p org-planning-line-re) (forward-line))
+       (org-back-to-heading-or-point-min t)
+     (org-with-limited-levels (org-back-to-heading-or-point-min t)))
+   (if (org-before-first-heading-p)
+       (while (and (bolp) (or (org-at-keyword-p) (org-at-comment-p))) (forward-line))
+     (progn
+       (forward-line)
+       (when (looking-at-p org-planning-line-re) (forward-line))))
    (unless (looking-at-p org-property-drawer-re)
      ;; Make sure we start editing a line from current entry, not from
      ;; next one.  It prevents extending text properties or overlays
      ;; belonging to the latter.
-     (when (bolp) (backward-char))
-     (let ((begin (1+ (point)))
+     (when (and (bolp) (> (point) (point-min))) (backward-char))
+     (let ((begin (if (= (point) (point-min))
+		      (point)
+		    (1+ (point))))
 	   (inhibit-read-only t))
-       (insert "\n:PROPERTIES:\n:END:")
+       (unless (= begin (point-min)) (insert "\n"))
+       (insert ":PROPERTIES:\n:END:")
        (org-flag-drawer t nil (line-end-position 0) (point))
        (when (eobp) (insert "\n"))
        (org-indent-region begin (point))))))
@@ -20484,6 +20536,15 @@ interactive command with similar behavior."
     (error (error "Before first headline at position %d in buffer %s"
 		  (point) (current-buffer)))))
 
+(defun org-back-to-heading-or-point-min (&optional invisible-ok)
+  "Go back to heading or first point in buffer.
+If point is before first heading go to first point in buffer
+instead of back to heading."
+  (condition-case nil
+      (outline-back-to-heading invisible-ok)
+    (error
+     (goto-char (point-min)))))
+
 (defun org-before-first-heading-p ()
   "Before first heading?"
   (org-with-limited-levels
@@ -20517,6 +20578,12 @@ unless optional argument NO-INHERITANCE is non-nil."
       (beginning-of-line)
       (looking-at "^[ \t]*# "))))
 
+(defun org-at-keyword-p nil
+  "Is cursor at a keyword-line?"
+  (save-excursion
+    (move-beginning-of-line 1)
+    (looking-at org-keyword-regexp)))
+
 (defun org-at-drawer-p nil
   "Is cursor at a drawer keyword?"
   (save-excursion
@@ -20564,6 +20631,17 @@ make a significant difference in outlines with very many siblings."
 	   (re-search-backward (format "^\\*\\{1,%d\\} " level-up) nil t)
 	   (funcall outline-level)))))
 
+(defun org-up-heading-or-point-min ()
+  "Move to the heading line of which the present is a subheading, or point-min.
+This version is needed to make point-min behave like a virtual
+heading of level 0 for property-inheritance.  It will return the
+level of the headline found (down to 0) or nil if already at a
+point before the first headline or at point-min."
+  (when (ignore-errors (org-back-to-heading t))
+    (if (< 1 (funcall outline-level))
+	(org-up-heading-safe)
+      (unless (= (point) (point-min)) (goto-char (point-min))))))
+
 (defun org-first-sibling-p ()
   "Is this heading the first child of its parents?"
   (interactive)
@@ -20664,28 +20742,31 @@ If there is no such heading, return nil."
 (defun org-end-of-subtree (&optional invisible-ok to-heading)
   "Goto to the end of a subtree."
   ;; This contains an exact copy of the original function, but it uses
-  ;; `org-back-to-heading', to make it work also in invisible
-  ;; trees.  And is uses an invisible-ok argument.
+  ;; `org-back-to-heading-or-point-min', to make it work also in invisible
+  ;; trees and before first headline.  And is uses an invisible-ok argument.
   ;; Under Emacs this is not needed, but the old outline.el needs this fix.
   ;; Furthermore, when used inside Org, finding the end of a large subtree
   ;; with many children and grandchildren etc, this can be much faster
   ;; than the outline version.
-  (org-back-to-heading invisible-ok)
+  (org-back-to-heading-or-point-min invisible-ok)
   (let ((first t)
 	(level (funcall outline-level)))
-    (if (and (derived-mode-p 'org-mode) (< level 1000))
-	;; A true heading (not a plain list item), in Org
-	;; This means we can easily find the end by looking
-	;; only for the right number of stars.  Using a regexp to do
-	;; this is so much faster than using a Lisp loop.
-	(let ((re (concat "^\\*\\{1," (int-to-string level) "\\} ")))
-	  (forward-char 1)
-	  (and (re-search-forward re nil 'move) (beginning-of-line 1)))
-      ;; something else, do it the slow way
-      (while (and (not (eobp))
-		  (or first (> (funcall outline-level) level)))
-	(setq first nil)
-	(outline-next-heading)))
+    (cond ((= level 0)
+	   (goto-char (point-max)))
+	  ((and (derived-mode-p 'org-mode) (< level 1000))
+	   ;; A true heading (not a plain list item), in Org
+	   ;; This means we can easily find the end by looking
+	   ;; only for the right number of stars.  Using a regexp to do
+	   ;; this is so much faster than using a Lisp loop.
+	   (let ((re (concat "^\\*\\{1," (int-to-string level) "\\} ")))
+	     (forward-char 1)
+	     (and (re-search-forward re nil 'move) (beginning-of-line 1))))
+	  (t
+	   ;; something else, do it the slow way
+	   (while (and (not (eobp))
+		       (or first (> (funcall outline-level) level)))
+	     (setq first nil)
+	     (outline-next-heading))))
     (unless to-heading
       (when (memq (preceding-char) '(?\n ?\^M))
 	;; Go to end of line before heading
diff --git a/testing/examples/property-inheritance.org b/testing/examples/property-inheritance.org
index 6979de595..9c0b7e662 100644
--- a/testing/examples/property-inheritance.org
+++ b/testing/examples/property-inheritance.org
@@ -1,5 +1,7 @@
-#+property: header-args :var  foo=1
-#+property: header-args+ :var bar=2
+:PROPERTIES:
+:header-args: :var foo=1
+:header-args+: :var bar=2
+:END:
 
 #+begin_src emacs-lisp
   (+ foo bar)
diff --git a/testing/lisp/test-org-element.el b/testing/lisp/test-org-element.el
index 2a1716134..9e9cbb381 100644
--- a/testing/lisp/test-org-element.el
+++ b/testing/lisp/test-org-element.el
@@ -1925,6 +1925,15 @@ e^{i\\pi}+1=0
 	    (let ((element (org-element-at-point)))
 	      (list (org-element-property :key element)
 		    (org-element-property :value element))))))
+  ;; The insides of property blocks on document level are parsed the
+  ;; same way as headline property blocks.  I.e. the concept of
+  ;; `node-property' apply also for properties in those blocks.
+  (should
+   (equal '("abc" "value")
+	  (org-test-with-temp-text ":PROPERTIES:\n<point>:abc: value\n:END:"
+	    (let ((element (org-element-at-point)))
+	      (list (org-element-property :key element)
+		    (org-element-property :value element))))))
   ;; Value should be trimmed.
   (should
    (equal "value"
@@ -2111,6 +2120,18 @@ Outside list"
        (org-test-with-temp-text
 	   "* H\nDEADLINE: <2014-03-04 tue.>\n<point>:PROPERTIES:\n:prop: value\n:END:"
 	 (org-element-type (org-element-at-point)))))
+  (should
+   (eq 'property-drawer
+       (org-test-with-temp-text "<point>:PROPERTIES:\n:prop: value\n:END:"
+	 (org-element-type (org-element-at-point)))))
+  (should
+   (eq 'property-drawer
+       (org-test-with-temp-text "# C\n#+Key: V\n# C\n<point>:PROPERTIES:\n:prop: value\n:END:"
+	 (org-element-type (org-element-at-point)))))
+  (should-not
+   (eq 'property-drawer
+       (org-test-with-temp-text "\n<point>:PROPERTIES:\n:prop: value\n:END:"
+	 (org-element-type (org-element-at-point)))))
   ;; Allow properties without value and no property at all.
   (should
    (eq 'property-drawer
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 01e965ea7..2b5d3d280 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -386,10 +386,117 @@
 \f
 ;;; Drawers
 
+(ert-deftest test-org/at-property-p ()
+  "Test `org-at-property-p' specifications."
+  (should
+   (equal 't
+	  (org-test-with-temp-text "* H\n:PROPERTIES:\n<point>:PROP: t\n:END:\n"
+	    (org-at-property-p))))
+  (should
+   (equal 't
+	  (org-test-with-temp-text ":PROPERTIES:\n<point>:PROP: t\n:END:\n"
+	    (org-at-property-p)))))
+
+(ert-deftest test-org/at-property-block-p ()
+  "Test `org-at-property-block-p' specifications."
+  (should
+   (equal 't
+	  (org-test-with-temp-text "* H\n<point>:PROPERTIES:\n:PROP: t\n:END:\n"
+	    (org-at-property-block-p))))
+  (should
+   (equal 't
+	  (org-test-with-temp-text ":PROPERTIES:\n:PROP: t\n:END:\n"
+	    (org-at-property-block-p))))
+  ;; The function only returns t if point is at the first line of a
+  ;; property block.
+  (should-not
+   (equal 't
+	  (org-test-with-temp-text ":PROPERTIES:\n<point>:PROP: t\n:END:\n"
+	    (org-at-property-block-p))))
+  (should))
+
+(ert-deftest test-org/get-property-block ()
+  "Test `org-get-property-block' specifications."
+  (should
+   (equal '(14 . 14)
+	  (org-test-with-temp-text ":PROPERTIES:\n:END:\n* H\n"
+	    (org-get-property-block))))
+  (should
+   (equal '(14 . 14)
+	  (org-test-with-temp-text ":PROPERTIES:\n:END:\n"
+	    (org-get-property-block))))
+  ;; Comments above a document property block is ok.
+  (should
+   (equal '(18 . 18)
+	  (org-test-with-temp-text "# C\n:PROPERTIES:\n:END:\n"
+	    (org-get-property-block))))
+  ;; Keywords above a document property block is ok.
+  (should
+   (equal '(23 . 23)
+	  (org-test-with-temp-text "#+Key: v\n:PROPERTIES:\n:END:\n"
+	    (org-get-property-block))))
+  ;; Comments and keywords are allowed before a document property block.
+  (should
+   (equal '(27 . 36)
+	  (org-test-with-temp-text "# C\n#+key: v\n:PROPERTIES:\n:KEY: V:\n:END:\n"
+	    (org-get-property-block))))
+  (should
+   (equal '(36 . 45)
+	  (org-test-with-temp-text "#+Key: v\n# C\n#+Key: v\n:PROPERTIES:\n:KEY: V:\n:END:\n"
+	    (org-get-property-block))))
+  ;; A document property block will not be valid if there are lines
+  ;; with whitespace above it
+  (should-not
+   (org-test-with-temp-text "\n:PROPERTIES:\n:END:\n"
+     (org-get-property-block)))
+  (should
+   (equal '(18 . 18)
+	  (org-test-with-temp-text "* H\n:PROPERTIES:\n:END:\n<point>"
+	    (org-get-property-block))))
+  (should
+   (equal "* H\n:PROPERTIES:\n:END:\n"
+	  (org-test-with-temp-text "* H"
+	    (let ((org-adapt-indentation nil))
+	      (org-get-property-block nil 'force))
+	    (buffer-string))))
+  (should
+   (equal ":PROPERTIES:\n:END:\n"
+	  (org-test-with-temp-text ""
+	    (org-get-property-block nil 'force)
+	    (buffer-string))))
+  (should
+   (equal "* H1\n  :PROPERTIES:\n  :END:\n* H2"
+	  (org-test-with-temp-text "* H1\n* H2"
+	    (let ((org-adapt-indentation t))
+	      (org-get-property-block nil 'force))
+	    (buffer-string)))))
+
 (ert-deftest test-org/insert-property-drawer ()
   "Test `org-insert-property-drawer' specifications."
-  ;; Error before first headline.
-  (should-error (org-test-with-temp-text "" (org-insert-property-drawer)))
+  ;; Insert drawer in empty buffer
+  (should
+   (equal ":PROPERTIES:\n:END:\n"
+	  (org-test-with-temp-text ""
+	    (let ((org-adapt-indentation nil)) (org-insert-property-drawer))
+	    (buffer-string))))
+  ;; Insert drawer in document header with existing comment and
+  ;; keyword.
+  (should
+   (equal "# C\n#+TITLE: T\n:PROPERTIES:\n:END:\n"
+	  (org-test-with-temp-text "# C\n#+TITLE: T"
+	    (let ((org-adapt-indentation nil)) (org-insert-property-drawer))
+	    (buffer-string))))
+  (should
+   (equal ":PROPERTIES:\n:END:"
+	  (org-test-with-temp-text ":PROPERTIES:\n:END:"
+	    (let ((org-adapt-indentation nil)) (org-insert-property-drawer))
+	    (buffer-string))))
+  ;; Insert drawer in document header with one existing heading in buffer.
+  (should
+   (equal ":PROPERTIES:\n:END:\n* T\n"
+	  (org-test-with-temp-text "<point>\n* T\n"
+	    (let ((org-adapt-indentation nil)) (org-insert-property-drawer))
+	    (buffer-string))))
   ;; Insert drawer right after headline if there is no planning line,
   ;; or after it otherwise.
   (should
@@ -2178,19 +2285,19 @@ SCHEDULED: <2014-03-04 tue.>"
    (equal "foo=1"
 	  (org-test-with-temp-text "#+PROPERTY: var foo=1"
 	    (org-mode-restart)
-	    (cdr (assoc "var" org-file-properties)))))
+	    (cdr (assoc "var" org-keyword-properties)))))
   (should
    (equal
     "foo=1 bar=2"
     (org-test-with-temp-text "#+PROPERTY: var foo=1\n#+PROPERTY: var+ bar=2"
       (org-mode-restart)
-      (cdr (assoc "var" org-file-properties)))))
+      (cdr (assoc "var" org-keyword-properties)))))
   (should
    (equal
     "foo=1 bar=2"
     (org-test-with-temp-text "#+PROPERTY: var foo=1\n#+PROPERTY: VAR+ bar=2"
       (org-mode-restart)
-      (cdr (assoc "var" org-file-properties)))))
+      (cdr (assoc "var" org-keyword-properties)))))
   ;; ARCHIVE keyword.
   (should
    (equal "%s_done::"
@@ -2207,7 +2314,7 @@ SCHEDULED: <2014-03-04 tue.>"
    (equal "test"
 	  (org-test-with-temp-text "#+CATEGORY: test"
 	    (org-mode-restart)
-	    (cdr (assoc "CATEGORY" org-file-properties)))))
+	    (cdr (assoc "CATEGORY" org-keyword-properties)))))
   ;; COLUMNS keyword.
   (should
    (equal "%25ITEM %TAGS %PRIORITY %TODO"
@@ -2291,7 +2398,7 @@ SCHEDULED: <2014-03-04 tue.>"
 	  (org-test-with-temp-text
 	      (format "#+SETUPFILE: \"%s/examples/setupfile.org\"" org-test-dir)
 	    (org-mode-restart)
-	    (cdr (assoc "a" org-file-properties))))))
+	    (cdr (assoc "a" org-keyword-properties))))))
 
 
 \f
@@ -4941,6 +5048,66 @@ Paragraph<point>"
 	  (org-test-with-temp-text "* H\n:PROPERTIES:\n:A: 1\n:A+: 2\n:END:"
 	    (org-property-values "A")))))
 
+(ert-deftest test-org/set-property ()
+  "Test `org-set-property' specifications."
+  (should
+   (equal
+    ":PROPERTIES:\n:TEST: t\n:END:\n"
+    (org-test-with-temp-text ""
+      (let ((org-property-format "%s %s"))
+	(org-set-property "TEST" "t"))
+      (buffer-string))))
+  (should
+   (equal
+    "* H\n:PROPERTIES:\n:TEST: t\n:END:\n"
+    (org-test-with-temp-text "* H"
+      (let ((org-adapt-indentation nil)
+	    (org-property-format "%s %s"))
+	(org-set-property "TEST" "t"))
+      (buffer-string)))))
+
+(ert-deftest test-org/delete-property ()
+  "Test `org-delete-property' specifications."
+  (should
+   (equal
+    ""
+    (org-test-with-temp-text ":PROPERTIES:\n:TEST: t\n:END:\n"
+      (org-delete-property "TEST")
+      (buffer-string))))
+  (should
+   (equal
+    ":PROPERTIES:\n:TEST1: t\n:END:\n"
+    (org-test-with-temp-text ":PROPERTIES:\n:TEST1: t\n:TEST2: t\n:END:\n"
+      (org-delete-property "TEST2")
+      (buffer-string))))
+  (should
+   (equal
+    "* H\n"
+    (org-test-with-temp-text "* H\n:PROPERTIES:\n:TEST: t\n:END:\n"
+      (org-delete-property "TEST")
+      (buffer-string))))
+  (should
+   (equal
+    "* H\n:PROPERTIES:\n:TEST1: t\n:END:\n"
+    (org-test-with-temp-text "* H\n:PROPERTIES:\n:TEST1: t\n:TEST2: t\n:END:\n"
+      (org-delete-property "TEST2")
+      (buffer-string)))))
+
+(ert-deftest test-org/delete-property-globally ()
+  "Test `org-delete-property-global' specifications."
+  (should
+   (equal
+    ""
+    (org-test-with-temp-text ":PROPERTIES:\n:TEST: t\n:END:\n"
+      (org-delete-property-globally "TEST")
+      (buffer-string))))
+  (should
+   (equal
+    "* H\n"
+    (org-test-with-temp-text ":PROPERTIES:\n:TEST: t\n:END:\n* H\n:PROPERTIES:\n:TEST: nil\n:END:"
+      (org-delete-property-globally "TEST")
+      (buffer-string)))))
+
 (ert-deftest test-org/find-property ()
   "Test `org-find-property' specifications."
   ;; Regular test.
@@ -5022,6 +5189,10 @@ Paragraph<point>"
 (ert-deftest test-org/entry-get ()
   "Test `org-entry-get' specifications."
   ;; Regular test.
+  (should
+   (equal "1"
+	  (org-test-with-temp-text ":PROPERTIES:\n:A: 1\n:END:"
+	    (org-entry-get (point) "A"))))
   (should
    (equal "1"
 	  (org-test-with-temp-text "* H\n:PROPERTIES:\n:A: 1\n:END:"
@@ -5061,6 +5232,11 @@ Paragraph<point>"
      (org-entry-get (point) "B" nil t)))
   ;; Handle inheritance, when allowed.  Include extended values and
   ;; possibly global values.
+  (should
+   (equal
+    "1"
+    (org-test-with-temp-text ":PROPERTIES:\n:A: 1\n:END:\n* H"
+      (org-entry-get (point-max) "A" t))))
   (should
    (equal
     "1"
@@ -5076,12 +5252,30 @@ Paragraph<point>"
    (org-test-with-temp-text "* H\n:PROPERTIES:\n:A: 1\n:END:\n** H2"
      (let ((org-use-property-inheritance nil))
        (org-entry-get (point-max) "A" 'selective))))
+  (should
+   (equal
+    "1 2"
+    (org-test-with-temp-text
+	":PROPERTIES:\n:A: 1\n:END:\n* H\n:PROPERTIES:\n:A+: 2\n:END:"
+      (org-entry-get (point-max) "A" t))))
   (should
    (equal
     "1 2"
     (org-test-with-temp-text
 	"* H\n:PROPERTIES:\n:A: 1\n:END:\n** H2\n:PROPERTIES:\n:A+: 2\n:END:"
       (org-entry-get (point-max) "A" t))))
+  (should
+   (equal
+    "1 2"
+    (org-test-with-temp-text
+	":PROPERTIES:\n:A: 1\n:END:\n* H1\n* H2\n:PROPERTIES:\n:A+: 2\n:END:"
+      (org-entry-get (point-max) "A" t))))
+  (should
+   (equal
+    "1 2"
+    (org-test-with-temp-text
+	"* H1\n:PROPERTIES:\n:A: 1\n:END:\n* H2.1\n* H2.2\n:PROPERTIES:\n:A+: 2\n:END:"
+      (org-entry-get (point-max) "A" t))))
   (should
    (equal "1"
 	  (org-test-with-temp-text
@@ -5093,6 +5287,14 @@ Paragraph<point>"
 	  (org-test-with-temp-text
 	      "#+PROPERTY: A 0\n* H\n:PROPERTIES:\n:A+: 1\n:END:"
 	    (org-mode-restart)
+	    (org-entry-get (point-max) "A" t))))
+  ;; document level property-drawer has precedance over
+  ;; global-property by PROPERTY-keyword.
+  (should
+   (equal "0 2"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 0\n:END:\n#+PROPERTY: A 1\n* H\n:PROPERTIES:\n:A+: 2\n:END:"
+	    (org-mode-restart)
 	    (org-entry-get (point-max) "A" t)))))
 
 (ert-deftest test-org/entry-properties ()
@@ -5416,8 +5618,44 @@ Paragraph<point>"
 	    (let ((org-use-property-inheritance t))
 	      (org-refresh-properties "A" 'org-test))
 	    (get-text-property (point) 'org-test))))
+  ;; When a document level property-drawer is used, those properties
+  ;; should work exactly like headline-properties as if at a
+  ;; headline-level 0.
+  (should
+   (equal "1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 1\n:END:\n"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-properties "A" 'org-test))
+	    (get-text-property (point) 'org-test))))
+  (should-not
+   (equal "1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 1\n:END:\n<point>* H1"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance nil))
+	      (org-refresh-properties "A" 'org-test))
+	    (get-text-property (point) 'org-test))))
+  (should
+   (equal "1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 1\n:END:\n<point>* H1"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-properties "A" 'org-test))
+	    (get-text-property (point) 'org-test))))
+  (should
+   (equal "2"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 1\n:END:\n<point>* H1\n:PROPERTIES:\n:A: 2\n:END:"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-properties "A" 'org-test))
+	    (get-text-property (point) 'org-test))))
   ;; When property is inherited, use global value across the whole
-  ;; buffer.  However local values have precedence.
+  ;; buffer.  However local values have precedence, as well as the
+  ;; document level property-drawer.
   (should-not
    (equal "1"
 	  (org-test-with-temp-text "#+PROPERTY: A 1\n<point>* H1"
@@ -5437,10 +5675,62 @@ Paragraph<point>"
 	  (org-test-with-temp-text
 	      "#+PROPERTY: A 1\n<point>* H\n:PROPERTIES:\n:A: 2\n:END:"
 	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-properties "A" 'org-test))
+	    (get-text-property (point) 'org-test))))
+  ;; When both keyword-property and document-level property-block is
+  ;; defined, the property-block has precedance.
+  (should
+   (equal "1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:A: 1\n:END:\n#+PROPERTY: A 2\n<point>* H1"
+	    (org-mode-restart)
 	    (let ((org-use-property-inheritance t))
 	      (org-refresh-properties "A" 'org-test))
 	    (get-text-property (point) 'org-test)))))
 
+(ert-deftest test-org/refresh-category-properties ()
+  "Test `org-refresh-category-properties' specifications"
+  (should
+   (equal "cat1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:CATEGORY: cat1\n:END:"
+	    (org-refresh-category-properties)
+	    (get-text-property (point) 'org-category))))
+  (should
+   (equal "cat1"
+	  (org-test-with-temp-text
+	      "* H\n:PROPERTIES:\n:CATEGORY: cat1\n:END:"
+	    (org-refresh-category-properties)
+	    (get-text-property (point) 'org-category))))
+  ;; Even though property-inheritance is deactivated, category
+  ;; property should be inherited.  As described in
+  ;; `org-use-property-inheritance'.
+  (should
+   (equal "cat1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:CATEGORY: cat1\n:END:\n<point>* H"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance nil))
+	      (org-refresh-category-properties))
+	    (get-text-property (point) 'org-category))))
+  (should
+   (equal "cat1"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:CATEGORY: cat1\n:END:\n<point>* H"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-category-properties))
+	    (get-text-property (point) 'org-category))))
+  (should
+   (equal "cat2"
+	  (org-test-with-temp-text
+	      ":PROPERTIES:\n:CATEGORY: cat1\n:END:\n<point>* H\n:PROPERTIES:\n:CATEGORY: cat2\n:END:\n"
+	    (org-mode-restart)
+	    (let ((org-use-property-inheritance t))
+	      (org-refresh-category-properties))
+	    (get-text-property (point) 'org-category)))))
+
 \f
 ;;; Refile
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-08-31 18:49 Gustav Wikström
@ 2019-09-01  5:20 ` Adam Porter
  2019-09-01 10:38 ` Nicolas Goaziou
  2019-09-01 15:25 ` Gustav Wikström
  2 siblings, 0 replies; 13+ messages in thread
From: Adam Porter @ 2019-09-01  5:20 UTC (permalink / raw)
  To: emacs-orgmode

Gustav Wikström <gustav@whil.se> writes:

> I'm continuing on my proposal to introduce a "document" element in
> org-mode and the idea of seeing everything before the first headline
> as the base level 0 outline for a file. I've attached two patches that
> I'd like some public review of before pushing to master.
>
> Patch 0001 introduces the document element into org-element.el, and
> some restructuring related to that.
>
> Patch 0002 makes it possible to use property drawers at the document
> level. I've hopefully covered all related commands to make this work.
> And I've added a bunch of tests to guard against future regressions.
>
> Waiting for your comments!

This is a very interesting idea, and I don't want to dismiss your work,
but I am concerned about how much third-party code will likely break by
changing the results returned by org-element for parsing an Org buffer.
I haven't thoroughly studied all of the code in your patches, so I may
be wrong, but I think the breakage could be extensive.  For example,
simple operations like destructuring the results of org-element parsing
functions may be broken.  Have you done any investigation into this
issue?

Maybe there should be a transitional period in which the existing
org-element parsing functions would work as before, and the new
document-level elements would be returned by a new org-element
document-level parsing function.  Frankly, if there is breakage,the
transition would probably take a few years, because there is a lot of
code out there that has worked for years and may not be updated or
replaced for years.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-08-31 18:49 Gustav Wikström
  2019-09-01  5:20 ` Adam Porter
@ 2019-09-01 10:38 ` Nicolas Goaziou
  2019-09-01 14:41   ` Gustav Wikström
  2019-09-01 15:25 ` Gustav Wikström
  2 siblings, 1 reply; 13+ messages in thread
From: Nicolas Goaziou @ 2019-09-01 10:38 UTC (permalink / raw)
  To: Gustav Wikström; +Cc: Carsten Dominik, emacs-orgmode@gnu.org

Hello,

Gustav Wikström <gustav@whil.se> writes:

> I'm continuing on my proposal to introduce a "document" element in
> org-mode and the idea of seeing everything before the first headline
> as the base level 0 outline for a file. I've attached two patches that
> I'd like some public review of before pushing to master.

I will not review fully the patches, as I have no time for that.
However, I will make a few comments about it.

First, you should show a few examples of what an Org document would look
like, compared to what we have already, focusing particularly on the
advantages, and what is now invalid. It is a good thing to do if you
expect comments, as you cannot ask everyone to eyeball through the whole
patch set.

Also, whatever the outcome of the discussion is, /nothing should go in
master as long as Org 9.3 is not released/. This looks like a breaking
change at the most lower level (syntax, parser...), I think it may
trigger a new major release.

> Patch 0001 introduces the document element into org-element.el, and
> some restructuring related to that.

This should be explained in comments, and, if it lands at some point,
Worg pages about syntax and exporter should be updated, too.

> ** (renamed, modified) org--setup-collect-keywords -> org-collect-keywords
> Renamed and generalized org--setup-collect-keywords to make it work
> for multiple purposes.  Is not limited to a fixed set of keywords any
> longer.  New name: org-collect-keywords.

An important typo note: we use "Org mode", or "an Org document", not
"Org-mode" and "an org-document". Hyphens are only used to refer
explicitly to a Lisp symbol, or its value or function.

> ** (modified) org-element-keyword-parser
> Uses (new) org-keyword-regexp instead of hardcoding it's own regexp.

Keep in mind that Org Element library should ultimately be as
independent as possible to the other parts of Org, including "org.el".

> +;; Org-element can parse org-mode documents.  The top-node in the
> +;; parse-tree will always have TYPE `org-data' and PROPERTIES nil.

See my remark about typography above.

> +;; The following part creates a fully recursive org-mode parser.  

Ditto.

> +(defun org-back-to-heading-or-point-min (&optional invisible-ok)
> +  "Go back to heading or first point in buffer.
> +If point is before first heading go to first point in buffer
> +instead of back to heading."
> +  (condition-case nil
> +      (outline-back-to-heading invisible-ok)
> +    (error
> +     (goto-char (point-min)))))

Try to limit use of Outline functions. They are generally slower than
their Org counterpart. This is not true in this case, but, one day, we
might optimize `org-back-to-heading'.

> +(defun org-at-keyword-p nil
> +  "Is cursor at a keyword-line?"

Non-nil if ...

> +  (save-excursion
> +    (move-beginning-of-line 1)
> +    (looking-at org-keyword-regexp)))

While this is technically correct today, please don't write a predicate
only based on regexps, use the parser for that. For example, the parser
can understand

  #+begin_example
  #+keyword: not a keyword
  #+end_example

whereas your function cannot.

Also, you could use `org-match-line' in this case.


Regards,

-- 
Nicolas Goaziou

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-09-01 10:38 ` Nicolas Goaziou
@ 2019-09-01 14:41   ` Gustav Wikström
  2019-09-06 20:09     ` Nicolas Goaziou
  0 siblings, 1 reply; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 14:41 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode@gnu.org

Hi Nicolas,

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> 
> > I'm continuing on my proposal to introduce a "document" element in
> > org-mode and the idea of seeing everything before the first headline
> > as the base level 0 outline for a file. I've attached two patches that
> > I'd like some public review of before pushing to master.
> 
> I will not review fully the patches, as I have no time for that.
> However, I will make a few comments about it.
> 
> First, you should show a few examples of what an Org document would look
> like, compared to what we have already, focusing particularly on the
> advantages, and what is now invalid. It is a good thing to do if you
> expect comments, as you cannot ask everyone to eyeball through the whole
> patch set.

Sure, I'll do that as a reply to my own mail to not make that comment
disappear among the details here.

> Also, whatever the outcome of the discussion is, /nothing should go in
> master as long as Org 9.3 is not released/. This looks like a breaking
> change at the most lower level (syntax, parser...), I think it may
> trigger a new major release.

I'm starting out slow by making this a non-breaking change. At least
I'm trying to. I.e. everything that worked before should continue to
work.

> > Patch 0001 introduces the document element into org-element.el, and
> > some restructuring related to that.
> 
> This should be explained in comments, and, if it lands at some point,
> Worg pages about syntax and exporter should be updated, too.

Which comments do you mean? I've tried to be rigorous in the patch
notes. But you mean in the mail itself?

> 
> > ** (renamed, modified) org--setup-collect-keywords -> org-collect-keywords
> > Renamed and generalized org--setup-collect-keywords to make it work
> > for multiple purposes.  Is not limited to a fixed set of keywords any
> > longer.  New name: org-collect-keywords.
> 
> An important typo note: we use "Org mode", or "an Org document", not
> "Org-mode" and "an org-document". Hyphens are only used to refer
> explicitly to a Lisp symbol, or its value or function.

Ah of course. I keep forgetting that one.

> 
> > ** (modified) org-element-keyword-parser
> > Uses (new) org-keyword-regexp instead of hardcoding it's own regexp.
> 
> Keep in mind that Org Element library should ultimately be as
> independent as possible to the other parts of Org, including "org.el".

Got it - is it preferred to do it the other way around - I.e. define
the regexp in org-element.el rather than org.el if there is use of the
regexp in both files?

> > +;; Org-element can parse org-mode documents.  The top-node in the
> > +;; parse-tree will always have TYPE `org-data' and PROPERTIES nil.
> 
> See my remark about typography above.

Got it.

> 
> > +;; The following part creates a fully recursive org-mode parser.
> 
> Ditto.

Ok, ofc.

> > +(defun org-back-to-heading-or-point-min (&optional invisible-ok)
> > +  "Go back to heading or first point in buffer.
> > +If point is before first heading go to first point in buffer
> > +instead of back to heading."
> > +  (condition-case nil
> > +      (outline-back-to-heading invisible-ok)
> > +    (error
> > +     (goto-char (point-min)))))
> 
> Try to limit use of Outline functions. They are generally slower than
> their Org counterpart. This is not true in this case, but, one day, we
> might optimize `org-back-to-heading'.

Noted, except in this case I think it's warranted since
org-back-to-heading behaves exactly the same with the exception of
raising an error if before the first heading. Since both functions are
defined next to another I'm sure a later refactor will take care of
both since they're structurally identical and defined two lines apart.

> > +(defun org-at-keyword-p nil
> > +  "Is cursor at a keyword-line?"
> 
> Non-nil if ...

I've just "pattern-matched" myself into that docstring. It matches the
existing definitions of org-at-drawer-p, org-at-comment-p,
org-at-block-p. Which are defined around this.

> > +  (save-excursion
> > +    (move-beginning-of-line 1)
> > +    (looking-at org-keyword-regexp)))
> 
> While this is technically correct today, please don't write a predicate
> only based on regexps, use the parser for that. For example, the parser
> can understand
> 
>   #+begin_example
>   #+keyword: not a keyword
>   #+end_example
> 
> whereas your function cannot.

Hmm... I see your point. Have to think a bit here - because I have the
feeling that defining the predicate using org-element-at-point will
incur quite a performance hit. Or what is your intuition regarding
that?

> 
> Also, you could use `org-match-line' in this case.
> 
> 
> Regards,
> 
> --
> Nicolas Goaziou

Thanks for your initial thoughts.

/G

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-08-31 18:49 Gustav Wikström
  2019-09-01  5:20 ` Adam Porter
  2019-09-01 10:38 ` Nicolas Goaziou
@ 2019-09-01 15:25 ` Gustav Wikström
  2019-09-01 16:11   ` Adam Porter
  2 siblings, 1 reply; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 15:25 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org; +Cc: Carsten Dominik, Nicolas Goaziou

Hi again,

Nicolas requested a more thorough introduction to the patch so here it
comes.

To start with, this relates to the topic presented here:
- https://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg00000.html 

The first patch deals with formalities. It introduces one new greater
element called "document". Parsers and everything around it are
modified to work with this new concept. No new functionality is
introduced. I'd call this patch an "enabler" since it allows us to
(hopefully) reason better about intended behaviors and such moving
forward, but doesn't really do anything. As an example I for one
wouldn't mind to take it even one step further by introducing
"project" as another concept[1]. But that is for another day.

The second patch introduces property-drawers on document level. No
existing code will stop working, i.e. property keywords and all other
keywords will behave just as today.

The first five lines in the following example will work just as
property drawers inside headings with this patch. All commands and
functions that work with "regular" property-drawers are updated to
work also with this document level drawer.

#+begin_src org
  :PROPERTIES:
  :DIR: ~/
  :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
  :CATEGORY: Test
  :END:

  Section 1 before first headline.

  ,* TODO Headline 1
  Section 1 in first headline.

  ,** TODO Sub-headline 1                                              :Testtag:
  :PROPERTIES:
  :DIR:      _2018/1809 Spark/
  :CATEGORY: Test-cat
  :END:
#+end_src

I wouldn't be surprised if I've managed to hide a few bugs in the
code. But all tests, and a couple of new ones as well, signal an OK.

Thoughts? Comments?

Kind regards 
Gustav

[1] Sidenote: We already define projects today when we declare that
multiple files together are seen as our "agendas" for example. Or when
we configure publishing. But we lack a common framework for what a
"project" is in our code.

> -----Original Message-----
> From: Gustav Wikström
> Sent: den 31 augusti 2019 20:50
> To: emacs-orgmode@gnu.org
> Cc: Nicolas Goaziou <mail@nicolasgoaziou.fr>; Carsten Dominik
> <dominik@uva.nl>
> Subject: [RFC] Org document concept + document property drawers
> 
> Hi!
> 
> I'm continuing on my proposal to introduce a "document" element in
> org-mode and the idea of seeing everything before the first headline
> as the base level 0 outline for a file. I've attached two patches that
> I'd like some public review of before pushing to master.
> 
> Patch 0001 introduces the document element into org-element.el, and
> some restructuring related to that.
> 
> Patch 0002 makes it possible to use property drawers at the document
> level. I've hopefully covered all related commands to make this work.
> And I've added a bunch of tests to guard against future regressions.
> 
> Waiting for your comments!
> 
> Kind regards Gustav

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
@ 2019-09-01 16:08 Gustav Wikström
  2019-09-01 16:26 ` Adam Porter
  0 siblings, 1 reply; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 16:08 UTC (permalink / raw)
  To: adam@alphapapa.net; +Cc: emacs-orgmode@gnu.org

Hi Adam,

> This is a very interesting idea, and I don't want to dismiss your work,
> but I am concerned about how much third-party code will likely break by
> changing the results returned by org-element for parsing an Org buffer.
> I haven't thoroughly studied all of the code in your patches, so I may
> be wrong, but I think the breakage could be extensive.  For example,
> simple operations like destructuring the results of org-element parsing
> functions may be broken.  Have you done any investigation into this
> issue?
> 
> Maybe there should be a transitional period in which the existing
> org-element parsing functions would work as before, and the new
> document-level elements would be returned by a new org-element
> document-level parsing function.  Frankly, if there is breakage,the
> transition would probably take a few years, because there is a lot of
> code out there that has worked for years and may not be updated or
> replaced for years.

I have not investigated much into that to be honest. I'd argue that
it's a fairly trivial change in terms of the parser though. Everything
will work as before except when you're after the whole buffer
syntax-tree. In that case one will have to dig one step deeper into
the tree to find the content.

Previous tree: 
(org-data nil CONTENTS)

With this patch:
(org-data nil (document (doc-props) CONTENTS))

Yeah.. The structure changed a bit. But it's a fairly trivial change 
in my opinion. Everything else works as before AFAIK... But I might be 
overseeing something. Please enlighten me in that case!

I've made specification-changes only at two locations in the
test-cases for org-element. (ref. patch nr.1) 

Best
Gustav

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-09-01 15:25 ` Gustav Wikström
@ 2019-09-01 16:11   ` Adam Porter
  0 siblings, 0 replies; 13+ messages in thread
From: Adam Porter @ 2019-09-01 16:11 UTC (permalink / raw)
  To: emacs-orgmode

Gustav Wikström <gustav@whil.se> writes:

> Nicolas requested a more thorough introduction to the patch so here it
> comes.

Thank you, this is helpful.

> The first patch deals with formalities. It introduces one new greater
> element called "document". Parsers and everything around it are
> modified to work with this new concept. No new functionality is
> introduced. I'd call this patch an "enabler" since it allows us to
> (hopefully) reason better about intended behaviors and such moving
> forward, but doesn't really do anything.

I guess you mean that no new user-facing functionality is introduced,
because it does introduce new functionality into the code, and as you
say, it modifies "parsers and everything around it."

> The second patch introduces property-drawers on document level. No
> existing code will stop working, i.e. property keywords and all other
> keywords will behave just as today.

In my previous message in this thread, I asked about third-party
packages and how your code may impact them.  Would you please address
that question specifically?  Have you investigated this concern at all?


I'm especially concerned about the results of org-element parsing
functions changing, e.g. being nested inside an extra level of something
like (document ...) would likely break a lot of code.  As well, code
that expects to find "in-buffer settings" in the form of "#+KEYWORD:",
having been parsed by org-element, won't work properly if instead it
finds document-level property drawers that are supposed to apply to the
entire buffer.  It's a very significant change that would have rippling
effects on downstream code.

> The first five lines in the following example will work just as
> property drawers inside headings with this patch. All commands and
> functions that work with "regular" property-drawers are updated to
> work also with this document level drawer.
>
> #+begin_src org
>   :PROPERTIES:
>   :DIR: ~/
>   :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
>   :CATEGORY: Test
>   :END:
>
>   Section 1 before first headline.
>
>   ,* TODO Headline 1
>   Section 1 in first headline.
>
>   ,** TODO Sub-headline 1                            :Testtag:
>   :PROPERTIES:
>   :DIR:      _2018/1809 Spark/
>   :CATEGORY: Test-cat
>   :END:
> #+end_src

If I may be honest, I don't feel very enthusiastic about this
document-level property drawer.  I think it's because I'm accustomed to
thinking of property drawers as not affecting the entire document, and
expecting "#+KEYWORD:" for in-buffer settings.

I do recognize the advantage of being able to collapse them to hide
clutter.  However, as the manual explains, almost the same benefit can
be achieved by putting them in an outline node at the bottom of the
file, or in another file altogether:

    In-buffer settings may appear anywhere in the file, either directly
    or indirectly through a file included using `#+SETUPFILE: filename'
    syntax.

I usually put a "Config" or "Footer" node at the bottom of the file,
marked "COMMENT" and ":noexport:", containing such settings that I don't
want cluttering the top of the file.

> [1] Sidenote: We already define projects today when we declare that
> multiple files together are seen as our "agendas" for example. Or when
> we configure publishing. But we lack a common framework for what a
> "project" is in our code.

As you said, that's another issue--however, if I may, I'll point out
that that's your concept of what "project" means, and not all users
think of a "project" in those terms.  For example, it's not at all what
it means in "GTD" terms, which many Org users think in.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-09-01 16:08 [RFC] Org document concept + document property drawers Gustav Wikström
@ 2019-09-01 16:26 ` Adam Porter
  0 siblings, 0 replies; 13+ messages in thread
From: Adam Porter @ 2019-09-01 16:26 UTC (permalink / raw)
  To: emacs-orgmode

Gustav Wikström <gustav@whil.se> writes:

Hi Gustav,

Thanks for your reply.  I sent my most recent message, which reiterated
my question, before this message of yours came through.

>> This is a very interesting idea, and I don't want to dismiss your work,
>> but I am concerned about how much third-party code will likely break by
>> changing the results returned by org-element for parsing an Org buffer.
>> I haven't thoroughly studied all of the code in your patches, so I may
>> be wrong, but I think the breakage could be extensive.  For example,
>> simple operations like destructuring the results of org-element parsing
>> functions may be broken.  Have you done any investigation into this
>> issue?
>> 
>> Maybe there should be a transitional period in which the existing
>> org-element parsing functions would work as before, and the new
>> document-level elements would be returned by a new org-element
>> document-level parsing function.  Frankly, if there is breakage,the
>> transition would probably take a few years, because there is a lot of
>> code out there that has worked for years and may not be updated or
>> replaced for years.
>
> I have not investigated much into that to be honest. I'd argue that
> it's a fairly trivial change in terms of the parser though. Everything
> will work as before except when you're after the whole buffer
> syntax-tree. In that case one will have to dig one step deeper into
> the tree to find the content.
>
> Previous tree: 
> (org-data nil CONTENTS)
>
> With this patch:
> (org-data nil (document (doc-props) CONTENTS))
>
> Yeah.. The structure changed a bit. But it's a fairly trivial change 
> in my opinion. Everything else works as before AFAIK... But I might be 
> overseeing something. Please enlighten me in that case!

Thanks, that's a helpful illustration of the difference.

I'm glad that it only changes the result of parsing entire buffers.
That will limit the breakage.

However, there is code in the wild that does parse entire buffers that
way.  So that will break some code.  At least the necessary changes
would be minor.

However, if it does indeed precipitate a major Org version increment,
then such code will likely need to maintain compatibility with
older code for some time, because, like Emacs, Org versions tend to
stick around for a while.

Another question: how does this patch affect org-export backends?  I've
only a passing familiarity with them.  I'm guessing that some may break,
some of which may live outside of the Org repo.  It might be a good idea
to take a look at some of the ones on MELPA to see if they would be
affected.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
@ 2019-09-01 18:51 Gustav Wikström
  0 siblings, 0 replies; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 18:51 UTC (permalink / raw)
  To: adam@alphapapa.net; +Cc: emacs-orgmode@gnu.org

[-- Attachment #1: Type: text/plain, Size: 3688 bytes --]

> From:             Adam Porter

> > #+begin_src org
> >   :PROPERTIES:
> >   :DIR: ~/
> >   :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
> >   :CATEGORY: Test
> >   :END:
> >
> >   Section 1 before first headline.
> >
> >   ,* TODO Headline 1
> >   Section 1 in first headline.
> >
> >   ,** TODO Sub-headline 1                            :Testtag:
> >   :PROPERTIES:
> >   :DIR:      _2018/1809 Spark/
> >   :CATEGORY: Test-cat
> >   :END:
> > #+end_src
>
> If I may be honest, I don't feel very enthusiastic about this
> document-level property drawer.  I think it's because I'm accustomed to
> thinking of property drawers as not affecting the entire document, and
> expecting "#+KEYWORD:" for in-buffer settings.

You can think of property drawers in exactly the same way as before.
There is nothing magical about this drawer. Just think of a document
as a level-0 node. Things won't be inherited into level-1 and above
nodes unless explicitly set so by your configuration. In exactly the
same way as was previously done. To be clear - this property drawer is
not a substitute for all kinds of keywords - this drawer only aims at
substituting the "#+PROPERTY:" keyword. The end goal is alignment of
functionality, to make it more obvious both in code and for the user
what a property is and how it's defined.

> I do recognize the advantage of being able to collapse them to hide
> clutter.  However, as the manual explains, almost the same benefit can
> be achieved by putting them in an outline node at the bottom of the
> file, or in another file altogether:
>
>     In-buffer settings may appear anywhere in the file, either directly
>     or indirectly through a file included using `#+SETUPFILE: filename'
>     syntax.
>
> I usually put a "Config" or "Footer" node at the bottom of the file,
> marked "COMMENT" and ":noexport:", containing such settings that I don't
> want cluttering the top of the file.

Configurations is something else than setting properties in my opinion.
I've addressed configurations in my first mail (calling them "settings",
but "options" or "config" can be seen as synonyms if you like).
Keywords today is a source of confusion. They are the multi-purpose
swiss army knife for everything that didn't fit elsewhere. I'd like to
change that too - I've done a fairly rigorous investigation of the
multiple uses of keywords today. Simplifying how we use keywords will
aid both users and plug-in developers, I'm sure. But that, too, is
another topic! Not something that is a part of the patch I'm bringing
forward here anyways.

> > [1] Sidenote: We already define projects today when we declare that
> > multiple files together are seen as our "agendas" for example. Or when
> > we configure publishing. But we lack a common framework for what a
> > "project" is in our code.
>
> As you said, that's another issue--however, if I may, I'll point out
> that that's your concept of what "project" means, and not all users
> think of a "project" in those terms.  For example, it's not at all what
> it means in "GTD" terms, which many Org users think in.

Yes, project can be a dangerous word. A word of many meanings. I'm not
very attached to it in this context but think it fits well even though
it is overloaded. I'm a GTD'er myself and have learnt to separate the
meaning of the word (any overloaded word really) depending on context.
"Collection" might be another fitting word for what I have in mind.
I'm not sure this thread is the place to share my thoughts about Org
mode projects/collections though. We can start a separate thread for
that if you like!

Humbly
Gustav

[-- Attachment #2: Type: text/html, Size: 11240 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
@ 2019-09-01 19:12 Gustav Wikström
  0 siblings, 0 replies; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 19:12 UTC (permalink / raw)
  To: adam@alphapapa.net; +Cc: emacs-orgmode@gnu.org

> From: Adam Porter

> Gustav Wikström <address@hidden> writes:
> 
> > Previous tree: 
> > (org-data nil CONTENTS)
> >
> > With this patch:
> > (org-data nil (document (doc-props) CONTENTS))
> >
> > Yeah.. The structure changed a bit. But it's a fairly trivial change 
> > in my opinion. Everything else works as before AFAIK... But I might be 
> > overseeing something. Please enlighten me in that case!
> 
> Thanks, that's a helpful illustration of the difference.
> 
> I'm glad that it only changes the result of parsing entire buffers.
> That will limit the breakage.
> 
> However, there is code in the wild that does parse entire buffers that
> way.  So that will break some code.  At least the necessary changes
> would be minor.
> 
> However, if it does indeed precipitate a major Org version increment,
> then such code will likely need to maintain compatibility with
> older code for some time, because, like Emacs, Org versions tend to
> stick around for a while.
 
I'd like very much to avoid this patch be pushed into a major version
update. Which leads me to think of your first suggestion to maybe
provide some backwards-compatibility for the parser. Specifically for
the insertion of the document-node in the syntax-tree. I'm not sure
how to go about that in a nice way though.

> Another question: how does this patch affect org-export backends?  I've
> only a passing familiarity with them.  I'm guessing that some may break,
> some of which may live outside of the Org repo.  It might be a good idea
> to take a look at some of the ones on MELPA to see if they would be
> affected.

I'm relying heavily on the test suite of org-mode for this, but I've
only made a few lines of changes inside ox.el and as far as I can see
no breakage at all for the respective built in exporters. Which leads
me to believe that if some external exports are built on the ox.el
core they'll have the same no-breakage behavior. Either that, or we
have holes to patch in our test suite.

Thanks again Adam for your comments!

Regards
Gustav

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
@ 2019-09-01 19:17 Gustav Wikström
  0 siblings, 0 replies; 13+ messages in thread
From: Gustav Wikström @ 2019-09-01 19:17 UTC (permalink / raw)
  To: adam@alphapapa.net; +Cc: emacs-orgmode@gnu.org

(Resending this mail due to formatting-issues. Sorry!)

> From:	Adam Porter
> Subject:	Re: [O] [RFC] Org document concept + document property drawers
> Date:	Sun, 01 Sep 2019 11:11:22 -0500
>
> > #+begin_src org
> >   :PROPERTIES:
> >   :DIR: ~/
> >   :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
> >   :CATEGORY: Test
> >   :END:
> >
> >   Section 1 before first headline.
> >
> >   ,* TODO Headline 1
> >   Section 1 in first headline.
> >
> >   ,** TODO Sub-headline 1                            :Testtag:
> >   :PROPERTIES:
> >   :DIR:      _2018/1809 Spark/
> >   :CATEGORY: Test-cat
> >   :END:
> > #+end_src
> 
> If I may be honest, I don't feel very enthusiastic about this
> document-level property drawer.  I think it's because I'm accustomed to
> thinking of property drawers as not affecting the entire document, and
> expecting "#+KEYWORD:" for in-buffer settings.

You can think of property drawers in exactly the same way as before.
There is nothing magical about this drawer. Just think of a document
as a level-0 node. Things won't be inherited into level-1 and above
nodes unless explicitly set so by your configuration. In exactly the
same way as was previously done. To be clear - this property drawer is
not a substitute for all kinds of keywords - this drawer only aims at
substituting the "#+PROPERTY:" keyword. The end goal is alignment of
functionality, to make it more obvious both in code and for the user
what a property is and how it's defined.

> I do recognize the advantage of being able to collapse them to hide
> clutter.  However, as the manual explains, almost the same benefit can
> be achieved by putting them in an outline node at the bottom of the
> file, or in another file altogether:
> 
>     In-buffer settings may appear anywhere in the file, either directly
>     or indirectly through a file included using `#+SETUPFILE: filename'
>     syntax.
> 
> I usually put a "Config" or "Footer" node at the bottom of the file,
> marked "COMMENT" and ":noexport:", containing such settings that I don't
> want cluttering the top of the file.

Configurations is something else than setting properties in my opinion. 
I've addressed configurations in my first mail (calling them "settings",
but "options" or "config" can be seen as synonyms if you like).
Keywords today is a source of confusion. They are the multi-purpose
swiss army knife for everything that didn't fit elsewhere. I'd like to
change that too - I've done a fairly rigorous investigation of the
multiple uses of keywords today. Simplifying how we use keywords will
aid both users and plug-in developers, I'm sure. But that, too, is
another topic! Not something that is a part of the patch I'm bringing
forward here anyways.

> > [1] Sidenote: We already define projects today when we declare that
> > multiple files together are seen as our "agendas" for example. Or when
> > we configure publishing. But we lack a common framework for what a
> > "project" is in our code.
> 
> As you said, that's another issue--however, if I may, I'll point out
> that that's your concept of what "project" means, and not all users
> think of a "project" in those terms.  For example, it's not at all what
> it means in "GTD" terms, which many Org users think in.

Yes, project can be a dangerous word. A word of many meanings. I'm not
very attached to it in this context but think it fits well even though
it is overloaded. I'm a GTD'er myself and have learnt to separate the
meaning of the word (any overloaded word really) depending on context.
"Collection" might be another fitting word for what I have in mind.
I'm not sure this thread is the place to share my thoughts about Org
mode projects/collections though. We can start a separate thread for 
that if you like!

Humbly
Gustav

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-09-01 14:41   ` Gustav Wikström
@ 2019-09-06 20:09     ` Nicolas Goaziou
  2019-09-29  9:39       ` Gustav Wikström
  0 siblings, 1 reply; 13+ messages in thread
From: Nicolas Goaziou @ 2019-09-06 20:09 UTC (permalink / raw)
  To: Gustav Wikström; +Cc: emacs-orgmode@gnu.org

Hello,

Gustav Wikström <gustav@whil.se> writes:

> I'm starting out slow by making this a non-breaking change. At least
> I'm trying to. I.e. everything that worked before should continue to
> work.

This still is a syntax change, which is not to be taken lightly. Since
Org 9.3 is way overdue (I'm not blaming anyone for it, mind you),
I suggest to wait, and discuss about it with users meanwhile.

>> > Patch 0001 introduces the document element into org-element.el, and
>> > some restructuring related to that.
>> 
>> This should be explained in comments, and, if it lands at some point,
>> Worg pages about syntax and exporter should be updated, too.
>
> Which comments do you mean? I've tried to be rigorous in the patch
> notes. But you mean in the mail itself?

In the "org-element.el" file. There are some comments there. You could
add some info about the changes there.

>> Keep in mind that Org Element library should ultimately be as
>> independent as possible to the other parts of Org, including "org.el".
>
> Got it - is it preferred to do it the other way around - I.e. define
> the regexp in org-element.el rather than org.el if there is use of the
> regexp in both files?

Currently, syntax regexp are mainly defined in "org.el". At some point,
they could move into "org-element.el". Just something to keep in mind.

> Noted, except in this case I think it's warranted since
> org-back-to-heading behaves exactly the same with the exception of
> raising an error if before the first heading. Since both functions are
> defined next to another I'm sure a later refactor will take care of
> both since they're structurally identical and defined two lines apart.

Your call.

> I've just "pattern-matched" myself into that docstring. It matches the
> existing definitions of org-at-drawer-p, org-at-comment-p,
> org-at-block-p. Which are defined around this.

Then, these docstrings should be fixed too. Actually, "Non-nil if..." is
for variables with a boolean value. For predicates, like these
functions, it should be : "Return t if ...".

See (info "(elisp) Documentation Tips") for more information about the
subject.

> Hmm... I see your point. Have to think a bit here - because I have the
> feeling that defining the predicate using org-element-at-point will
> incur quite a performance hit.

Of course it will be slower. But "almost correct" predicates are also
bad, in other ways. Besides, it may be premature optimization at this
point.

> Or what is your intuition regarding that?

I think that if used correctly (i.e., not called too often), the
overhead can be acceptable. In fact, by experience, this kind of
function is hardly useful. I generally use Org Element higher up in
functions, and can operate on the element with Org Element function
without requesting another full parsing. YMMV.

Regards,

-- 
Nicolas Goaziou

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [RFC] Org document concept + document property drawers
  2019-09-06 20:09     ` Nicolas Goaziou
@ 2019-09-29  9:39       ` Gustav Wikström
  0 siblings, 0 replies; 13+ messages in thread
From: Gustav Wikström @ 2019-09-29  9:39 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode@gnu.org

Hi again,

I'm separating the Org document concept from the document property drawer to not block the document property drawer by syntax changes.

New RFC coming in a moment. Now only for the document property drawer functionality.

/G 

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Sent: den 6 september 2019 22:10
> To: Gustav Wikström <gustav@whil.se>
> Cc: emacs-orgmode@gnu.org
> Subject: Re: [RFC] Org document concept + document property drawers
> 
> Hello,
> 
> Gustav Wikström <gustav@whil.se> writes:
> 
> > I'm starting out slow by making this a non-breaking change. At least
> > I'm trying to. I.e. everything that worked before should continue to
> > work.
> 
> This still is a syntax change, which is not to be taken lightly. Since
> Org 9.3 is way overdue (I'm not blaming anyone for it, mind you),
> I suggest to wait, and discuss about it with users meanwhile.

I'll separate the patches and propose to go ahead with the document property drawer without the conceptual addition of Org documents in the parser.

> 
> >> > Patch 0001 introduces the document element into org-element.el, and
> >> > some restructuring related to that.
> >>
> >> This should be explained in comments, and, if it lands at some point,
> >> Worg pages about syntax and exporter should be updated, too.
> >
> > Which comments do you mean? I've tried to be rigorous in the patch
> > notes. But you mean in the mail itself?
> 
> In the "org-element.el" file. There are some comments there. You could
> add some info about the changes there.

Got it.

> 
> > Noted, except in this case I think it's warranted since
> > org-back-to-heading behaves exactly the same with the exception of
> > raising an error if before the first heading. Since both functions are
> > defined next to another I'm sure a later refactor will take care of
> > both since they're structurally identical and defined two lines apart.
> 
> Your call.

My call is to follow current pattern and make sure it's at least not adding any complexity for possible later refactor.

> 
> > I've just "pattern-matched" myself into that docstring. It matches the
> > existing definitions of org-at-drawer-p, org-at-comment-p,
> > org-at-block-p. Which are defined around this.
> 
> Then, these docstrings should be fixed too. Actually, "Non-nil if..." is
> for variables with a boolean value. For predicates, like these
> functions, it should be : "Return t if ...".
> 
> See (info "(elisp) Documentation Tips") for more information about the
> subject.

Fixed existing docstrings in separate patch already pushed to master.

> > Hmm... I see your point. Have to think a bit here - because I have the
> > feeling that defining the predicate using org-element-at-point will
> > incur quite a performance hit.
> 
> Of course it will be slower. But "almost correct" predicates are also
> bad, in other ways. Besides, it may be premature optimization at this
> point.

I've not made any changes to this just yet. It may be worth a revisit later though, because I do see your point. The issue is bigger than the one added predicate here though, and should most likely be looked at more holistically. I propose to take that in a separate change as refactoring isn't made more complicated by the patch I've suggested (although the amount of work will increase slightly).

BR
Gustav

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2019-09-29  9:40 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-01 16:08 [RFC] Org document concept + document property drawers Gustav Wikström
2019-09-01 16:26 ` Adam Porter
  -- strict thread matches above, loose matches on Subject: below --
2019-09-01 19:17 Gustav Wikström
2019-09-01 19:12 Gustav Wikström
2019-09-01 18:51 Gustav Wikström
2019-08-31 18:49 Gustav Wikström
2019-09-01  5:20 ` Adam Porter
2019-09-01 10:38 ` Nicolas Goaziou
2019-09-01 14:41   ` Gustav Wikström
2019-09-06 20:09     ` Nicolas Goaziou
2019-09-29  9:39       ` Gustav Wikström
2019-09-01 15:25 ` Gustav Wikström
2019-09-01 16:11   ` Adam Porter

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).