[PATCH] org-test: Create a collaborative test set for Org buffer parser

[Ihor Radchenko <yantar92@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Dear all,

The attached patch is an attempt to create something like shared repo
for Org element parser tests.

The idea is moving the tests out from Elisp into a set of text files.
That way, anyone interested in developing Org syntax support can use our
tests and potentially contribute more test files to the benefit of Org
mode for Emacs.

The test set is essentially a series of .org files alongside .el files
containing normalised output of `org-element-parse-buffer'. (see the
patch)

Anyone can contribute to the test set by adding new .org files and
generating the canonical parser output with new
M-x test-org-element-parser-save-expected-result function.

README.org in the repo also serves as a test file 😝.

Any comments or suggestions?
I am particularly looking for thoughts about licensing and possible
distribution of the test set in separate repository.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-test-Create-a-separate-testset-for-Org-element-p.patch --]
[-- Type: text/x-diff, Size: 14719 bytes --]

From f7bb947517e8793a45864b614f06460d1132539d Mon Sep 17 00:00:00 2001
Message-Id: <f7bb947517e8793a45864b614f06460d1132539d.1639232929.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 11 Dec 2021 22:24:39 +0800
Subject: [PATCH] org-test: Create a separate testset for Org element parser

* testing/lisp/test-org-element-parser-sources/README.org: Add readme
file describing the test file format and organisation.

* testing/lisp/test-org-element-parser-sources/simple-heading.org: Add
an example test file.

* testing/lisp/test-org-element-parser-sources/README.el:
* testing/lisp/test-org-element-parser-sources/simple-heading.el: Add
normalised expected parser output files.

* testing/lisp/test-org-element-parser.el: New testset integration to
main Org test suite.  The file defines ERT tests for files inside
test-org-element-parser-sources and an interactive function
`test-org-element-parser-save-expected-result' to generate parser
output files.
---
 .../test-org-element-parser-sources/README.el |  42 ++++++
 .../README.org                                |  64 +++++++++
 .../simple-heading.el                         |  11 ++
 .../simple-heading.org                        |   5 +
 testing/lisp/test-org-element-parser.el       | 129 ++++++++++++++++++
 5 files changed, 251 insertions(+)
 create mode 100644 testing/lisp/test-org-element-parser-sources/README.el
 create mode 100644 testing/lisp/test-org-element-parser-sources/README.org
 create mode 100644 testing/lisp/test-org-element-parser-sources/simple-heading.el
 create mode 100644 testing/lisp/test-org-element-parser-sources/simple-heading.org
 create mode 100644 testing/lisp/test-org-element-parser.el

diff --git a/testing/lisp/test-org-element-parser-sources/README.el b/testing/lisp/test-org-element-parser-sources/README.el
new file mode 100644
index 000000000..852df032f
--- /dev/null
+++ b/testing/lisp/test-org-element-parser-sources/README.el
@@ -0,0 +1,42 @@
+(org-data
+ (:begin 1 :contents-begin 2 :contents-end 1306 :end 1306 :post-affiliated 1 :post-blank 0)
+ (section
+  (:begin 2 :contents-begin 2 :contents-end 837 :end 838 :post-affiliated 2 :post-blank 1)
+  (paragraph
+   (:begin 2 :contents-begin 2 :contents-end 51 :end 52 :post-affiliated 2 :post-blank 1)
+   "This is a shared test suite for Org mode syntax.\n")
+  (paragraph
+   (:begin 52 :contents-begin 52 :contents-end 247 :end 248 :post-affiliated 52 :post-blank 1)
+   "The test suite consists of a number of .org example files alongside\nwith the expected parser output.  Each .org file can be parsed as is\nand the result should match the corresponding .el file.  \n")
+  (paragraph
+   (:begin 248 :contents-begin 248 :contents-end 424 :end 425 :post-affiliated 248 :post-blank 1)
+   "The parser results in .el files are Emacs sexps.  Each sexp is an\noutput of "
+   (verbatim
+    (:begin 324 :end 351 :post-blank 1))
+   "stripped from unessential\nproperties.  Each sexp has the following form:\n")
+  (src-block
+   (:begin 425 :end 773 :post-affiliated 425 :post-blank 1))
+  (paragraph
+   (:begin 773 :contents-begin 773 :contents-end 837 :end 837 :post-affiliated 773 :post-blank 0)
+   "The properties of elements can be specified in arbitrary order.\n"))
+ (headline
+  (:archivedp nil :begin 838 :commentedp nil :contents-begin 854 :contents-end 1306 :end 1306 :footnote-section-p nil :level 1 :post-affiliated 838 :post-blank 0 :pre-blank 1 :priority nil :raw-value "Contributing" :tags nil :title
+	      ("Contributing")
+	      :todo-keyword nil :todo-type nil)
+  (section
+   (:begin 854 :contents-begin 854 :contents-end 1306 :end 1306 :post-affiliated 854 :post-blank 0)
+   (paragraph
+    (:begin 854 :contents-begin 854 :contents-end 983 :end 984 :post-affiliated 854 :post-blank 1)
+    "To add new test files to this suite, send a patch to Org mode mailing\nlist, as described in "
+    (link
+     (:begin 946 :contents-begin nil :contents-end nil :end 981 :post-blank 0))
+    ".\n")
+   (paragraph
+    (:begin 984 :contents-begin 984 :contents-end 1306 :end 1306 :post-affiliated 984 :post-blank 0)
+    "The expected parser output can be generated using Emacs and latest\nversion of Org mode.  You need to open an Org file in Emacs, load\n"
+    (verbatim
+     (:begin 1117 :end 1180 :post-blank 0))
+    ", and\nrun "
+    (verbatim
+     (:begin 1190 :end 1240 :post-blank 0))
+    ".  The expected\noutput will be saved alongside with the Org file.\n"))))
diff --git a/testing/lisp/test-org-element-parser-sources/README.org b/testing/lisp/test-org-element-parser-sources/README.org
new file mode 100644
index 000000000..78e33eb36
--- /dev/null
+++ b/testing/lisp/test-org-element-parser-sources/README.org
@@ -0,0 +1,64 @@
+#+TITLE: Shared Org parser testing fileset
+#+AUTHOR: Ihor Radchenko
+#+EMAIL: yantar92 at gmail dot com
+
+This is a shared test suite for Org mode syntax.  It is a part of Org
+mode's own test suite extracted for easier contributions.
+
+The test suite consists of a number of .org example files alongside
+with the expected parser output.  Each .org file can be parsed as is
+and the result should match the corresponding .el file.  
+
+The parser results in .el files are Emacs sexps.  Each sexp is an
+output of =org-element-parse-buffer= stripped from unessential
+properties.  Each sexp has the following form:
+
+#+begin_src emacs-lisp
+(org-data
+ (:property1 value-1 :property2 value-2 ...)
+ (inner-element-1 
+  (:inner-element-property1 value ...)
+  ... <other elements inside inner-element-1>)
+ ...
+ "string element is not a list, but a string"
+ ...
+ (heading
+  (:property1 value1 ... :title (object-inside-title1 (<properties>) ...))
+  ...)
+ ...)
+#+end_src
+
+The properties of elements can be specified in arbitrary order.
+
+The common properties are =:begin=, =:end=, =:contents-begin=, and
+=:contents-end=.  Their values are 1-indexed char positions from the
+beginning of the Org file.
+
+* Contributing
+
+To add new test files to this suite, send a patch to Org mode mailing
+list, as described in https://orgmode.org/contribute.html.
+
+The expected parser output can be generated using Emacs and latest
+version of Org mode.  You need to open an Org file in Emacs, load
+=/path/to/Org/git/repo/testing/lisp/test-org-element-parser.el=, and
+run =M-x test-org-element-parser-save-expected-result=.  The expected
+output will be saved alongside with the Org file.
+
+* License
+
+Org-mode is published under [[https://www.gnu.org/licenses/gpl-3.0.html][the GNU GPLv3 license]] or any later
+version, the same as GNU Emacs.
+
+Org-mode is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Org mode.  If not, see <https://www.gnu.org/licenses/>.
diff --git a/testing/lisp/test-org-element-parser-sources/simple-heading.el b/testing/lisp/test-org-element-parser-sources/simple-heading.el
new file mode 100644
index 000000000..6ca7a54f6
--- /dev/null
+++ b/testing/lisp/test-org-element-parser-sources/simple-heading.el
@@ -0,0 +1,11 @@
+(org-data
+ (:begin 1 :contents-begin 3 :contents-end 46 :end 46 :post-affiliated 1 :post-blank 0)
+ (headline
+  (:archivedp nil :begin 3 :commentedp nil :contents-begin 24 :contents-end 46 :end 46 :footnote-section-p nil :level 1 :post-affiliated 3 :post-blank 0 :pre-blank 1 :priority nil :raw-value "this is a heading" :tags nil :title
+	      ("this is a heading")
+	      :todo-keyword nil :todo-type nil)
+  (section
+   (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0)
+   (paragraph
+    (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0)
+    "With some text below.\n"))))
diff --git a/testing/lisp/test-org-element-parser-sources/simple-heading.org b/testing/lisp/test-org-element-parser-sources/simple-heading.org
new file mode 100644
index 000000000..b508ecfec
--- /dev/null
+++ b/testing/lisp/test-org-element-parser-sources/simple-heading.org
@@ -0,0 +1,5 @@
+
+
+* this is a heading
+
+With some text below.
diff --git a/testing/lisp/test-org-element-parser.el b/testing/lisp/test-org-element-parser.el
new file mode 100644
index 000000000..d12307d98
--- /dev/null
+++ b/testing/lisp/test-org-element-parser.el
@@ -0,0 +1,129 @@
+;;; test-org-element-parser.el --- Tests for org-element.el parser
+
+;; Copyright (C) 2021  Ihor Radchenko
+
+;; Author: Ihor Radchenko <yantar92 at gmail dot com>
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Code:
+
+(require 'org-element)
+
+(defvar test-org-element-parser-properties
+  '((:global :begin :end :contents-begin :contents-end :pre-blank :post-blank :post-affiliated)
+    (headline :raw-value :title :level :priority :tags :todo-keyword :todo-type :footnote-section-p :archivedp :commentedp))
+  "List of important properties that should be parsed.")
+
+(defvar test-org-element-parser-source-directory "../lisp/test-org-element-parser-sources/"
+  "Path to directory containing all the test Org files.
+The expected parsed representation is stored alongside in .el files.
+For example, parsed representation of file.org is in file.el.")
+
+(defun test-org-element-parser-generate-syntax-sexp ()
+  "Return SEXP with important parts of parsed representation of current Org buffer."
+  (unless (derived-mode-p 'org-mode) (user-error "Not an Org buffer."))
+  (let ((datum (org-element-parse-buffer 'object))
+	(strip-func (lambda (el)
+		      (let ((type (org-element-type el))
+			    (plist (when (listp el) (nth 1 el)))
+			    prop value tmpalist)
+                        (if (eq type 'plain-text)
+                            (set-text-properties 0 (length el) nil el)
+			  (while plist
+			    (setq prop (car plist))
+			    (setq value (cadr plist))
+			    (when (stringp value) (setq value (substring-no-properties value)))
+			    (setq plist (cddr plist))
+			    (when (or (memq prop (alist-get :global test-org-element-parser-properties))
+				      (memq prop (alist-get type test-org-element-parser-properties)))
+                              (push (cons prop value) tmpalist)))
+                          (setq tmpalist (sort tmpalist (lambda (a b) (string< (symbol-name (car a))
+                                                                          (symbol-name (car b))))))
+			  (setf (nth 1 el)
+                                (apply #'append
+                                       (mapcar (lambda (c) (list (car c) (cdr c)))
+                                               tmpalist))))))))
+    (org-element-map datum (append '(plain-text)  org-element-all-elements org-element-all-objects)
+      strip-func nil nil nil 'with-affiliated)
+    ;; `org-element-map' never maps over `org-data'. Update it separately.
+    (funcall strip-func datum)
+    datum))
+
+(defun test-org-element-parser-save-expected-result (&optional file)
+  "Save reference parsed representation of current Org buffer or FILE.
+The parsed representation will be saved alongside with the buffer file."
+  (interactive)
+  (with-current-buffer (if file
+			   (get-buffer-create file)
+                         (current-buffer))
+    (save-buffer)
+    (let ((datum (test-org-element-parser-generate-syntax-sexp))
+	  (path (buffer-file-name))
+          newpath)
+      (unless (and path (file-exists-p path)) (user-error "Not in a file buffer."))
+      (setq newpath (format "%s.el" (file-name-base path)))
+      (with-temp-file newpath
+        (condition-case err
+            (progn
+	      (pp datum (current-buffer))
+              (message "Parsed representation saved to %s" (expand-file-name newpath)))
+          (err (message "Failed to save parsed representation: \"%S\"" err)))))))
+
+(defmacro org-test-element-verify (&optional file)
+  "Verify `org-element-parse-buffer' for current Org buffer or FILE."
+  `(progn
+     (unless ,file
+       (setq file (buffer-file-name))
+       (save-buffer))
+     (unless (and ,file (file-exists-p ,file))
+       (user-error "%s does not exist." ,file))
+     (let ((reference-file (format "%s%s.el"
+                                   (file-name-directory ,file)
+                                   (file-name-base ,file))))
+       (unless (file-exists-p reference-file)
+         (user-error "Reference result file %s does not exist." reference-file))
+       (with-temp-buffer
+         (insert-file-contents ,file)
+         (org-mode)
+         (should
+          (equal (test-org-element-parser-generate-syntax-sexp)
+	         (with-temp-buffer
+                   (insert-file-contents reference-file)
+	           (read (current-buffer)))))))))
+
+(defmacro test-org-element-parser-files (&rest files)
+  "Run `org-test-element-verify' for each file in FILES."
+  `(progn
+     (unless (and test-org-element-parser-source-directory
+                  (file-exists-p test-org-element-parser-source-directory))
+       (error "%s does not exist." test-org-element-parser-source-directory))
+     (dolist (file '(,@files))
+       (setq file (format "%s%s.org"
+                          (file-name-as-directory test-org-element-parser-source-directory)
+                          (file-name-base file)))
+       (org-test-element-verify file))))
+
+\f
+
+(ert-deftest test-org-element-parser/simple-headlines ()
+  "Basic tests for Org files with headings and plain text paragraphs."
+  (test-org-element-parser-files "simple-heading"))
+
+(ert-deftest test-org-element-parser/README ()
+  "Test README.org in the example file repo."
+  (test-org-element-parser-files "README"))
+
+(provide 'test-org-element-parser)
+;;; test-org-element-parser.el ends here
-- 
2.32.0