Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser

[Max Nikulin <manikulin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7012 bytes --]

Sorry for delay. I expected that a next step with inline raw AST 
fragments would be easier, but I have got working example for parser 
tests grouped in single file earlier.

Certainly org files may be source for parser tests. They are readable 
enough to be convenient for developers and, I hope, it would not require 
a lot of work to convert them in machine-readable files. However tests 
should be distributed in some format(s) with widely available parsers:
s-expressions, YAML, JSON. I am against JSON for source files since 
working with multiline text will likely be a pain (besides editing of 
files, patches and VCS diffs should be taken into account).

Actually there is a requirement for external projects related to parser 
already. Expected results are provided as S-expressions.

For simplicity I described a test suite with a couple of cases using 
s-expressions (see attachments). Names of macro an functions are 
arbitrary. Interface should be like

     (demo-parse-test-suite "demo-test-suite.el")

Current implementation creates an ert test for each entry in such file, 
but it may be changed later. I just had example of macro for 
parametrized test from earlier experiments.

On 15/12/2021 20:23, Ihor Radchenko wrote:
> Max Nikulin writes:
>> On 11/12/2021 21:39, Ihor Radchenko wrote:
>>>
>>> The attached patch is an attempt to create something like shared repo
>>> for Org element parser tests.
>>
>> "[PATCH]" prefix in the subject might be a reason why you message
>> received less attention than it should.

At least Tom and Timothy have their parsers, so I expected some response 
from them. I am unsure whether posts about Organice were from developers 
or from users. Patch is something too internal to development of namely 
Org Mode as Emacs major mode. Once I asked to avoid proposal to 
integrate everything meaning mostly particular person who was extremely 
active that time, but it had effect on more people than I thought. You 
intention may have stronger result than it should as well.

> I am not sure here. In a way, we already have such a format in
> test-org-element.el:
> 
> (ert-deftest test-org-element/bold-parser ()
>    "Test `bold' parser."
>    ;; Standard test.
>    (should
>     (org-test-with-temp-text "*bold*"
>       (org-element-map (org-element-parse-buffer) 'bold #'identity nil t))))

It is a bit harder to parse than it is acceptable for external tools. 
However it may be converted to more convenient format using elisp 
facilities.

> So, in addition to writing the Org parser, third-party dev
> will also have to write a parser for the test format.

I assume available parser for some wide spread format and some simple 
interpretation of the structures specific to these tests.

> Most existing parser libraries can
> handle individual files, but not individual pieces of text grouped into
> bigger file using yet another standard convention.

It should take a several line of code to put string input into a 
temporary file. Likely it is possible to implement interface of file 
stream for input string. Finally, it may stimulate developers to add a 
method that parses strings.

I found the following test corpus impressive and unmanageable in the 
form of separate files:
https://github.com/tgbugs/laundry/blob/master/laundry/test.rkt

> But is there a
> standard multi-test in one file format that can be used in multiple
> programming languages?

Some testing frameworks have facilities for parametrized tests with 
lists of arbitrary structures, others explicitly recommends to write a 
loop over such lists.

> Or even allow inline tests via buffer-local after-save-hook:
> 
>     - text :: "*Bold* emphasis"
>       - keywords :: emphasis, heading
>       - description :: Despite line is started from a star, there is no
>                        space after it, so it is not a heading.
> 
> Upon save, the text:: field will be automatically converted to a
> test-hash.org+test-hash.el files.

Or to a single file with some top-level metadata and a list (maybe 
nested lists) of tests. Unsure that such files should be stored in git.

>> Version of Org and test set should be included into metadata for the
>> whole suite.
> 
> If the test set is distributed together with Org, I see no reason to do
> it. Otherwise, the test set should simply track bugfix and main branches
> of Org and follow Org releases. Does it sound reasonable?

I have no particular opinion if test sources should be maintained in the 
same repository with Org or Emacs. If test are described in Org files 
then test suites should be distributed using some other way.

>> Since partial compliance is assumed, format of test results should be
>> declared as well to be able to publish overview or comparison.
> 
> I provided a short description of the format in the README (see the
> original patch). Is it not enough?

I mean result of *group* of tests to present comparison of parsers. 
Unsure if JUnit XML files are appropriate.

>> Are properties like :begin and :end mandatory for reference results of
>> parsing? They make structures more verbose and harder to read. Often it
>> is enough to compare content and similar properties.
> 
> I afraid that if we put contents of every headline/element in place of
> :begin :end, the results will be even less readable. The results will
> have to dump multiple cumulative instances of the original file.

I had in mind that only leaf elements have contents. It may result in 
some ambiguity however.

> Actually, there have been talks about including Org mode tests into
> Emacs itself (https://yhetil.org/emacs-devel/87o8629h8g.fsf_-_@gmx.de/)

I have seen proposals to drop separate repository for Org and do 
everything inside the one for Emacs earlier. In my opinion, it would 
mean broken compatibility with earlier version of Emacs, so I do not 
like it. On the other hand I am not an active developer, so my arguments 
may be ignored.

P.S.

Output or Ert requires more tuning:

---- >8 ----

Selector: t
Passed:  1
Failed:  1 (1 unexpected)
Skipped: 0
Total:   2/2

Started at:   2021-12-19 14:13:54+0700
Finished.
Finished at:  2021-12-19 14:13:54+0700

F.

F demo-parse-test/demo-test-suite--bold-emphasis-at-beginning-of-line
     Bold text marker at beginning of line should not be confused with 
heading: no space after star (‘demo-parse-test/demo-test-suite’)
     (ert-test-failed
      ((should
        (funcall #<subr equal>
		 '(org-data ... ...)
		 (apply #'demo-test-parse-input '...)))
       :form
       (funcall #<subr equal>
		(org-data
		 (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 
1 ...)
		 (paragraph
		  (:begin 1 :contents-begin 1 :contents-end 13 :end 13 
:post-affiliated 1 ...)
		  (bold ... "Bold text")))
		(org-data
		 (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 
1 ...)
		 (section
		  (:begin 1 :contents-begin 1 :contents-end 13 :end 13 
:post-affiliated 1 ...)
		  (paragraph ... ... "
"))))
       :value nil))



[-- Attachment #2: demo-test-suite.el --]
[-- Type: text/x-emacs-lisp, Size: 1420 bytes --]

(
 :description
 "Just a demo of test suite for org-element parser"
 :cases
 (
  (:id simple-heading
       :description "Heading and section with paragraph"
       :input "

* this is a heading

With some text below.
"
       :result (org-data
		(:begin 1 :contents-begin 3 :contents-end 46 :end 46 :post-affiliated 1 :post-blank 0)
		(headline
		  (:archivedp nil :begin 3 :commentedp nil :contents-begin 24 :contents-end 46 :end 46 :footnote-section-p nil :level 1 :post-affiliated 3 :post-blank 0 :pre-blank 1 :priority nil :raw-value "this is a heading" :tags nil :title
			      ("this is a heading")
			      :todo-keyword nil :todo-type nil)
		  (section
		    (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0)
		    (paragraph
		      (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0)
		      "With some text below.\n")))))
  (:id bold-emphasis-at-beginning-of-line
       :description "Bold text marker at beginning of line should not be confused with heading: no space after star"
       :input "*Bold text*
"
       :result (org-data
		(:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 :post-blank 0)
		    (paragraph
		      (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 :post-blank 0)
		      (bold
			(:begin 1 :contents-begin 2 :contents-end 11 :end 13)
			"Bold text"))))
))

[-- Attachment #3: demo-test.el --]
[-- Type: text/x-emacs-lisp, Size: 2666 bytes --]

;;;; Example of usage:
;;;;      (demo-parse-test-suite "demo-test-suite.el")

(defmacro nm-deftest-parametrized
    (prefix-sym func-predicate &rest doc-cases)
  "Define parametrized test

For each SUFFIX, CASE-DOCSTRING, EXPECTATION, ARGS list
call `ert-deftest' with SUITE--SUFFIX name, CASE-DOCSTRING,
and `should' that checks whether EXPECTATION is consistent
with result of FUNCTION applied to ARGS using PREDICATE or `equal'.

\(fn SUITE (FUNCTION [PREDICATE])  \
  [SUITE-DOCSTRING] \
  (SUFFIX CASE-DOCSTRING EXPECTATION ARGS...)...)"
  (declare (debug (&define [&name "test@" symbolp] sexp
			   [&optional strinp] def-body))
	   (indent 2)
	   (doc-string 3))
  (let* ((func (car func-predicate))
	 (predicate (or (cadr func-predicate) (symbol-function 'equal)))
	 (prefix (symbol-name prefix-sym))
	 (maybe-doc (car doc-cases))
	 (cases (if (stringp maybe-doc) (cdr doc-cases) doc-cases))
	 (case-list (mapcar
		     (lambda (case)
		       (format "- `%s--%s'"
			       prefix
			       (symbol-name (car case))))
		     cases))
	 ;; Unfortunately `ert-describe-test' works only in ert mode
	 ;; and links to particular subtests are inactive.
	 (suite-doc (if (stringp maybe-doc)
			(cons maybe-doc case-list)
		      case-list)))
    (append
     `(,#'progn
	;; A function to assing doc string that is linked from each test.
	(defun ,prefix-sym () ,(mapconcat #'identity suite-doc "\n")))
     (mapcar
      (lambda (case)
	;; Have not managed to express "&rest" using `pcase-let'.
	(apply (lambda (id-sym case-docstring expectation &rest args)
		 (let* ((id (symbol-name id-sym))
			(name (intern (concat prefix "--" id)))
			(docstring (format "%s (`%s')" case-docstring prefix)))
		   `(ert-deftest ,name ()
		      ,docstring
		      (should (funcall ,predicate ,expectation
				     (apply ,func (quote ,args)))))))
	       case))
      cases))))

(defun demo-test-parse-input (text)
  (with-temp-buffer
    (insert text)
    (org-mode)
    (test-org-element-parser-generate-syntax-sexp)))

(defmacro demo-parse-test-suite (file-name)
  (let* ((prefix (file-name-base file-name))
	 (prefix-sym (intern (concat "demo-parse-test/" prefix)))
	 (suite (with-temp-buffer
		  (insert-file-contents file-name)
		  (read (current-buffer))))
	 (case-list (plist-get suite :cases))
	 (suite-description (plist-get suite :description)))
    `(nm-deftest-parametrized
	 ,prefix-sym (#'demo-test-parse-input)
       ,suite-description
       ,@(mapcar (lambda (case)
		  (list (plist-get case :id)
			(or (plist-get case :description) "Warning: no description for this case")
			`(quote ,(plist-get case :result))
			(plist-get case :input)))
		case-list))))

[-- Attachment #4: demo-test-suite.json --]
[-- Type: application/json, Size: 2620 bytes --]

[-- Attachment #5: demo-test-suite.yaml --]
[-- Type: application/x-yaml, Size: 1675 bytes --]