Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser

[Ihor Radchenko <yantar92@gmail.com>]

Max Nikulin <manikulin@gmail.com> writes:

> On 11/12/2021 21:39, Ihor Radchenko wrote:
>> 
>> The attached patch is an attempt to create something like shared repo
>> for Org element parser tests.
>
> "[PATCH]" prefix in the subject might be a reason why you message 
> received less attention than it should.

Well. I really wanted to set a technical tone of the discussion in
contrast to my previous email (proposing this exact idea, among others)
that generated little technical feedback and a lot of non-technical.

> I think, the set should contain hundreds of tests to be helpful, thus 2 
> files per test will likely be inconvenient since most of samples will be 
> short. I suggest to group test input and results into large files. Such 
> tests should be augmented by some metadata: keywords (labels, tags). 
> E.g. besides heading
>
> - sample: "* Simple Heading"
>    keywords: heading

I am not sure here. In a way, we already have such a format in
test-org-element.el:

(ert-deftest test-org-element/bold-parser ()
  "Test `bold' parser."
  ;; Standard test.
  (should
   (org-test-with-temp-text "*bold*"
     (org-element-map (org-element-parse-buffer) 'bold #'identity nil t))))

The problem with grouping short tests into a single file is that we put
an extra requirement on the testing code. The code will have to parse
the test files with multiple tests, extract those tests, and then run
the parser. So, in addition to writing the Org parser, third-party dev
will also have to write a parser for the test format. I find individual
files easier to get started with. Most existing parser libraries can
handle individual files, but not individual pieces of text grouped into
bigger file using yet another standard convention.

The usual solution to the above problem is fixed test file format that
can be processed by testing module of a given language. But is there a
standard multi-test in one file format that can be used in multiple
programming languages?

Having said that, I like your idea about adding metadata to the tests.
Probably, we can simply do it in the readme like the following:

* Test group heading
  - file :: file:./test-files/test1.org
    - keywords :: k1, k2
    - description :: description text

Or even allow inline tests via buffer-local after-save-hook:

   - text :: "*Bold* emphasis"
     - keywords :: emphasis, heading
     - description :: Despite line is started from a star, there is no
                      space after it, so it is not a heading.

Upon save, the text:: field will be automatically converted to a
test-hash.org+test-hash.el files.

> Version of Org and test set should be included into metadata for the 
> whole suite.

If the test set is distributed together with Org, I see no reason to do
it. Otherwise, the test set should simply track bugfix and main branches
of Org and follow Org releases. Does it sound reasonable?

> Since partial compliance is assumed, format of test results should be 
> declared as well to be able to publish overview or comparison.

I provided a short description of the format in the README (see the
original patch). Is it not enough?

> Are properties like :begin and :end mandatory for reference results of 
> parsing? They make structures more verbose and harder to read. Often it 
> is enough to compare content and similar properties.

I afraid that if we put contents of every headline/element in place of
:begin :end, the results will be even less readable. The results will
have to dump multiple cumulative instances of the original file.

Or do you have an alternative suggestion about the format of the
reference parser output?

>> Any comments or suggestions?
>> I am particularly looking for thoughts about licensing and possible
>> distribution of the test set in separate repository.
>
> Since these tests will unlikely become a part of some software, I do not 
> think that GPL may be an obstacle for any project. Requirement of signed 
> consent will likely prevent contributing of new cases from some developers.

Actually, there have been talks about including Org mode tests into
Emacs itself (https://yhetil.org/emacs-devel/87o8629h8g.fsf_-_@gmx.de/)

Not all our tests are under GPL, but we may want to change this
situation.

Also, if we decide to not distribute this test set under GPL, what about
the usual fear with someone contributing and its persons' employer
coming on us with copyright claims? I do not have enough knowledge to
judge this.

Maybe Bastien or others know better. Or we may try to contact GNU legal
team.

Best,
Ihor