From: Ihor Radchenko <firstname.lastname@example.org> To: Max Nikulin <email@example.com> Cc: firstname.lastname@example.org Subject: Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser Date: Wed, 15 Dec 2021 21:23:26 +0800 [thread overview] Message-ID: <871r2et31t.fsf@localhost> (raw) In-Reply-To: <email@example.com> Max Nikulin <firstname.lastname@example.org> writes: > On 11/12/2021 21:39, Ihor Radchenko wrote: >> >> The attached patch is an attempt to create something like shared repo >> for Org element parser tests. > > "[PATCH]" prefix in the subject might be a reason why you message > received less attention than it should. Well. I really wanted to set a technical tone of the discussion in contrast to my previous email (proposing this exact idea, among others) that generated little technical feedback and a lot of non-technical. > I think, the set should contain hundreds of tests to be helpful, thus 2 > files per test will likely be inconvenient since most of samples will be > short. I suggest to group test input and results into large files. Such > tests should be augmented by some metadata: keywords (labels, tags). > E.g. besides heading > > - sample: "* Simple Heading" > keywords: heading I am not sure here. In a way, we already have such a format in test-org-element.el: (ert-deftest test-org-element/bold-parser () "Test `bold' parser." ;; Standard test. (should (org-test-with-temp-text "*bold*" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t)))) The problem with grouping short tests into a single file is that we put an extra requirement on the testing code. The code will have to parse the test files with multiple tests, extract those tests, and then run the parser. So, in addition to writing the Org parser, third-party dev will also have to write a parser for the test format. I find individual files easier to get started with. Most existing parser libraries can handle individual files, but not individual pieces of text grouped into bigger file using yet another standard convention. The usual solution to the above problem is fixed test file format that can be processed by testing module of a given language. But is there a standard multi-test in one file format that can be used in multiple programming languages? Having said that, I like your idea about adding metadata to the tests. Probably, we can simply do it in the readme like the following: * Test group heading - file :: file:./test-files/test1.org - keywords :: k1, k2 - description :: description text Or even allow inline tests via buffer-local after-save-hook: - text :: "*Bold* emphasis" - keywords :: emphasis, heading - description :: Despite line is started from a star, there is no space after it, so it is not a heading. Upon save, the text:: field will be automatically converted to a test-hash.org+test-hash.el files. > Version of Org and test set should be included into metadata for the > whole suite. If the test set is distributed together with Org, I see no reason to do it. Otherwise, the test set should simply track bugfix and main branches of Org and follow Org releases. Does it sound reasonable? > Since partial compliance is assumed, format of test results should be > declared as well to be able to publish overview or comparison. I provided a short description of the format in the README (see the original patch). Is it not enough? > Are properties like :begin and :end mandatory for reference results of > parsing? They make structures more verbose and harder to read. Often it > is enough to compare content and similar properties. I afraid that if we put contents of every headline/element in place of :begin :end, the results will be even less readable. The results will have to dump multiple cumulative instances of the original file. Or do you have an alternative suggestion about the format of the reference parser output? >> Any comments or suggestions? >> I am particularly looking for thoughts about licensing and possible >> distribution of the test set in separate repository. > > Since these tests will unlikely become a part of some software, I do not > think that GPL may be an obstacle for any project. Requirement of signed > consent will likely prevent contributing of new cases from some developers. Actually, there have been talks about including Org mode tests into Emacs itself (https://email@example.com/) Not all our tests are under GPL, but we may want to change this situation. Also, if we decide to not distribute this test set under GPL, what about the usual fear with someone contributing and its persons' employer coming on us with copyright claims? I do not have enough knowledge to judge this. Maybe Bastien or others know better. Or we may try to contact GNU legal team. Best, Ihor
next prev parent reply other threads:[~2021-12-15 14:20 UTC|newest] Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-12-11 14:39 Ihor Radchenko 2021-12-14 16:16 ` Max Nikulin 2021-12-14 20:27 ` Tim Cross 2021-12-15 13:44 ` Ihor Radchenko 2021-12-15 13:23 ` Ihor Radchenko [this message] 2021-12-19 8:23 ` Max Nikulin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://www.orgmode.org/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=871r2et31t.fsf@localhost \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).