From: Max Nikulin <email@example.com> To: firstname.lastname@example.org Subject: Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser Date: Sun, 19 Dec 2021 15:23:32 +0700 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <871r2et31t.fsf@localhost> [-- Attachment #1: Type: text/plain, Size: 7012 bytes --] Sorry for delay. I expected that a next step with inline raw AST fragments would be easier, but I have got working example for parser tests grouped in single file earlier. Certainly org files may be source for parser tests. They are readable enough to be convenient for developers and, I hope, it would not require a lot of work to convert them in machine-readable files. However tests should be distributed in some format(s) with widely available parsers: s-expressions, YAML, JSON. I am against JSON for source files since working with multiline text will likely be a pain (besides editing of files, patches and VCS diffs should be taken into account). Actually there is a requirement for external projects related to parser already. Expected results are provided as S-expressions. For simplicity I described a test suite with a couple of cases using s-expressions (see attachments). Names of macro an functions are arbitrary. Interface should be like (demo-parse-test-suite "demo-test-suite.el") Current implementation creates an ert test for each entry in such file, but it may be changed later. I just had example of macro for parametrized test from earlier experiments. On 15/12/2021 20:23, Ihor Radchenko wrote: > Max Nikulin writes: >> On 11/12/2021 21:39, Ihor Radchenko wrote: >>> >>> The attached patch is an attempt to create something like shared repo >>> for Org element parser tests. >> >> "[PATCH]" prefix in the subject might be a reason why you message >> received less attention than it should. At least Tom and Timothy have their parsers, so I expected some response from them. I am unsure whether posts about Organice were from developers or from users. Patch is something too internal to development of namely Org Mode as Emacs major mode. Once I asked to avoid proposal to integrate everything meaning mostly particular person who was extremely active that time, but it had effect on more people than I thought. You intention may have stronger result than it should as well. > I am not sure here. In a way, we already have such a format in > test-org-element.el: > > (ert-deftest test-org-element/bold-parser () > "Test `bold' parser." > ;; Standard test. > (should > (org-test-with-temp-text "*bold*" > (org-element-map (org-element-parse-buffer) 'bold #'identity nil t)))) It is a bit harder to parse than it is acceptable for external tools. However it may be converted to more convenient format using elisp facilities. > So, in addition to writing the Org parser, third-party dev > will also have to write a parser for the test format. I assume available parser for some wide spread format and some simple interpretation of the structures specific to these tests. > Most existing parser libraries can > handle individual files, but not individual pieces of text grouped into > bigger file using yet another standard convention. It should take a several line of code to put string input into a temporary file. Likely it is possible to implement interface of file stream for input string. Finally, it may stimulate developers to add a method that parses strings. I found the following test corpus impressive and unmanageable in the form of separate files: https://github.com/tgbugs/laundry/blob/master/laundry/test.rkt > But is there a > standard multi-test in one file format that can be used in multiple > programming languages? Some testing frameworks have facilities for parametrized tests with lists of arbitrary structures, others explicitly recommends to write a loop over such lists. > Or even allow inline tests via buffer-local after-save-hook: > > - text :: "*Bold* emphasis" > - keywords :: emphasis, heading > - description :: Despite line is started from a star, there is no > space after it, so it is not a heading. > > Upon save, the text:: field will be automatically converted to a > test-hash.org+test-hash.el files. Or to a single file with some top-level metadata and a list (maybe nested lists) of tests. Unsure that such files should be stored in git. >> Version of Org and test set should be included into metadata for the >> whole suite. > > If the test set is distributed together with Org, I see no reason to do > it. Otherwise, the test set should simply track bugfix and main branches > of Org and follow Org releases. Does it sound reasonable? I have no particular opinion if test sources should be maintained in the same repository with Org or Emacs. If test are described in Org files then test suites should be distributed using some other way. >> Since partial compliance is assumed, format of test results should be >> declared as well to be able to publish overview or comparison. > > I provided a short description of the format in the README (see the > original patch). Is it not enough? I mean result of *group* of tests to present comparison of parsers. Unsure if JUnit XML files are appropriate. >> Are properties like :begin and :end mandatory for reference results of >> parsing? They make structures more verbose and harder to read. Often it >> is enough to compare content and similar properties. > > I afraid that if we put contents of every headline/element in place of > :begin :end, the results will be even less readable. The results will > have to dump multiple cumulative instances of the original file. I had in mind that only leaf elements have contents. It may result in some ambiguity however. > Actually, there have been talks about including Org mode tests into > Emacs itself (https://firstname.lastname@example.org/) I have seen proposals to drop separate repository for Org and do everything inside the one for Emacs earlier. In my opinion, it would mean broken compatibility with earlier version of Emacs, so I do not like it. On the other hand I am not an active developer, so my arguments may be ignored. P.S. Output or Ert requires more tuning: ---- >8 ---- Selector: t Passed: 1 Failed: 1 (1 unexpected) Skipped: 0 Total: 2/2 Started at: 2021-12-19 14:13:54+0700 Finished. Finished at: 2021-12-19 14:13:54+0700 F. F demo-parse-test/demo-test-suite--bold-emphasis-at-beginning-of-line Bold text marker at beginning of line should not be confused with heading: no space after star (‘demo-parse-test/demo-test-suite’) (ert-test-failed ((should (funcall #<subr equal> '(org-data ... ...) (apply #'demo-test-parse-input '...))) :form (funcall #<subr equal> (org-data (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 ...) (paragraph (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 ...) (bold ... "Bold text"))) (org-data (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 ...) (section (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 ...) (paragraph ... ... " ")))) :value nil)) [-- Attachment #2: demo-test-suite.el --] [-- Type: text/x-emacs-lisp, Size: 1420 bytes --] ( :description "Just a demo of test suite for org-element parser" :cases ( (:id simple-heading :description "Heading and section with paragraph" :input " * this is a heading With some text below. " :result (org-data (:begin 1 :contents-begin 3 :contents-end 46 :end 46 :post-affiliated 1 :post-blank 0) (headline (:archivedp nil :begin 3 :commentedp nil :contents-begin 24 :contents-end 46 :end 46 :footnote-section-p nil :level 1 :post-affiliated 3 :post-blank 0 :pre-blank 1 :priority nil :raw-value "this is a heading" :tags nil :title ("this is a heading") :todo-keyword nil :todo-type nil) (section (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0) (paragraph (:begin 24 :contents-begin 24 :contents-end 46 :end 46 :post-affiliated 24 :post-blank 0) "With some text below.\n"))))) (:id bold-emphasis-at-beginning-of-line :description "Bold text marker at beginning of line should not be confused with heading: no space after star" :input "*Bold text* " :result (org-data (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 :post-blank 0) (paragraph (:begin 1 :contents-begin 1 :contents-end 13 :end 13 :post-affiliated 1 :post-blank 0) (bold (:begin 1 :contents-begin 2 :contents-end 11 :end 13) "Bold text")))) )) [-- Attachment #3: demo-test.el --] [-- Type: text/x-emacs-lisp, Size: 2666 bytes --] ;;;; Example of usage: ;;;; (demo-parse-test-suite "demo-test-suite.el") (defmacro nm-deftest-parametrized (prefix-sym func-predicate &rest doc-cases) "Define parametrized test For each SUFFIX, CASE-DOCSTRING, EXPECTATION, ARGS list call `ert-deftest' with SUITE--SUFFIX name, CASE-DOCSTRING, and `should' that checks whether EXPECTATION is consistent with result of FUNCTION applied to ARGS using PREDICATE or `equal'. \(fn SUITE (FUNCTION [PREDICATE]) \ [SUITE-DOCSTRING] \ (SUFFIX CASE-DOCSTRING EXPECTATION ARGS...)...)" (declare (debug (&define [&name "test@" symbolp] sexp [&optional strinp] def-body)) (indent 2) (doc-string 3)) (let* ((func (car func-predicate)) (predicate (or (cadr func-predicate) (symbol-function 'equal))) (prefix (symbol-name prefix-sym)) (maybe-doc (car doc-cases)) (cases (if (stringp maybe-doc) (cdr doc-cases) doc-cases)) (case-list (mapcar (lambda (case) (format "- `%s--%s'" prefix (symbol-name (car case)))) cases)) ;; Unfortunately `ert-describe-test' works only in ert mode ;; and links to particular subtests are inactive. (suite-doc (if (stringp maybe-doc) (cons maybe-doc case-list) case-list))) (append `(,#'progn ;; A function to assing doc string that is linked from each test. (defun ,prefix-sym () ,(mapconcat #'identity suite-doc "\n"))) (mapcar (lambda (case) ;; Have not managed to express "&rest" using `pcase-let'. (apply (lambda (id-sym case-docstring expectation &rest args) (let* ((id (symbol-name id-sym)) (name (intern (concat prefix "--" id))) (docstring (format "%s (`%s')" case-docstring prefix))) `(ert-deftest ,name () ,docstring (should (funcall ,predicate ,expectation (apply ,func (quote ,args))))))) case)) cases)))) (defun demo-test-parse-input (text) (with-temp-buffer (insert text) (org-mode) (test-org-element-parser-generate-syntax-sexp))) (defmacro demo-parse-test-suite (file-name) (let* ((prefix (file-name-base file-name)) (prefix-sym (intern (concat "demo-parse-test/" prefix))) (suite (with-temp-buffer (insert-file-contents file-name) (read (current-buffer)))) (case-list (plist-get suite :cases)) (suite-description (plist-get suite :description))) `(nm-deftest-parametrized ,prefix-sym (#'demo-test-parse-input) ,suite-description ,@(mapcar (lambda (case) (list (plist-get case :id) (or (plist-get case :description) "Warning: no description for this case") `(quote ,(plist-get case :result)) (plist-get case :input))) case-list)))) [-- Attachment #4: demo-test-suite.json --] [-- Type: application/json, Size: 2620 bytes --] [-- Attachment #5: demo-test-suite.yaml --] [-- Type: application/x-yaml, Size: 1675 bytes --]
prev parent reply other threads:[~2021-12-19 8:24 UTC|newest] Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-12-11 14:39 Ihor Radchenko 2021-12-14 16:16 ` Max Nikulin 2021-12-14 20:27 ` Tim Cross 2021-12-15 13:44 ` Ihor Radchenko 2021-12-15 13:23 ` Ihor Radchenko 2021-12-19 8:23 ` Max Nikulin [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://www.orgmode.org/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com' \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).