From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oGmWI8L5uWEWcwAAgWs5BA (envelope-from ) for ; Wed, 15 Dec 2021 15:20:50 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6LciH8L5uWFtSgAAbx9fmQ (envelope-from ) for ; Wed, 15 Dec 2021 14:20:50 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BFF7E767F for ; Wed, 15 Dec 2021 15:20:49 +0100 (CET) Received: from localhost ([::1]:45590 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxV9D-00080D-Nu for larch@yhetil.org; Wed, 15 Dec 2021 09:20:47 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60986) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxUES-0007gz-8P for emacs-orgmode@gnu.org; Wed, 15 Dec 2021 08:22:09 -0500 Received: from [2a00:1450:4864:20::12a] (port=44807 helo=mail-lf1-x12a.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxUEQ-0005qf-DM for emacs-orgmode@gnu.org; Wed, 15 Dec 2021 08:22:07 -0500 Received: by mail-lf1-x12a.google.com with SMTP id z7so43066828lfi.11 for ; Wed, 15 Dec 2021 05:22:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=tYq1lQe/6tDNNPdzGZixqe34CC2uMkKTrpIi8DUrPOY=; b=MO3XvFAbJzLAZ2XZ7eoJzkUJZPn7X5z8dj9V5MZXpZlmXny4pfw6YitYN3JxzfO0Sr a4w7agfeUfho4pSb2OdQ2/CuPhBV3n6xwDSVhEAdAx8lXhiH3J/dRzDeevC0TGaBZxjx WTz+7/nzSEOlyQQhoy1AUmX1KEtQS0WYs0FevVDqlSgAyPZXzI+05jPp8M7t75gVuj4x R3YkYvC1nlvCxLj9HNzrKRJdO/JEzEQhnmKJ2LAukmoWzEZZyZ7MxILQMxhAPMi3Jc+2 dsAUyp7VuS07hDZXfp9Ultbjo1rD89wTbip/2uxEt6SE9b1Yt974TvxoBFjX4JG+gNTR mOsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=tYq1lQe/6tDNNPdzGZixqe34CC2uMkKTrpIi8DUrPOY=; b=AvRWQuZfLhjFidY3EPQWoFZMRttDMXg5FnGH+STWejvP3KTU4Q+VNGqLql5biiRgcy /6YKAVHZHr8rOgtxG237zPbycm/lAC3wRi87HsFPm9usJU6qzsijOnKPQ7ZwQmI9dDsj 8MfOR7pPwzAU+bi+1rTzge9kaY3PY5EQn9Xt1ZTNBWdOgjoinemzTjz0FYVKSC9i2jml w/6Jbjg5odJ7IRoePxNLmylV49ZkpUXisZF7CotMqpjzW3vuFHVTYCiLT/57INcnzg/o Shzz0/BuHQr/LldY/WpkkiBwNM6tCiekKoHohz/4u533jfSo8fR0Vv4WjmH7w5Nraato m9Yw== X-Gm-Message-State: AOAM5300fuytDw5UN7u9+2ABb4ZowKoe76r8K6ikuF4OHUUZkESOcP3l ksrE3rcP+r+Oyurme2QgsAc= X-Google-Smtp-Source: ABdhPJxkjrm+7ixirmKAuop3PSzQBkpm0bWTWwoMc9Z1WRQ06lYVpPVdgK7wkMs3SdylgrhTkA9k+A== X-Received: by 2002:ac2:4d19:: with SMTP id r25mr9629996lfi.82.1639574524644; Wed, 15 Dec 2021 05:22:04 -0800 (PST) Received: from localhost ([141.105.67.194]) by smtp.gmail.com with ESMTPSA id z23sm444499ljn.23.2021.12.15.05.22.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 05:22:04 -0800 (PST) From: Ihor Radchenko To: Max Nikulin Subject: Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser In-Reply-To: References: <87fsqzi4tw.fsf@localhost> Date: Wed, 15 Dec 2021 21:23:26 +0800 Message-ID: <871r2et31t.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::12a (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::12a; envelope-from=yantar92@gmail.com; helo=mail-lf1-x12a.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode@gnu.org Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639578049; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=tYq1lQe/6tDNNPdzGZixqe34CC2uMkKTrpIi8DUrPOY=; b=Im0DWShCmsrqqa2AaWteVV99glV6reXKBJWwNSdjVDchumsqDyJ4rQCkVdZv1/HxyMKuA/ A6kJLeULehh5ha5IZV5EJtvU7rwPuHiqbusIGTfWsaeWuh5/2xBfm04/8/Swt+Ennbzzo/ mmWcJfs9R4DFmY8Cmk6DyHlvtHR46e2/tAfnjedX2jyLDpUze3I43cVV11/BsGAdydc+ng ep/ZbX16Pdb1MZ5F4DuNAehCoJKCZxHHiRtNTVunUIK0G9TSs7FjTMjYzcBfpFIwjy6rnY 1omLd8gOLIdv3bYcL0UZCbjYWQySKfV45xNRHWjeDNRg866cogsCJjxlhEPbwQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639578049; a=rsa-sha256; cv=none; b=H6HK9XGg14oUGbOqV3ZVvnmwNlviPrbY2OilpPfwTW2qRt6Z21k7YEQOqg/sv58ii2dCu2 ulGYgiN33Y0TRw0FGn2+L0T3AxvbGhOdI21RQkjfN63bUmnyvWgweVChxDP1t5/Cj/Mcwb j8Core4uPvIN8cE1fMjqDPiiLXwcCKC4rRnZO8hw4dOMiJ9urW95DqTBbByyB7Et2VjY7Z 8kgjSSs0+Thhf7kZcgnUsrpGPNkSHQiotCx8SU03vT6jC+BK7P87vtrKjJYbazGADUHUL8 FM4QdPUhcq9tVfB+SbIYfP+v7LGzDORpYIx7BXu9IBOLBdwPMfLf9cKImn2tbQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MO3XvFAb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.78 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MO3XvFAb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: BFF7E767F X-Spam-Score: -5.78 X-Migadu-Scanner: scn0.migadu.com X-TUID: 0EzLjJZhYmnG Max Nikulin writes: > On 11/12/2021 21:39, Ihor Radchenko wrote: >> >> The attached patch is an attempt to create something like shared repo >> for Org element parser tests. > > "[PATCH]" prefix in the subject might be a reason why you message > received less attention than it should. Well. I really wanted to set a technical tone of the discussion in contrast to my previous email (proposing this exact idea, among others) that generated little technical feedback and a lot of non-technical. > I think, the set should contain hundreds of tests to be helpful, thus 2 > files per test will likely be inconvenient since most of samples will be > short. I suggest to group test input and results into large files. Such > tests should be augmented by some metadata: keywords (labels, tags). > E.g. besides heading > > - sample: "* Simple Heading" > keywords: heading I am not sure here. In a way, we already have such a format in test-org-element.el: (ert-deftest test-org-element/bold-parser () "Test `bold' parser." ;; Standard test. (should (org-test-with-temp-text "*bold*" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t)))) The problem with grouping short tests into a single file is that we put an extra requirement on the testing code. The code will have to parse the test files with multiple tests, extract those tests, and then run the parser. So, in addition to writing the Org parser, third-party dev will also have to write a parser for the test format. I find individual files easier to get started with. Most existing parser libraries can handle individual files, but not individual pieces of text grouped into bigger file using yet another standard convention. The usual solution to the above problem is fixed test file format that can be processed by testing module of a given language. But is there a standard multi-test in one file format that can be used in multiple programming languages? Having said that, I like your idea about adding metadata to the tests. Probably, we can simply do it in the readme like the following: * Test group heading - file :: file:./test-files/test1.org - keywords :: k1, k2 - description :: description text Or even allow inline tests via buffer-local after-save-hook: - text :: "*Bold* emphasis" - keywords :: emphasis, heading - description :: Despite line is started from a star, there is no space after it, so it is not a heading. Upon save, the text:: field will be automatically converted to a test-hash.org+test-hash.el files. > Version of Org and test set should be included into metadata for the > whole suite. If the test set is distributed together with Org, I see no reason to do it. Otherwise, the test set should simply track bugfix and main branches of Org and follow Org releases. Does it sound reasonable? > Since partial compliance is assumed, format of test results should be > declared as well to be able to publish overview or comparison. I provided a short description of the format in the README (see the original patch). Is it not enough? > Are properties like :begin and :end mandatory for reference results of > parsing? They make structures more verbose and harder to read. Often it > is enough to compare content and similar properties. I afraid that if we put contents of every headline/element in place of :begin :end, the results will be even less readable. The results will have to dump multiple cumulative instances of the original file. Or do you have an alternative suggestion about the format of the reference parser output? >> Any comments or suggestions? >> I am particularly looking for thoughts about licensing and possible >> distribution of the test set in separate repository. > > Since these tests will unlikely become a part of some software, I do not > think that GPL may be an obstacle for any project. Requirement of signed > consent will likely prevent contributing of new cases from some developers. Actually, there have been talks about including Org mode tests into Emacs itself (https://yhetil.org/emacs-devel/87o8629h8g.fsf_-_@gmx.de/) Not all our tests are under GPL, but we may want to change this situation. Also, if we decide to not distribute this test set under GPL, what about the usual fear with someone contributing and its persons' employer coming on us with copyright claims? I do not have enough knowledge to judge this. Maybe Bastien or others know better. Or we may try to contact GNU legal team. Best, Ihor