From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 6KFiDqbDuGGCfwEAgWs5BA (envelope-from ) for ; Tue, 14 Dec 2021 17:17:42 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id yPIhCqbDuGEOTwAAbx9fmQ (envelope-from ) for ; Tue, 14 Dec 2021 16:17:42 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E74BFDDB4 for ; Tue, 14 Dec 2021 17:17:41 +0100 (CET) Received: from localhost ([::1]:38660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxAUl-0006RG-2z for larch@yhetil.org; Tue, 14 Dec 2021 11:17:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxAU2-0006Qq-Hc for emacs-orgmode@gnu.org; Tue, 14 Dec 2021 11:16:55 -0500 Received: from ciao.gmane.io ([116.202.254.214]:50334) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxAU1-0003Vh-63 for emacs-orgmode@gnu.org; Tue, 14 Dec 2021 11:16:54 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1mxATy-0006EO-CI for emacs-orgmode@gnu.org; Tue, 14 Dec 2021 17:16:50 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Max Nikulin Subject: Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser Date: Tue, 14 Dec 2021 23:16:38 +0700 Message-ID: References: <87fsqzi4tw.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 In-Reply-To: <87fsqzi4tw.fsf@localhost> Content-Language: en-US Received-SPF: pass client-ip=116.202.254.214; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 18 X-Spam_score: 1.8 X-Spam_bar: + X-Spam_report: (1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FORGED_MUA_MOZILLA=2.309, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, NICE_REPLY_A=-0.962, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639498661; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=62WFK141QpP+BTI2ahJGEEe1cLVm08eYE3HLHZqj4jU=; b=CXz6uKUAbbs+jTjRPr9bm8N7tUGxQzvBBWOwAQF2z/YD8aqgW0qGiPdM56AOeL/Dyutzub bNLQ4ens6F908fsVMkD0s0aitrqan6hWY1Re9Og+kkcPJjjnhpqU+1rCfjUtsXCli5mMCx 48zSsYQ/whIDtwfvsYSRQwon/uUIIoaFIlaS2LUF0sDFVbWjS/qz+f8Xlkn2Ex/nW5M66c 4HI/E/DN+yfmzqNaTEbWsmF9JcHVEQHmhUezl3mHkM5GlpzPoljanEsz4Vlb6Oh6PFJRZO 240T/h6gWKTZZUFWliqNJ1ulbgmZkvRixdMygxhZ9M5YGzzMRnDOkVLTMpYd9Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639498661; a=rsa-sha256; cv=none; b=BO7XPRBG6pYQ2P7zRqhyXFU7/HmiM3tyjXRiw1zdgI3C2bH3kDxC6dA66VN/0HJXu5pCPH spWCWUKt8nGUlc+OAj5vrKYY3w53DQGGQtuVGZFeFLIoUhviFQUQTh3HULrTjx5WG8HF22 jiyL1pd3Fee2MnT1XjlqYay5lp4uBaXYDtosQ+FKcGBwPbdYHDMern7wCYrdBrRa4l1nns inO/55rHHwREvKuVgCFG5hM9Bc0OTmnpXfYxfIU8xGumpNSJ3C7N64NtEhouhCpHLmi/dk 0sZyyW/Fj3RYDwXUXFBap+RgZ41mZPDplioVry/JnxiWl6001lJ2AJ2KBz5Svg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.37 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E74BFDDB4 X-Spam-Score: -2.37 X-Migadu-Scanner: scn1.migadu.com X-TUID: YAK9F4OoOdst On 11/12/2021 21:39, Ihor Radchenko wrote: > > The attached patch is an attempt to create something like shared repo > for Org element parser tests. "[PATCH]" prefix in the subject might be a reason why you message received less attention than it should. > The test set is essentially a series of .org files alongside .el files > containing normalised output of `org-element-parse-buffer'. (see the > patch) I think, the set should contain hundreds of tests to be helpful, thus 2 files per test will likely be inconvenient since most of samples will be short. I suggest to group test input and results into large files. Such tests should be augmented by some metadata: keywords (labels, tags). E.g. besides heading - sample: "* Simple Heading" keywords: heading it should be possible to filter related cases with similar markup - sample: "*Bold* emphasis" keywords: emphasis, heading description: Despite line is started from a star, there is no space after it, so it is not a heading. - sample: " * Unordered list item" keywords: list, heading description: Due to leading space it is a list item, not a heading. - sample: "*" keywords: text, heading description: Not a heading since there is no space after the star. I omitted test IDs above. Version of Org and test set should be included into metadata for the whole suite. Since partial compliance is assumed, format of test results should be declared as well to be able to publish overview or comparison. Are properties like :begin and :end mandatory for reference results of parsing? They make structures more verbose and harder to read. Often it is enough to compare content and similar properties. > Any comments or suggestions? > I am particularly looking for thoughts about licensing and possible > distribution of the test set in separate repository. Since these tests will unlikely become a part of some software, I do not think that GPL may be an obstacle for any project. Requirement of signed consent will likely prevent contributing of new cases from some developers.