From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oOBcGEuL0mFluwAAgWs5BA (envelope-from ) for ; Mon, 03 Jan 2022 06:36:11 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id YD0fEUuL0mFyAwAAG6o9tA (envelope-from ) for ; Mon, 03 Jan 2022 06:36:11 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 18F9B1EDD7 for ; Mon, 3 Jan 2022 06:36:10 +0100 (CET) Received: from localhost ([::1]:43538 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4G0u-0007NG-V2 for larch@yhetil.org; Mon, 03 Jan 2022 00:36:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46372) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4G0J-0007Mh-4u for emacs-orgmode@gnu.org; Mon, 03 Jan 2022 00:35:31 -0500 Received: from ciao.gmane.io ([116.202.254.214]:32964) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4G0H-0005b3-LG for emacs-orgmode@gnu.org; Mon, 03 Jan 2022 00:35:30 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1n4G0E-0006WZ-Gn for emacs-orgmode@gnu.org; Mon, 03 Jan 2022 06:35:26 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Max Nikulin Subject: Re: [PATCH] make test: Make failure results more verbose Date: Mon, 3 Jan 2022 12:35:16 +0700 Message-ID: References: <87ee5q5lic.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 In-Reply-To: <87ee5q5lic.fsf@localhost> Content-Language: en-US Received-SPF: pass client-ip=116.202.254.214; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 17 X-Spam_score: 1.7 X-Spam_bar: + X-Spam_report: (1.7 / 5.0 requ) DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FORGED_MUA_MOZILLA=2.309, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, NICE_REPLY_A=-3.024, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641188170; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=WkeAMSMPYSdzdYBmWf4ZU4nGnoLhdGyxevCQhYFOF/g=; b=HWjOt5OFIEFApTGUfpbJ8D4LM67wL4MOpvjRAPi/f26uOQFknLrNRP2eCWg28RH+OHhvFB yyQXKiEwEHww38IKGVRM01JOFGdzzZPB3KmM5etL4mVVlSdK/8ASTjivekcR5xx/B2NjdB GV2r3kE0vQTfyX6gmuI2qzAzcNFAiOfeYV+Q6TEdZFBMPQAn/BU2KFWDF2BaxBQhkhbSP4 7O5zm+1750xMbE8y+6gw6clyZTpKXEfee5MMoGdnbkU4Lk5pA+L/KT8m2NwV72W1JksDjq bgRLQ4iYZXGBVG6wyOZ/U6KPuhmVHkTY1n90mpoJJfu3wvsNAVmXcvfT8MHzig== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641188170; a=rsa-sha256; cv=none; b=GXMWtnSknLrSXi3HUmrSl4OSxvIRnET/oh1FX+7eXBmyA2s1SHbtzGAQNa5XqQWhBru9z2 7zDQIAFywr22d/5GnxpDhPFNiVffglEMF8IwiKaO2LV0VFx0ni5uJ4d4z8ct5/ZIZCTlAM nv0ma0Ih1OoCw3hL+EbNkLo+dLjDVHcVJA4ZlzWPdOrABbqOe3fm+SH5Dpyf6TVWHdaJ9+ 1XuV4q6Y/X5s0A66g1uOKWwCsZPwdso1Nv4EUPs61tWq7Qs68nYOwqH6Z8V5EgNxqmAcqM 767CDd6sujMCpVQK0adD0HjcAhOyGpa1TVBqHlO5Vf2wylbwACuRdH/jc9INrQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.49 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 18F9B1EDD7 X-Spam-Score: -2.49 X-Migadu-Scanner: scn1.migadu.com X-TUID: ffGoMcZx8Ymj On 02/01/2022 20:12, Ihor Radchenko wrote: > > In newer Emacs, ERT is capable of providing more info about FAILED > tests. Maybe we can enable this option by default in the Org test suite? I like you attempts to make tests better. Ihor, are there examples of new error reports in mail lists, blogs, etc? I am not motivated enough to try development version of emacs, but my impression that current error log looks like rectangles of garbage. In my opinion, code of test should be written having clear error reports in mind. > +BTEST_ERT_VERBOSE = yes I am unsure if this line or local.mk has priority. I am unsure the the following is better as well. BTEST_ERT_VERBOSE ?= yes Is there an easy way to limit number of failures before termination of tests in the case of verbose reporting? It should prevent test log from blowing too much. Usually there is no point in all details if all or even 1/4 of tests fails. > + TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST) A purist would say that it is not a directory, it is something like ...FLAGS or ...ARGS. I know, it was abused before your patch. Shouldn't it be mentioned in testing/README?