From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id aGC9M6jv4mEpaQEAgWs5BA (envelope-from ) for ; Sat, 15 Jan 2022 17:00:40 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +AtxMKjv4mEB2AAAauVa8A (envelope-from ) for ; Sat, 15 Jan 2022 17:00:40 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 522512C9A5 for ; Sat, 15 Jan 2022 17:00:40 +0100 (CET) Received: from localhost ([::1]:59946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n8lTq-0006XO-FK for larch@yhetil.org; Sat, 15 Jan 2022 11:00:38 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57866) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n8lSE-00069o-FM for emacs-orgmode@gnu.org; Sat, 15 Jan 2022 10:58:58 -0500 Received: from ciao.gmane.io ([116.202.254.214]:33274) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n8lSC-0002Ss-S5 for emacs-orgmode@gnu.org; Sat, 15 Jan 2022 10:58:58 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1n8lS6-0007dl-To for emacs-orgmode@gnu.org; Sat, 15 Jan 2022 16:58:50 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Max Nikulin Subject: Re: [PATCH] make test: Make failure results more verbose Date: Sat, 15 Jan 2022 22:58:41 +0700 Message-ID: References: <87ee5q5lic.fsf@localhost> <87lezrr3i7.fsf@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------BAF5A94D3746876AD0EF64FA" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 In-Reply-To: Content-Language: en-US Received-SPF: pass client-ip=116.202.254.214; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 28 X-Spam_score: 2.8 X-Spam_bar: ++ X-Spam_report: (2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FORGED_MUA_MOZILLA=2.309, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, NICE_REPLY_A=-0.001, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1642262440; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=5guu1MB6l+DOMuBA7NIVrjmO4kun69XXYC0mCJ0ssBc=; b=s+itJWD73iX9ihKIF9VnIA7cYHwjBdeIPlkLyqqfoU4e3Qg7gygSBzu+1pvZqFKhSmDqtV 30M0LFt1oWtxIsUVeWmmQupEtdZOlV7SXaeM72uRuRyhyAJMdwKne4NRT0/X3FuK39AVlO LAulN/CYJGcwh5J/8aPKyNIr7YZniQSxT62jMIk7JeFqQ4EsB/HcS8CQzVruGL6XIiAsPy RY3ciz60e3GhGsJz5C96kqGWmxG8Npjb9D76+wdPRz/TZBMPAzATcfuMKEhdBZVYLrymtD beRwEA1HYwsgyXgNMJGlDYEVSBx5OCFrXsntWWeumMmMRHr8nTPNCJwTDUAAkg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1642262440; a=rsa-sha256; cv=none; b=XGBwWbiUJ9bmBuQsJa9S3WMjzBZzpulHVEtkWaz6sFtfk22NsBfO8s88jI/AibmgRSG871 QtSW/BHHg7iuD/d7UzBJnK31Y+St+iRj2o47BTMhZB70ToZ3QwKTWH3QxWHuX0+NExi50C 9XsIo28JP95n9kEA1aJ9h7SwGzZeMpe8TcYYBDzdCvkCDNs5FactHhUZdSR84zrfFkwvkH FfTXakwEL04XjF7lvGn/Izzx28UKIqjn93scaQHd7UMmV4Y14tTgQxKrm9dJzLtlrwk8GL RULfCBd51A4DwQu0dgNa/U4l7q+3j2JsEuSxsKtROLKDw1lY7uQuNK6agtTpNQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.53 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 522512C9A5 X-Spam-Score: -2.53 X-Migadu-Scanner: scn1.migadu.com X-TUID: /YBe3ZmRs/BL This is a multi-part message in MIME format. --------------BAF5A94D3746876AD0EF64FA Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 02/01/2022 20:12, Ihor Radchenko wrote: > > In newer Emacs, ERT is capable of providing more info about FAILED > tests. Maybe we can enable this option by default in the Org test suite? Thinking more, I have realized that something is wrong. Behavior of tests in Org should be controlled by EMACS_TEST_VERBOSE *environment* variable. Org makefiles may have it on by default, but it should not override explicitly chosen value. Fortunately Emacs repository contains no makefiles from Org, so it will not cause conflict with Emacs builds. The problem is that EMACS_TEST_VERBOSE interface is broken. In the following case I expect that test summary should not be verbose: ert-sample.el ---- >8 ---- (require 'ert) (ert-deftest ert-sample () (should (equal 5 (* 2 2)))) (ert-run-tests-batch-and-exit) ---- 8< ---- (I am tried it with Emacs-27, so I put a copy of ert.el from git HEAD to ~/ert) EMACS_TEST_VERBOSE= emacs --batch -Q -L ~/ert -l ert-sample.el 1 unexpected results: FAILED ert-sample ((should (equal 5 (* 2 2))) :form (equal 5 4) :value nil :explanation (different-atoms (5 "#x5" "?") (4 "#x4" "?"))) That is why I am going to file a bug against ert. In a minimal variant empty string (`getenv' return value) should not be considered as t. For better user experience, I expect that the following case-insensitive strings should be considered as false: "0", "false", "no", "n", "off", "f" (borrowed from .ini), "none", "nil". Other values should be considered as true or anything besides "1", "true", "yes", "y", "on", "t" should be considered as invalid value. I am attaching a tentative patch for Org that should make EMACS_TEST_VERBOSE setting more transparent. --------------BAF5A94D3746876AD0EF64FA Content-Type: text/x-patch; charset=UTF-8; name="0001-make-test-Make-failure-summary-more-verbose.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-make-test-Make-failure-summary-more-verbose.patch" >From 5f8d93a9895b67ce89bac3b2d95ca723fe754ab4 Mon Sep 17 00:00:00 2001 From: Max Nikulin Date: Sat, 15 Jan 2022 22:54:30 +0700 Subject: [PATCH] make test: Make failure summary more verbose * mk/default.mk: By default enable verbose failure summary for Emacs-28 or newer. Set or unset EMACS_TEST_VERBOSE environment to control reporting of failure reasons in summary since ERT switches to verbose mode even by an empty string. As an extension, allow values as "0", "off", "no" to disable verbosity. --- mk/default.mk | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mk/default.mk b/mk/default.mk index c8a15bdd2..11ddcc2e1 100644 --- a/mk/default.mk +++ b/mk/default.mk @@ -32,6 +32,24 @@ TMPDIR ?= /tmp testdir = $(TMPDIR)/tmp-orgtest # Configuration for testing +# Verbose ERT summary by default for Emacs-28 and above. +# To override: +# - Add to local.mk +# EMACS_TEST_VERBOSE = +# - Export EMACS_TEST_VERBOSE environment variable with empty value +# - Run tests as +# EMACS_TEST_VERBOSE= make test [OTHER_ARGUMENTS...] +# or as +# make test EMACS_TEST_VERBOSE= [OTHER_ARGUMENTS...] +# For convenience some other values are recognized as false by Org makefiles +# ("0", "no", "off", "false") despite any set value (including empty one) +# is considered as true by ERT. +EMACS_TEST_VERBOSE ?= yes +ifneq (,$(filter-out 0 n N no No NO f F false False FALSE off Off OFF none None NONE nil NIL,$(EMACS_TEST_VERBOSE))) +export EMACS_TEST_VERBOSE +else +unexport EMACS_TEST_VERBOSE +endif # add options before standard load-path BTEST_PRE = # add options after standard load path -- 2.25.1 --------------BAF5A94D3746876AD0EF64FA--