Re: [PATCH] make test: Make failure results more verbose

[Max Nikulin <manikulin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2078 bytes --]

On 21/01/2022 20:33, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
> 
>> I am attaching a tentative patch for Org that should make
>> EMACS_TEST_VERBOSE setting more transparent.
> 
> Thanks! Your patch looks cleaner.

In the meanwhile Lars fixed ERT in Emacs-29, so empty string is 
considered as false now, see https://debbugs.gnu.org/53313
Values like "no" should not be considered as false by Org makefiles, so 
I am attaching an updated version.

> In addition, I am attaching a tentative hack to make ERT pretty-print
> the failure reason as you wished. WDYT?

Sorry, I was not clear enough. Summary should remain single-line. There 
are pretty-printed failure reason in the main part of the log. I was 
complained concerning the following brick

>   signal(ert-test-failed (((should (equal "aB " (org-test-with-parsed-
>   ert-fail(((should (equal "aB " (org-test-with-parsed-data "* Headlin
>   (if (unwind-protect (setq value-5061 (apply fn-5059 args-5060)) (set
>   (let (form-description-5063) (if (unwind-protect (setq value-5061 (a
>   (let ((value-5061 'ert-form-evaluation-aborted-5062)) (let (form-des
>   (let* ((fn-5059 #'equal) (args-5060 (condition-case err (let ((signa
>   (closure (t) nil (let* ((fn-5059 #'equal) (args-5060 (condition-case
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name test-org-export/get-previous-element 
>   ert-run-or-rerun-test(#s(ert--stats :selector "test-org-export/get-p
>   ert-run-tests("test-org-export/get-previous-element" #f(compiled-fun
>   ert-run-tests-batch("test-org-export/get-previous-element")
>   ert-run-tests-batch-and-exit("test-org-export/get-previous-element")
>   (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
>   org-test-run-batch-tests("test-org-export/get-previous-element")
>   eval((org-test-run-batch-tests org-test-select-re) t)
>   command-line-1(("-L" "/home/ubuntu/ert" "--eval" "(setq vc-handled-b
>   command-line()
>   normal-top-level()

However it will be too long when pretty printed.

[-- Attachment #2: 0001-make-test-Make-failure-summary-more-verbose.patch --]
[-- Type: text/x-patch, Size: 1365 bytes --]

From a53095fa1c6679376708dcc1a05a446c3967d914 Mon Sep 17 00:00:00 2001
From: Max Nikulin <manikulin@gmail.com>
Date: Sat, 15 Jan 2022 22:54:30 +0700
Subject: [PATCH] make test: Make failure summary more verbose

* mk/default.mk: By default enable verbose failure summary for Emacs-28
or newer.

Set or unset EMACS_TEST_VERBOSE environment to control reporting of failure
reasons in summary since in Emacs-28 ERT switches to verbose mode even by
an empty string (fixed in Emacs-29).
---
 mk/default.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/mk/default.mk b/mk/default.mk
index c8a15bdd2..804089280 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -32,6 +32,20 @@ TMPDIR ?= /tmp
 testdir = $(TMPDIR)/tmp-orgtest
 
 # Configuration for testing
+# Verbose ERT summary by default for Emacs-28 and above.
+# To override:
+# - Add to local.mk
+#   EMACS_TEST_VERBOSE =
+# - Export EMACS_TEST_VERBOSE environment variable with empty value
+# - Run tests as
+#   EMACS_TEST_VERBOSE= make test [OTHER_ARGUMENTS...]
+#   or as
+#   make test EMACS_TEST_VERBOSE= [OTHER_ARGUMENTS...]
+EMACS_TEST_VERBOSE ?= yes
+ifeq (,$(EMACS_TEST_VERBOSE))
+# Emacs-28 considers empty value as true, fixed in Emacs-29
+unexport EMACS_TEST_VERBOSE
+endif
 # add options before standard load-path
 BTEST_PRE   =
 # add options after standard load path
-- 
2.25.1