emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: Max Nikulin <manikulin@gmail.com>
To: emacs-orgmode@gnu.org
Subject: Re: [PATCH] make test: Make failure results more verbose
Date: Sat, 15 Jan 2022 22:58:41 +0700	[thread overview]
Message-ID: <sruqvj$s3k$1@ciao.gmane.io> (raw)
In-Reply-To: <srugt5$h10$1@ciao.gmane.io>

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

On 02/01/2022 20:12, Ihor Radchenko wrote:
> 
> In newer Emacs, ERT is capable of providing more info about FAILED
> tests. Maybe we can enable this option by default in the Org test suite?
Thinking more, I have realized that something is wrong.

Behavior of tests in Org should be controlled by EMACS_TEST_VERBOSE 
*environment* variable. Org makefiles may have it on by default, but it 
should not override explicitly chosen value. Fortunately Emacs 
repository contains no makefiles from Org, so it will not cause conflict 
with Emacs builds.

The problem is that EMACS_TEST_VERBOSE interface is broken. In the 
following case I expect that test summary should not be verbose:

ert-sample.el
---- >8 ----
(require 'ert)
(ert-deftest ert-sample ()
   (should (equal 5 (* 2 2))))
(ert-run-tests-batch-and-exit)
---- 8< ----

(I am tried it with Emacs-27, so I put a copy of ert.el from git HEAD to 
~/ert)

EMACS_TEST_VERBOSE= emacs --batch -Q -L ~/ert -l ert-sample.el

1 unexpected results:
    FAILED  ert-sample  ((should (equal 5 (* 2 2))) :form (equal 5 4) 
:value nil :explanation (different-atoms (5 "#x5" "?") (4 "#x4" "?")))

That is why I am going to file a bug against ert.

In a minimal variant empty string (`getenv' return value) should not be 
considered as t.

For better user experience, I expect that the following case-insensitive 
strings should be considered as false: "0", "false", "no", "n", "off", 
"f" (borrowed from .ini), "none", "nil". Other values should be 
considered as true or anything besides "1", "true", "yes", "y", "on", 
"t" should be considered as invalid value.

I am attaching a tentative patch for Org that should make 
EMACS_TEST_VERBOSE setting more transparent.

[-- Attachment #2: 0001-make-test-Make-failure-summary-more-verbose.patch --]
[-- Type: text/x-patch, Size: 1654 bytes --]

From 5f8d93a9895b67ce89bac3b2d95ca723fe754ab4 Mon Sep 17 00:00:00 2001
From: Max Nikulin <manikulin@gmail.com>
Date: Sat, 15 Jan 2022 22:54:30 +0700
Subject: [PATCH] make test: Make failure summary more verbose

* mk/default.mk: By default enable verbose failure summary for Emacs-28
or newer.

Set or unset EMACS_TEST_VERBOSE environment to control reporting of failure
reasons in summary since ERT switches to verbose mode even by an empty
string. As an extension, allow values as "0", "off", "no" to disable
verbosity.
---
 mk/default.mk | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/mk/default.mk b/mk/default.mk
index c8a15bdd2..11ddcc2e1 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -32,6 +32,24 @@ TMPDIR ?= /tmp
 testdir = $(TMPDIR)/tmp-orgtest
 
 # Configuration for testing
+# Verbose ERT summary by default for Emacs-28 and above.
+# To override:
+# - Add to local.mk
+#   EMACS_TEST_VERBOSE =
+# - Export EMACS_TEST_VERBOSE environment variable with empty value
+# - Run tests as
+#   EMACS_TEST_VERBOSE= make test [OTHER_ARGUMENTS...]
+#   or as
+#   make test EMACS_TEST_VERBOSE= [OTHER_ARGUMENTS...]
+# For convenience some other values are recognized as false by Org makefiles
+# ("0", "no", "off", "false") despite any set value (including empty one)
+# is considered as true by ERT.
+EMACS_TEST_VERBOSE ?= yes
+ifneq (,$(filter-out 0 n N no No NO f F false False FALSE off Off OFF none None NONE nil NIL,$(EMACS_TEST_VERBOSE)))
+export EMACS_TEST_VERBOSE
+else
+unexport EMACS_TEST_VERBOSE
+endif
 # add options before standard load-path
 BTEST_PRE   =
 # add options after standard load path
-- 
2.25.1


  reply	other threads:[~2022-01-15 16:00 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-01-02 13:12 Ihor Radchenko
2022-01-03  5:35 ` Max Nikulin
2022-01-07 15:04   ` Ihor Radchenko
2022-01-11 16:46     ` Max Nikulin
2022-01-15 12:52       ` Max Nikulin
2022-01-15 13:06         ` Max Nikulin
2022-01-15 15:58           ` Max Nikulin [this message]
2022-01-21 13:33             ` Ihor Radchenko
2022-01-21 15:01               ` Max Nikulin
2022-01-23 13:31                 ` Ihor Radchenko
2022-01-21 13:31         ` Ihor Radchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='sruqvj$s3k$1@ciao.gmane.io' \
    --to=manikulin@gmail.com \
    --cc=emacs-orgmode@gnu.org \
    --subject='Re: [PATCH] make test: Make failure results more verbose' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).