Re: [PATCH] make test: Make failure results more verbose

[Max Nikulin <manikulin@gmail.com>]

On 07/01/2022 22:04, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Ihor, are there examples of new error reports in mail lists, blogs, etc?
>> I am not motivated enough to try development version of emacs, but my
>> impression that current error log looks like rectangles of garbage.
> 
> Yeah, I should have attached examples to the original message.
> Also, we can adjust the ERT output using
> ert-batch-backtrace-right-margin, ert-batch-print-level,
> ert-batch-print-level, and ert-batch-backtrace-line-length
> 
> Here are the examples:

Thank you. After sending that message I realized that I did not try to 
load just ert.el from git to emacs version that I have installed.

>> In my opinion, code of test should be written having clear error reports
>> in mind.
> 
> I guess. Though I feel that ERT is better when used interactively.

Your patch improves batch error reporting. Good error message should 
allow to figure out the origin of the problem without interactive debugging.

> Do you have good ideas what could be changed?

Side note. In f0c474e659b81da0d2ab75e7ec109355965f7a1c I have noticed
"* H1\nP1\n<point*H2\n"
that likely should be
"* H1\nP1\n<point>* H2\n"

I do not remember where I have seen tests with rather ugly failure 
reports. Example of what I consider as minor improvement (sorry for the 
wrapped text):

(ert-deftest tst-inline-tags-orig ()
   (should
    (equal '("foo" "baz")
           (progn
             (require 'org-inlinetask)
             (org-test-with-temp-text (concat (make-string 
org-inlinetask-min-level ?*) " Test :foo:bar:")
               (org-get-tags (org-element-at-point)))))))

(defun org-test-no-properties (arg)
   (cond
    ((stringp arg) (org-no-properties arg))
    ((listp arg) (mapcar #'org-test-no-properties arg))
    (t arg)))

(ert-deftest tst-inline-tags-new ()
   (let ((inline-stars (make-string org-inlinetask-min-level ?*)))
     (require 'org-inlinetask)
     (should
      (equal '("foo" "baz")
	    (org-test-with-temp-text (concat inline-stars " Test :foo:bar:")
               (org-test-no-properties (org-get-tags 
(org-element-at-point))))))))


F tst-inline-tags-new
     (ert-test-failed
      ((should
        (equal
	'("foo" "baz")
	(org-test-with-temp-text
	    (concat inline-stars " Test :foo:bar:")
	  (org-test-no-properties ...))))
       :form
       (equal
        ("foo" "baz")
        ("foo" "bar"))
       :value nil :explanation
       (list-elt 1
		(array-elt 2
			   (different-atoms
			    (122 "#x7a" "?z")
			    (114 "#x72" "?r"))))))

F tst-inline-tags-orig
     (ert-test-failed
      ((should
        (equal
	'("foo" "baz")
	(progn
	  (require ...)
	  (org-test-with-temp-text ... ...))))
       :form
       (equal
        ("foo" "baz")
        (#("foo" 0 3
	  (keymap ... mouse-face highlight font-lock-fontified t face ...))
	 #("bar" 0 3
	   (keymap ... mouse-face highlight font-lock-fontified t face ...))))
       :value nil :explanation
       (list-elt 1
		(array-elt 2
			   (different-atoms
			    (122 "#x7a" "?z")
			    (114 "#x72" "?r"))))))

>>> +BTEST_ERT_VERBOSE = yes
>>
>> I am unsure if this line or local.mk has priority. I am unsure the the
>> following is better as well.
>> BTEST_ERT_VERBOSE ?= yes
> 
> I am not very familiar with Makefile conventions. Just followed the
> existing settings in the same file. All other BTEST_ERT_* settings just
> use "=".

OK, maybe I will try it later. I just do not remember in which order 
assignments are evaluated. Likely everything works as expected and 
features like delayed assignment (simple "=" vs. ":=" ) do not cause any 
problem.

>> Is there an easy way to limit number of failures before termination of
>> tests in the case of verbose reporting? It should prevent test log from
>> blowing too much. Usually there is no point in all details if all or
>> even 1/4 of tests fails.
> 
> The best approach I know is using BTEST_RE

BTEST_RE is useful when you know in advance which tests you are 
interested in. I had in mind continuous integration and notifications. 
Single full error (or several ones) included into a message is 
convenient and may provide enough information for quick fix. If there 
are 500 errors there is no point to send all of them as notification. 
Either test is aborted after a couple of dozens of failures or total 
number of failures and full text of several errors are enough to avoid 
excessively long notification. Anyway nobody is going to read all 
hundreds of failures.

>>> +	TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST)
>>
>> A purist would say that it is not a directory, it is something like
>> ...FLAGS or ...ARGS. I know, it was abused before your patch.
> 
> I do not follow you here.
> VARIABLE1=1 VARIABLE2=2 /path/to/script
> is the usual way to set variables in script environment in bash.

My bad. I did not noticed that it is a part of script of a rule. I 
believed that the whole line is assignment to TMPDIR variable. I am 
unsure if the following approach to avoid excessive ifeq's without 
modification of the rule is better:

ifeq ($(BTEST_ERT_VERBOSE),yes)
EMACS_TEST_VERBOSE=yes
export EMACS_TEST_VERBOSE
endif

>> Shouldn't it be mentioned in testing/README?
> 
> Only BTEST_RE is documented there. BTEST_PRE, BTEST_POST,
> BTEST_OB_LANGUAGES, and BTEST_EXTRA are not documented.
> I believe that the odds for the user to change BTEST_ERT_VERBOSE are
> rather low and it is not worth mentioning it explicitly in README.
> 
> Or, alternatively, we can document all the settings in README.
> WDYT?

I agree that default verbose log is preferred in most cases. The only 
exception when it becomes rather useless and just consumes disk space is 
something basic is broken (see my comment concerning limit on failures 
above). Let's assume that you convinced me and only rare developers will 
customize the value.