From: Max Nikulin <email@example.com> To: firstname.lastname@example.org Subject: Re: [PATCH] make test: Make failure results more verbose Date: Tue, 11 Jan 2022 23:46:08 +0700 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <87lezrr3i7.fsf@localhost> On 07/01/2022 22:04, Ihor Radchenko wrote: > Max Nikulin writes: > >> Ihor, are there examples of new error reports in mail lists, blogs, etc? >> I am not motivated enough to try development version of emacs, but my >> impression that current error log looks like rectangles of garbage. > > Yeah, I should have attached examples to the original message. > Also, we can adjust the ERT output using > ert-batch-backtrace-right-margin, ert-batch-print-level, > ert-batch-print-level, and ert-batch-backtrace-line-length > > Here are the examples: Thank you. After sending that message I realized that I did not try to load just ert.el from git to emacs version that I have installed. >> In my opinion, code of test should be written having clear error reports >> in mind. > > I guess. Though I feel that ERT is better when used interactively. Your patch improves batch error reporting. Good error message should allow to figure out the origin of the problem without interactive debugging. > Do you have good ideas what could be changed? Side note. In f0c474e659b81da0d2ab75e7ec109355965f7a1c I have noticed "* H1\nP1\n<point*H2\n" that likely should be "* H1\nP1\n<point>* H2\n" I do not remember where I have seen tests with rather ugly failure reports. Example of what I consider as minor improvement (sorry for the wrapped text): (ert-deftest tst-inline-tags-orig () (should (equal '("foo" "baz") (progn (require 'org-inlinetask) (org-test-with-temp-text (concat (make-string org-inlinetask-min-level ?*) " Test :foo:bar:") (org-get-tags (org-element-at-point))))))) (defun org-test-no-properties (arg) (cond ((stringp arg) (org-no-properties arg)) ((listp arg) (mapcar #'org-test-no-properties arg)) (t arg))) (ert-deftest tst-inline-tags-new () (let ((inline-stars (make-string org-inlinetask-min-level ?*))) (require 'org-inlinetask) (should (equal '("foo" "baz") (org-test-with-temp-text (concat inline-stars " Test :foo:bar:") (org-test-no-properties (org-get-tags (org-element-at-point)))))))) F tst-inline-tags-new (ert-test-failed ((should (equal '("foo" "baz") (org-test-with-temp-text (concat inline-stars " Test :foo:bar:") (org-test-no-properties ...)))) :form (equal ("foo" "baz") ("foo" "bar")) :value nil :explanation (list-elt 1 (array-elt 2 (different-atoms (122 "#x7a" "?z") (114 "#x72" "?r")))))) F tst-inline-tags-orig (ert-test-failed ((should (equal '("foo" "baz") (progn (require ...) (org-test-with-temp-text ... ...)))) :form (equal ("foo" "baz") (#("foo" 0 3 (keymap ... mouse-face highlight font-lock-fontified t face ...)) #("bar" 0 3 (keymap ... mouse-face highlight font-lock-fontified t face ...)))) :value nil :explanation (list-elt 1 (array-elt 2 (different-atoms (122 "#x7a" "?z") (114 "#x72" "?r")))))) >>> +BTEST_ERT_VERBOSE = yes >> >> I am unsure if this line or local.mk has priority. I am unsure the the >> following is better as well. >> BTEST_ERT_VERBOSE ?= yes > > I am not very familiar with Makefile conventions. Just followed the > existing settings in the same file. All other BTEST_ERT_* settings just > use "=". OK, maybe I will try it later. I just do not remember in which order assignments are evaluated. Likely everything works as expected and features like delayed assignment (simple "=" vs. ":=" ) do not cause any problem. >> Is there an easy way to limit number of failures before termination of >> tests in the case of verbose reporting? It should prevent test log from >> blowing too much. Usually there is no point in all details if all or >> even 1/4 of tests fails. > > The best approach I know is using BTEST_RE BTEST_RE is useful when you know in advance which tests you are interested in. I had in mind continuous integration and notifications. Single full error (or several ones) included into a message is convenient and may provide enough information for quick fix. If there are 500 errors there is no point to send all of them as notification. Either test is aborted after a couple of dozens of failures or total number of failures and full text of several errors are enough to avoid excessively long notification. Anyway nobody is going to read all hundreds of failures. >>> + TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST) >> >> A purist would say that it is not a directory, it is something like >> ...FLAGS or ...ARGS. I know, it was abused before your patch. > > I do not follow you here. > VARIABLE1=1 VARIABLE2=2 /path/to/script > is the usual way to set variables in script environment in bash. My bad. I did not noticed that it is a part of script of a rule. I believed that the whole line is assignment to TMPDIR variable. I am unsure if the following approach to avoid excessive ifeq's without modification of the rule is better: ifeq ($(BTEST_ERT_VERBOSE),yes) EMACS_TEST_VERBOSE=yes export EMACS_TEST_VERBOSE endif >> Shouldn't it be mentioned in testing/README? > > Only BTEST_RE is documented there. BTEST_PRE, BTEST_POST, > BTEST_OB_LANGUAGES, and BTEST_EXTRA are not documented. > I believe that the odds for the user to change BTEST_ERT_VERBOSE are > rather low and it is not worth mentioning it explicitly in README. > > Or, alternatively, we can document all the settings in README. > WDYT? I agree that default verbose log is preferred in most cases. The only exception when it becomes rather useless and just consumes disk space is something basic is broken (see my comment concerning limit on failures above). Let's assume that you convinced me and only rare developers will customize the value.
next prev parent reply other threads:[~2022-01-11 16:52 UTC|newest] Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-02 13:12 Ihor Radchenko 2022-01-03 5:35 ` Max Nikulin 2022-01-07 15:04 ` Ihor Radchenko 2022-01-11 16:46 ` Max Nikulin [this message] 2022-01-15 12:52 ` Max Nikulin 2022-01-15 13:06 ` Max Nikulin 2022-01-15 15:58 ` Max Nikulin 2022-01-21 13:33 ` Ihor Radchenko 2022-01-21 15:01 ` Max Nikulin 2022-01-23 13:31 ` Ihor Radchenko 2022-01-21 13:31 ` Ihor Radchenko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://www.orgmode.org/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org' \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [PATCH] make test: Make failure results more verbose' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).