From: Ihor Radchenko <yantar92@gmail.com>
To: Max Nikulin <manikulin@gmail.com>
Cc: emacs-orgmode@gnu.org
Subject: Re: [PATCH] make test: Make failure results more verbose
Date: Fri, 07 Jan 2022 23:04:32 +0800 [thread overview]
Message-ID: <87lezrr3i7.fsf@localhost> (raw)
In-Reply-To: <squ1um$nj1$1@ciao.gmane.io>
Max Nikulin <manikulin@gmail.com> writes:
> Ihor, are there examples of new error reports in mail lists, blogs, etc?
> I am not motivated enough to try development version of emacs, but my
> impression that current error log looks like rectangles of garbage.
Yeah, I should have attached examples to the original message.
Also, we can adjust the ERT output using
ert-batch-backtrace-right-margin, ert-batch-print-level,
ert-batch-print-level, and ert-batch-backtrace-line-length
Here are the examples:
1 unexpected results:
FAILED test-org-element/center-block-parser "This error is thrown"
vs
1 unexpected results:
FAILED test-org-element/center-block-parser
and
1 unexpected results:
FAILED test-org-element/bold-parser ((should (org-test-with-temp-text "*bold *" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t))) :form (let ((inside-text (if (stringp "*bold *") "*bold *" (eval "*bold *"))) (org-mode-hook nil)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn ... ... ... ...) (and ... ...))))) :value nil)
vs
1 unexpected results:
FAILED test-org-element/bold-parser
and
1 unexpected results:
FAILED test-org-element/bold-parser ((should (equal (org-element-contents (org-test-with-temp-text "*first line\nsecond blah line*" (org-element-map ... ... ... nil t))) '("first line\nsecond line"))) :form (equal (#("first line\nsecond blah line" 0 27 (:parent (bold ... #3)))) ("first line\nsecond line")) :value nil :explanation (list-elt 0 (arrays-of-different-length 27 22 #("first line\nsecond blah line" 0 27 (:parent (bold ... #3))) "first line\nsecond line" first-mismatch-at 18)))
and
1 unexpected results:
FAILED test-org-element/citation-parser ((should (equal '("a" "b" "c") (org-test-with-temp-text "[cite:@a;@bd;@c]" (org-element-map (org-element-parse-buffer) 'citation-reference (lambda ... ...))))) :form (equal ("a" "b" "c") ("a" "bd" "c")) :value nil :explanation (list-elt 1 (arrays-of-different-length 1 2 "b" "bd" first-mismatch-at 1)))
> In my opinion, code of test should be written having clear error reports
> in mind.
I guess. Though I feel that ERT is better when used interactively.
Do you have good ideas what could be changed?
>> +BTEST_ERT_VERBOSE = yes
>
> I am unsure if this line or local.mk has priority. I am unsure the the
> following is better as well.
> BTEST_ERT_VERBOSE ?= yes
I am not very familiar with Makefile conventions. Just followed the
existing settings in the same file. All other BTEST_ERT_* settings just
use "=".
> Is there an easy way to limit number of failures before termination of
> tests in the case of verbose reporting? It should prevent test log from
> blowing too much. Usually there is no point in all details if all or
> even 1/4 of tests fails.
The best approach I know is using BTEST_RE
>> + TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST)
>
> A purist would say that it is not a directory, it is something like
> ...FLAGS or ...ARGS. I know, it was abused before your patch.
I do not follow you here.
VARIABLE1=1 VARIABLE2=2 /path/to/script
is the usual way to set variables in script environment in bash.
> Shouldn't it be mentioned in testing/README?
Only BTEST_RE is documented there. BTEST_PRE, BTEST_POST,
BTEST_OB_LANGUAGES, and BTEST_EXTRA are not documented.
I believe that the odds for the user to change BTEST_ERT_VERBOSE are
rather low and it is not worth mentioning it explicitly in README.
Or, alternatively, we can document all the settings in README.
WDYT?
Best,
Ihor
next prev parent reply other threads:[~2022-01-07 15:04 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-02 13:12 [PATCH] make test: Make failure results more verbose Ihor Radchenko
2022-01-03 5:35 ` Max Nikulin
2022-01-07 15:04 ` Ihor Radchenko [this message]
2022-01-11 16:46 ` Max Nikulin
2022-01-15 12:52 ` Max Nikulin
2022-01-15 13:06 ` Max Nikulin
2022-01-15 15:58 ` Max Nikulin
2022-01-21 13:33 ` Ihor Radchenko
2022-01-21 15:01 ` Max Nikulin
2022-01-23 13:31 ` Ihor Radchenko
2022-01-21 13:31 ` Ihor Radchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lezrr3i7.fsf@localhost \
--to=yantar92@gmail.com \
--cc=emacs-orgmode@gnu.org \
--cc=manikulin@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).