emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: Ihor Radchenko <yantar92@gmail.com>
To: Max Nikulin <manikulin@gmail.com>
Cc: emacs-orgmode@gnu.org
Subject: Re: [PATCH] make test: Make failure results more verbose
Date: Fri, 07 Jan 2022 23:04:32 +0800	[thread overview]
Message-ID: <87lezrr3i7.fsf@localhost> (raw)
In-Reply-To: <squ1um$nj1$1@ciao.gmane.io>

Max Nikulin <manikulin@gmail.com> writes:

> Ihor, are there examples of new error reports in mail lists, blogs, etc? 
> I am not motivated enough to try development version of emacs, but my 
> impression that current error log looks like rectangles of garbage.

Yeah, I should have attached examples to the original message.
Also, we can adjust the ERT output using
ert-batch-backtrace-right-margin, ert-batch-print-level,
ert-batch-print-level, and ert-batch-backtrace-line-length

Here are the examples:

1 unexpected results:
   FAILED  test-org-element/center-block-parser  "This error is thrown"
vs
1 unexpected results:
   FAILED  test-org-element/center-block-parser

and

1 unexpected results:
   FAILED  test-org-element/bold-parser  ((should (org-test-with-temp-text "*bold *" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t))) :form (let ((inside-text (if (stringp "*bold *") "*bold *" (eval "*bold *"))) (org-mode-hook nil)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn ... ... ... ...) (and ... ...))))) :value nil)
vs
1 unexpected results:
   FAILED  test-org-element/bold-parser

and

1 unexpected results:
   FAILED  test-org-element/bold-parser  ((should (equal (org-element-contents (org-test-with-temp-text "*first line\nsecond blah line*" (org-element-map ... ... ... nil t))) '("first line\nsecond line"))) :form (equal (#("first line\nsecond blah line" 0 27 (:parent (bold ... #3)))) ("first line\nsecond line")) :value nil :explanation (list-elt 0 (arrays-of-different-length 27 22 #("first line\nsecond blah line" 0 27 (:parent (bold ... #3))) "first line\nsecond line" first-mismatch-at 18)))

and

1 unexpected results:
   FAILED  test-org-element/citation-parser  ((should (equal '("a" "b" "c") (org-test-with-temp-text "[cite:@a;@bd;@c]" (org-element-map (org-element-parse-buffer) 'citation-reference (lambda ... ...))))) :form (equal ("a" "b" "c") ("a" "bd" "c")) :value nil :explanation (list-elt 1 (arrays-of-different-length 1 2 "b" "bd" first-mismatch-at 1)))

> In my opinion, code of test should be written having clear error reports 
> in mind.

I guess. Though I feel that ERT is better when used interactively.
Do you have good ideas what could be changed?

>> +BTEST_ERT_VERBOSE = yes
>
> I am unsure if this line or local.mk has priority. I am unsure the the 
> following is better as well.
> BTEST_ERT_VERBOSE ?= yes

I am not very familiar with Makefile conventions. Just followed the
existing settings in the same file. All other BTEST_ERT_* settings just
use "=".

> Is there an easy way to limit number of failures before termination of 
> tests in the case of verbose reporting? It should prevent test log from 
> blowing too much. Usually there is no point in all details if all or 
> even 1/4 of tests fails.

The best approach I know is using BTEST_RE

>> +	TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST)
>
> A purist would say that it is not a directory, it is something like 
> ...FLAGS or ...ARGS. I know, it was abused before your patch.

I do not follow you here.
VARIABLE1=1 VARIABLE2=2 /path/to/script
is the usual way to set variables in script environment in bash.

> Shouldn't it be mentioned in testing/README?

Only BTEST_RE is documented there. BTEST_PRE, BTEST_POST,
BTEST_OB_LANGUAGES, and BTEST_EXTRA are not documented.
I believe that the odds for the user to change BTEST_ERT_VERBOSE are
rather low and it is not worth mentioning it explicitly in README.

Or, alternatively, we can document all the settings in README.
WDYT?

Best,
Ihor


  reply	other threads:[~2022-01-07 15:04 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-01-02 13:12 Ihor Radchenko
2022-01-03  5:35 ` Max Nikulin
2022-01-07 15:04   ` Ihor Radchenko [this message]
2022-01-11 16:46     ` Max Nikulin
2022-01-15 12:52       ` Max Nikulin
2022-01-15 13:06         ` Max Nikulin
2022-01-15 15:58           ` Max Nikulin
2022-01-21 13:33             ` Ihor Radchenko
2022-01-21 15:01               ` Max Nikulin
2022-01-23 13:31                 ` Ihor Radchenko
2022-01-21 13:31         ` Ihor Radchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87lezrr3i7.fsf@localhost \
    --to=yantar92@gmail.com \
    --cc=emacs-orgmode@gnu.org \
    --cc=manikulin@gmail.com \
    --subject='Re: [PATCH] make test: Make failure results more verbose' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).