From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id IOo/Bm9W2GG/+wAAgWs5BA (envelope-from ) for ; Fri, 07 Jan 2022 16:04:15 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id qT6ZA29W2GEHHgAA9RJhRA (envelope-from ) for ; Fri, 07 Jan 2022 16:04:15 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 52AEE329FC for ; Fri, 7 Jan 2022 16:04:14 +0100 (CET) Received: from localhost ([::1]:43570 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n5qmq-0006ip-05 for larch@yhetil.org; Fri, 07 Jan 2022 10:04:12 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5qly-0006gE-Ok for emacs-orgmode@gnu.org; Fri, 07 Jan 2022 10:03:18 -0500 Received: from [2a00:1450:4864:20::129] (port=35627 helo=mail-lf1-x129.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n5qlx-0005rb-68 for emacs-orgmode@gnu.org; Fri, 07 Jan 2022 10:03:18 -0500 Received: by mail-lf1-x129.google.com with SMTP id g11so16527964lfu.2 for ; Fri, 07 Jan 2022 07:03:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=JYrBXpoFLNEdzvs1SgzeoO71GQICtsvvhZP64GJwySs=; b=Rz8CFZ0kLJXl354QGsr8DbmKtAArlw93CPtHT7p3Q7+AqO2XNvA6z3/EKaXsHHnqp+ 0/MD+3CiWx3o1sDHBzTejJkPG+kRX7DiCNqPnu2y9fudEui1hCS0naSLvIlZfmOzcZyJ FytOigV8+9Datw7B4+zyQh5wepzb03Ta4fYeHu0zY/dxuU2MuToRixrHiA7uRe0fr+4c UgGSRfO7+D5KNTSLTQJk0WIGJ1sqCHKLToRK/FdP9dIBcF5j/I1vCTC0CvtSabMrIRzR KV7kgjm8j4PqbJ76o3aVSUsaB6I/RIpfVGmQsJCirKc441ux29Bc5JP5VP406zK0FbPH tENQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=JYrBXpoFLNEdzvs1SgzeoO71GQICtsvvhZP64GJwySs=; b=opg3t5TB2kpxibLt3PirtsHgnFhFqa8E05o7V4rMbj9JmN8lk6qQXx0jnKcvMltu8j VvdAMjdChzGrNv8p0Yml8zOG81/sX2S9wz8pIyrEqWwMT3IBrFQPQgjLXpYaGRBqkFbc Znkq07prax45YAt3ftS3dQzSHEswBLfBffRun10fnQypvsT2/LeIxnVz3TisxEKv5Gd1 jYBUvqb36LmgMjqgRHVgen+ZVC7Il4dDPXIlDqEJJcTrVNy4ia5djOeCBM9iesDl9CEE WYfnAKzbpncL5I9tfh2YbkOtnLu2Fg0txp0kgAH+wE6lDWudEMyDQ29Dyo38t+ohBJLa OdPg== X-Gm-Message-State: AOAM533jR5ALpnHXAJLjiJ4YPwyxWofl23pKy7yzCQmAOfRM4ZPpeCh4 E0YeKiHNUMjKFTRLbiRBRw/8UAHgMW7jYQ== X-Google-Smtp-Source: ABdhPJz8nXGDVW1Z67HfQzS9PAB8JwFtleQdfIQ/ww0Q2plLBAGwb/5cAt60KS6hf3AZ3+ocS8Wqvg== X-Received: by 2002:ac2:4e06:: with SMTP id e6mr55995476lfr.295.1641567795537; Fri, 07 Jan 2022 07:03:15 -0800 (PST) Received: from localhost ([91.210.107.150]) by smtp.gmail.com with ESMTPSA id u11sm603093lff.308.2022.01.07.07.03.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Jan 2022 07:03:15 -0800 (PST) From: Ihor Radchenko To: Max Nikulin Subject: Re: [PATCH] make test: Make failure results more verbose In-Reply-To: References: <87ee5q5lic.fsf@localhost> Date: Fri, 07 Jan 2022 23:04:32 +0800 Message-ID: <87lezrr3i7.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::129 (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::129; envelope-from=yantar92@gmail.com; helo=mail-lf1-x129.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode@gnu.org Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641567854; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=JYrBXpoFLNEdzvs1SgzeoO71GQICtsvvhZP64GJwySs=; b=SLryKRro/IsocXhQy9GPBlM7jcyAVeYpHSPx0dL2VviadfqX8EXSD0T8jdxcaCqzlxT3Ab 8+c17IMN5bpW/xOUT1Wb4DMfws3KiQv9klYzH1M62UGEXis5YsDKXVaCEtzA4rG17g/L4i 0QmxcE5EY88ETaFq24B29TAEoEiGphceiuATOD7hynZ5tBiLApGitPVlzgUyhzyzsTuIX7 lXMsXNjx1iO6BlSMFqdZRpwmdjVENtAFdhX4snABttco46bp6pCcl2dGuNv37mtBukog58 2s/m+kpFRfJNxSHGfOvVQ5IBQy72Zgyjr10uPeqc/SYWNO2b8QJULKUd58Resw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641567854; a=rsa-sha256; cv=none; b=VX6e3uHuPikFMHblq5XgTnk65I1tzm9Itfi0vRVOb/nrr3msM8rhhyVbOR1YRtuIzdtpZj 31o1A8Q+fgiTM6tXeWBGcCA4DgpdvFxU7DkwDX+byKXV7d8z/6JUhx0nqA6QFLn6Qo6ila V4tlhKVRc4VxdwGjySqKRL+8VTYr97//QMSDN2LkG16kU0knyF6C+0l49ZjwImTplF8RfU RS02TOPzQtXc5cpaQ+M7H67yG9qAkuD7cdgCKbSp6dmchd92YopqVN25OPs5eCR/AXXAfQ UDREd8vR2/cQnSYqOXLwrGe+5EPK3kurTbAl00R7oNEnSDIKRe3TeUDdpNGQag== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Rz8CFZ0k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.80 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Rz8CFZ0k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 52AEE329FC X-Spam-Score: -3.80 X-Migadu-Scanner: scn1.migadu.com X-TUID: N94nxjBBRcX7 Max Nikulin writes: > Ihor, are there examples of new error reports in mail lists, blogs, etc? > I am not motivated enough to try development version of emacs, but my > impression that current error log looks like rectangles of garbage. Yeah, I should have attached examples to the original message. Also, we can adjust the ERT output using ert-batch-backtrace-right-margin, ert-batch-print-level, ert-batch-print-level, and ert-batch-backtrace-line-length Here are the examples: 1 unexpected results: FAILED test-org-element/center-block-parser "This error is thrown" vs 1 unexpected results: FAILED test-org-element/center-block-parser and 1 unexpected results: FAILED test-org-element/bold-parser ((should (org-test-with-temp-text "*bold *" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t))) :form (let ((inside-text (if (stringp "*bold *") "*bold *" (eval "*bold *"))) (org-mode-hook nil)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn ... ... ... ...) (and ... ...))))) :value nil) vs 1 unexpected results: FAILED test-org-element/bold-parser and 1 unexpected results: FAILED test-org-element/bold-parser ((should (equal (org-element-contents (org-test-with-temp-text "*first line\nsecond blah line*" (org-element-map ... ... ... nil t))) '("first line\nsecond line"))) :form (equal (#("first line\nsecond blah line" 0 27 (:parent (bold ... #3)))) ("first line\nsecond line")) :value nil :explanation (list-elt 0 (arrays-of-different-length 27 22 #("first line\nsecond blah line" 0 27 (:parent (bold ... #3))) "first line\nsecond line" first-mismatch-at 18))) and 1 unexpected results: FAILED test-org-element/citation-parser ((should (equal '("a" "b" "c") (org-test-with-temp-text "[cite:@a;@bd;@c]" (org-element-map (org-element-parse-buffer) 'citation-reference (lambda ... ...))))) :form (equal ("a" "b" "c") ("a" "bd" "c")) :value nil :explanation (list-elt 1 (arrays-of-different-length 1 2 "b" "bd" first-mismatch-at 1))) > In my opinion, code of test should be written having clear error reports > in mind. I guess. Though I feel that ERT is better when used interactively. Do you have good ideas what could be changed? >> +BTEST_ERT_VERBOSE = yes > > I am unsure if this line or local.mk has priority. I am unsure the the > following is better as well. > BTEST_ERT_VERBOSE ?= yes I am not very familiar with Makefile conventions. Just followed the existing settings in the same file. All other BTEST_ERT_* settings just use "=". > Is there an easy way to limit number of failures before termination of > tests in the case of verbose reporting? It should prevent test log from > blowing too much. Usually there is no point in all details if all or > even 1/4 of tests fails. The best approach I know is using BTEST_RE >> + TMPDIR=$(testdir) EMACS_TEST_VERBOSE=yes $(BTEST) > > A purist would say that it is not a directory, it is something like > ...FLAGS or ...ARGS. I know, it was abused before your patch. I do not follow you here. VARIABLE1=1 VARIABLE2=2 /path/to/script is the usual way to set variables in script environment in bash. > Shouldn't it be mentioned in testing/README? Only BTEST_RE is documented there. BTEST_PRE, BTEST_POST, BTEST_OB_LANGUAGES, and BTEST_EXTRA are not documented. I believe that the odds for the user to change BTEST_ERT_VERBOSE are rather low and it is not worth mentioning it explicitly in README. Or, alternatively, we can document all the settings in README. WDYT? Best, Ihor