From: gerard.vermeulen@posteo.net
To: Ihor Radchenko <yantar92@posteo.net>
Cc: Emacs orgmode <emacs-orgmode@gnu.org>
Subject: Re: [PATCH] Patches for problems discovered with interactive testing
Date: Fri, 19 Jan 2024 14:45:48 +0000 [thread overview]
Message-ID: <dd2a8d43425b919682c85b55e03c41da@posteo.net> (raw)
In-Reply-To: <874jf9pjw4.fsf@localhost>
On 19.01.2024 13:50, Ihor Radchenko wrote:
> gerard.vermeulen@posteo.net writes:
>
>> This patch addresses a number of glitches discovered during
>> interactive
>> testing and contains possible fixes (sub-patch test-ob-maxima fixes
>> batch testing on my system, but not interactive testing).
>
> Thanks for your interest in improving Org mode tests!
>
>> The test-ob-maxima patch may be due to a different Maxima version, but
>> I do not think so. In batch testing test-ob-maxima passes contrary to
>> interactive mode.
>
> On my side, Maxima tests pass both interactively and non-interactively.
> I use Maxima 5.47.0
> What is your version?
The same: maxima --version
Maxima 5.47.0
In batch testing without the patch I get:
Test ob-maxima/batch+verbatim condition:
(ert-test-failed
((should
(equal (org-babel-execute-src-block)
"(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n
gamma(z + 1)"))
:form
(equal
"(linenum:0,\n(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n
gamma(z + 1)"
"(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n
gamma(z + 1)")
:value nil :explanation
(arrays-of-different-length 104 92
"(linenum:0,\n(assume(z >
0),integrate(exp(-t)*t^z,t,0,inf))\n
gamma(z + 1)"
"(assume(z >
0),integrate(exp(-t)*t^z,t,0,inf))\n
gamma(z + 1)"
first-mismatch-at 1)))
FAILED 162/1252 ob-maxima/batch+verbatim (0.694916 sec) at
../lisp/test-ob
-maxima.el:71
I do not know where the difference comes from. I am not a Maxima user
and not inclined to track this down unless you tell me it is important.
>
>> The test-ob-python patch unsets org-src-preserve-indentation which I
>> have set.
>>
>> The test-ob patch also unsets org-src-preserve-indentation.
>
>> The test-org patch also unsets org-src-preserve-indentation. This
>> reduces the number of failures to 2 while testing test-org
>> interactively.
>
> It is generally not expected that Org tests will pass when you try to
> run them on Emacs instance with your personal config. This is because
> you can have arbitrary customizations, hooks, or advices that change
> the
> Org mode defaults.
>
> let-binding a couple of variables cannot fix this general problem for
> all scenarios. So, I do not see any reason to accept patches that make
> the tests run with personal config of a single user.
>
Before I made the patch, I had reverted changes that worked
interactively
in my personal config but caused failures with make test.
Because of the Maxima test failure (also in batch), I was not sure that
all
tests pass on a reference system.
> What might be useful, however, is making sure that tests do not fail
> interactively with a clean Emacs config (make repro).
> Check out testing/README file.
>
I have a super-weird system where make repro opens Emacs with
as default-directory my home directory. So I have to make symbolic
links from my home to the testing and lisp directories.
But make test works.
Because of testing my
0001-org-babel-demarcate-block-split-using-element-API.patch
interactively in my personal config (before make test which takes
quite some time), I figured out that it would guess to suppress false
failures.
>
>> (ert-deftest ob-maxima/string-input ()
>> "Test of string input"
>> (org-test-at-id "b5842ed4-8e8b-4b18-a1c9-cef006b6a6c8"
>> (org-babel-next-src-block 2)
>> - (should (equal "- sin(x)" (org-babel-execute-src-block)))))
>> + (should (equal '(("(linenum:0," "") ("-" "sin(x)"))
>> + (org-babel-execute-src-block)))))
>
> The test in question is testing the output of
>
> #+begin_src maxima :var fun="sin(x)" :var q=2 :results silent
> print(diff(fun, x, q))$
> #+end_src
>
> It is certainly not expected that such code block returns "linenum:0".
I would say that it is an unexplained difference, I still get (with
"linenum:0") :
#+header: :wrap "src text -n"
#+name: lst:maxima-example
#+begin_src maxima -n :batch batch :cmdline --quiet :exports both
:results raw
rat(1/(x+1) + x/(x-1));
#+end_src
#+name: lst:maxima-example-results
#+results: lst:maxima-example
#+begin_src text -n
(%i1) (linenum:0,
(%i1) rat(1/(x+1)+x/(x-1))
2
x + 2 x - 1
(%o1)/R/ ------------
2
x - 1
#+end_src
>
> Tests in Org mode are verifying that Org mode behaves correctly. It
> makes no sense to change the test when Org mode result is not correct -
> we should instead fix Org mode.
>
> In the majority of cases, if the test is failing, it is not something
> wrong with the test, but something wrong with Org mode code that test
> is
> verifying.
>
>> (ert-deftest test-ob/does-not-replace-a-block-with-the-results ()
>> (org-test-with-temp-text "#+NAME: foo
>> diff --git a/testing/lisp/test-org-element.el
>> b/testing/lisp/test-org-element.el
>> index ca7d77e28..f0958ff59 100644
>> --- a/testing/lisp/test-org-element.el
>> +++ b/testing/lisp/test-org-element.el
>> @@ -4457,7 +4457,7 @@ Text
>> '(paragraph nil " Two spaces\n\n \n Two spaces"))
>> '(paragraph nil "Two spaces\n\n\nTwo spaces")))
>> (should
>> - (equal
>> + (equal ;; Fails in test. Why? (equal ...) form works in ielm.
>
> It would help if you provided the error ERT displayed when the test
> failed.
I cannot reproduce it. Now it works in interactive testing (personal
config
as when I first saw this).
If you ask me, I am still willing to collect the 2 sub-patches with the
let bindings which improve testing interactively in personal configs
somewhat. If not, it is OK (I understand now that it is not important).
Regards -- Gerard
prev parent reply other threads:[~2024-01-19 14:47 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-19 9:48 [PATCH] Patches for problems discovered with interactive testing gerard.vermeulen
2024-01-19 12:50 ` Ihor Radchenko
2024-01-19 14:45 ` gerard.vermeulen [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dd2a8d43425b919682c85b55e03c41da@posteo.net \
--to=gerard.vermeulen@posteo.net \
--cc=emacs-orgmode@gnu.org \
--cc=yantar92@posteo.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).