Re: [PATCH] Patches for problems discovered with interactive testing

[Ihor Radchenko <yantar92@posteo.net>]

gerard.vermeulen@posteo.net writes:

> This patch addresses a number of glitches discovered during interactive
> testing and contains possible fixes (sub-patch test-ob-maxima fixes
> batch testing on my system, but not interactive testing).

Thanks for your interest in improving Org mode tests!

> The test-ob-maxima patch may be due to a different Maxima version, but
> I do not think so.  In batch testing test-ob-maxima passes contrary to
> interactive mode.

On my side, Maxima tests pass both interactively and non-interactively.
I use Maxima 5.47.0
What is your version?

> The test-ob-python patch unsets org-src-preserve-indentation which I
> have set.
>
> The test-ob patch also unsets org-src-preserve-indentation.

> The test-org patch also unsets org-src-preserve-indentation.  This
> reduces the number of failures to 2 while testing test-org
> interactively.

It is generally not expected that Org tests will pass when you try to
run them on Emacs instance with your personal config. This is because
you can have arbitrary customizations, hooks, or advices that change the
Org mode defaults.

let-binding a couple of variables cannot fix this general problem for
all scenarios. So, I do not see any reason to accept patches that make
the tests run with personal config of a single user.

What might be useful, however, is making sure that tests do not fail
interactively with a clean Emacs config (make repro).
Check out testing/README file.

>  (ert-deftest ob-maxima/string-input ()
>    "Test of string input"
>    (org-test-at-id "b5842ed4-8e8b-4b18-a1c9-cef006b6a6c8"
>      (org-babel-next-src-block 2)
> -    (should (equal "- sin(x)" (org-babel-execute-src-block)))))
> +    (should (equal '(("(linenum:0," "") ("-" "sin(x)"))
> +                   (org-babel-execute-src-block)))))

The test in question is testing the output of

#+begin_src maxima :var fun="sin(x)" :var q=2 :results silent
print(diff(fun, x, q))$
#+end_src

It is certainly not expected that such code block returns "linenum:0".

Tests in Org mode are verifying that Org mode behaves correctly. It
makes no sense to change the test when Org mode result is not correct -
we should instead fix Org mode.

In the majority of cases, if the test is failing, it is not something
wrong with the test, but something wrong with Org mode code that test is
verifying.

>  (ert-deftest test-ob/does-not-replace-a-block-with-the-results ()
>    (org-test-with-temp-text "#+NAME: foo
> diff --git a/testing/lisp/test-org-element.el b/testing/lisp/test-org-element.el
> index ca7d77e28..f0958ff59 100644
> --- a/testing/lisp/test-org-element.el
> +++ b/testing/lisp/test-org-element.el
> @@ -4457,7 +4457,7 @@ Text
>       '(paragraph nil "  Two spaces\n\n \n  Two spaces"))
>      '(paragraph nil "Two spaces\n\n\nTwo spaces")))
>    (should
> -   (equal
> +   (equal ;; Fails in test. Why? (equal ...) form works in ielm.

It would help if you provided the error ERT displayed when the test failed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>