[PATCH] Ignore flaky ob-python tests

[PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

The new async feature of python seems to be implemented in a way
that can break with high CPU load. This patch still runs the 3 tests in
question, but does not fail the complete testsuite if any of those fail.

Kind regards,
Christian

[-- Attachment #2: 0001-testing-lisp-test-ob-python.el-Ignore-flaky-tests.patch --]
[-- Type: application/octet-stream, Size: 1506 bytes --]

From d351883f774cf90048b9adecee4c79c8e30ccd52 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Christian=20K=C3=B6stlin?= <christian.koestlin@gmail.com>
Date: Wed, 2 Nov 2022 01:02:05 +0100
Subject: [PATCH] testing/lisp/test-ob-python.el: Ignore flaky tests

TINYCHANGE
---
 testing/lisp/test-ob-python.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index 1f5674013..b45d2c05c 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -209,6 +209,7 @@ time.sleep(.1)
 	    (org-babel-execute-src-block)))))
 
 (ert-deftest test-ob-python/async-simple-session-output ()
+  :expected-result t
   (let ((org-babel-temporary-directory temporary-file-directory)
         (org-confirm-babel-evaluate nil))
     (org-test-with-temp-text
@@ -226,6 +227,7 @@ print('Yep!')
 			       (org-babel-read-result)))))))))
 
 (ert-deftest test-ob-python/async-named-output ()
+  :expected-result t
   (let (org-confirm-babel-evaluate
         (org-babel-temporary-directory temporary-file-directory)
         (src-block "#+begin_src python :async :session :results output
@@ -250,6 +252,7 @@ print(\"Yep!\")
                              (buffer-string)))))))
 
 (ert-deftest test-ob-python/async-output-drawer ()
+  :expected-result t
   (let (org-confirm-babel-evaluate
         (org-babel-temporary-directory temporary-file-directory)
         (src-block "#+begin_src python :async :session :results output drawer
-- 
2.38.0

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin <christian.koestlin@gmail.com> writes:

> The new async feature of python seems to be implemented in a way
> that can break with high CPU load. This patch still runs the 3 tests in
> question, but does not fail the complete testsuite if any of those fail.

Thanks for the patch, but I'm afraid that it will drive the tests useless.
After the patch, even if the tests fail for a valid reason, it will not
be visible upon running make test, AFAIU.

Maybe we can instead provide some variable that can turn-on ignoring
some problematic tests only during CI tests?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

On Wed, Nov 2, 2022 at 7:52 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
> > The new async feature of python seems to be implemented in a way
> > that can break with high CPU load. This patch still runs the 3 tests in
> > question, but does not fail the complete testsuite if any of those fail.
>
> Thanks for the patch, but I'm afraid that it will drive the tests useless.
> After the patch, even if the tests fail for a valid reason, it will not
> be visible upon running make test, AFAIU.
I think we know of the async problem, and you reported that as well,
so I would expect
if someone is able to fix it, he will also move the tests to "sharp" again.
With my patch the tests are still failing but the testsuite continues
to run and returns
with 0 as status code (one can see the failed test though).

> Maybe we can instead provide some variable that can turn-on ignoring
> some problematic tests only during CI tests?
Yes ... that would also be possible e.g. skip the test or test nothing
if it's running on the ci.
(I am talking only about the 3 tests!).

Kind regards,
Christian

Re: [PATCH] Ignore flaky ob-python tests

[Jack Kamm]

Christian Köstlin <christian.koestlin@gmail.com> writes:

> I think we know of the async problem, and you reported that as well,
> so I would expect
> if someone is able to fix it, he will also move the tests to "sharp" again.

Could you provide some references about the async problem, either how to
reproduce it or the underlying cause? I tried searching emacs-orgmode
and emacs-devel, but couldn't find a report of the problem.

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
>> I think we know of the async problem, and you reported that as well,
>> so I would expect
>> if someone is able to fix it, he will also move the tests to "sharp" again.
>
> Could you provide some references about the async problem, either how to
> reproduce it or the underlying cause? I tried searching emacs-orgmode
> and emacs-devel, but couldn't find a report of the problem.

https://lists.sr.ht/~bzg/org-build-failures/%3CCNRV9YIXSILF.2AF0TMJG974H4%40cirno%3E


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> Christian Köstlin <christian.koestlin@gmail.com> writes:
>>
>>> I think we know of the async problem, and you reported that as well,
>>> so I would expect
>>> if someone is able to fix it, he will also move the tests to "sharp" again.
>>
>> Could you provide some references about the async problem, either how to
>> reproduce it or the underlying cause? I tried searching emacs-orgmode
>> and emacs-devel, but couldn't find a report of the problem.

Also, https://lists.sr.ht/~bzg/org-build-failures/%3CCNZYPX9R5ABN.VI1C53TZJSTI%40cirno2%3E

BTW, Christian, failures of tests in the patch are valid failures that
revealed a real problem.
See https://orgmode.org/list/87bkprid1d.fsf@localhost

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin <christian.koestlin@gmail.com> writes:

> With my patch the tests are still failing but the testsuite continues
> to run and returns
> with 0 as status code (one can see the failed test though).

FYI, I am using automated scripts to run make test locally on multiple
Emacs versions. 0 status code would do no good for my testing.

>> Maybe we can instead provide some variable that can turn-on ignoring
>> some problematic tests only during CI tests?
> Yes ... that would also be possible e.g. skip the test or test nothing
> if it's running on the ci.
> (I am talking only about the 3 tests!).

More like just 1 test with underscore :)
The 3 tests in the patch are real failures because ob-python does not
fully support python-mode.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

Hi Ihor,

I somehow seem to remember that the async tests were also broken with
python.el (python-mode.el not installed).
at least sometimes? or is this only the underscore test?
If it's only the underscore test, I would suggest not installing
python-mode for ci and ignoring the really flaky tests (i think the
underscore one is a candidate here).

I somehow got the async things mixed up with the flaky tests, which is
not true for python-mode.el installed.

what do you think?

I really would like to see python tests pass ...

Kind regards,
Christian

On Thu, Nov 3, 2022 at 8:13 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
> > With my patch the tests are still failing but the testsuite continues
> > to run and returns
> > with 0 as status code (one can see the failed test though).
>
> FYI, I am using automated scripts to run make test locally on multiple
> Emacs versions. 0 status code would do no good for my testing.
>
> >> Maybe we can instead provide some variable that can turn-on ignoring
> >> some problematic tests only during CI tests?
> > Yes ... that would also be possible e.g. skip the test or test nothing
> > if it's running on the ci.
> > (I am talking only about the 3 tests!).
>
> More like just 1 test with underscore :)
> The 3 tests in the patch are real failures because ob-python does not
> fully support python-mode.el.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin <christian.koestlin@gmail.com> writes:

> I somehow seem to remember that the async tests were also broken with
> python.el (python-mode.el not installed).
> at least sometimes?

I only recall the underscore test.

Note that I debugged the async tests with your rake builds, and I have
identified that async tests are failing because of python-mode
inclusion. (python-mode is not supported by async ob-python code).

> or is this only the underscore test?
> If it's only the underscore test, I would suggest not installing
> python-mode for ci and ignoring the really flaky tests (i think the
> underscore one is a candidate here).

Yes. Please, go ahead and remove python-mode from the CI tests.

We are likely going to deprecate python-mode support. See
https://orgmode.org/list/87r0yk7bx8.fsf@localhost

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>