* [PATCH] Ignore flaky ob-python tests
@ 2022-11-02 0:41 Christian Köstlin
2022-11-02 6:53 ` Ihor Radchenko
0 siblings, 1 reply; 9+ messages in thread
From: Christian Köstlin @ 2022-11-02 0:41 UTC (permalink / raw)
To: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 235 bytes --]
The new async feature of python seems to be implemented in a way
that can break with high CPU load. This patch still runs the 3 tests in
question, but does not fail the complete testsuite if any of those fail.
Kind regards,
Christian
[-- Attachment #2: 0001-testing-lisp-test-ob-python.el-Ignore-flaky-tests.patch --]
[-- Type: application/octet-stream, Size: 1506 bytes --]
From d351883f774cf90048b9adecee4c79c8e30ccd52 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Christian=20K=C3=B6stlin?= <christian.koestlin@gmail.com>
Date: Wed, 2 Nov 2022 01:02:05 +0100
Subject: [PATCH] testing/lisp/test-ob-python.el: Ignore flaky tests
TINYCHANGE
---
testing/lisp/test-ob-python.el | 3 +++
1 file changed, 3 insertions(+)
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index 1f5674013..b45d2c05c 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -209,6 +209,7 @@ time.sleep(.1)
(org-babel-execute-src-block)))))
(ert-deftest test-ob-python/async-simple-session-output ()
+ :expected-result t
(let ((org-babel-temporary-directory temporary-file-directory)
(org-confirm-babel-evaluate nil))
(org-test-with-temp-text
@@ -226,6 +227,7 @@ print('Yep!')
(org-babel-read-result)))))))))
(ert-deftest test-ob-python/async-named-output ()
+ :expected-result t
(let (org-confirm-babel-evaluate
(org-babel-temporary-directory temporary-file-directory)
(src-block "#+begin_src python :async :session :results output
@@ -250,6 +252,7 @@ print(\"Yep!\")
(buffer-string)))))))
(ert-deftest test-ob-python/async-output-drawer ()
+ :expected-result t
(let (org-confirm-babel-evaluate
(org-babel-temporary-directory temporary-file-directory)
(src-block "#+begin_src python :async :session :results output drawer
--
2.38.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 0:41 [PATCH] Ignore flaky ob-python tests Christian Köstlin
@ 2022-11-02 6:53 ` Ihor Radchenko
2022-11-02 11:15 ` Christian Köstlin
0 siblings, 1 reply; 9+ messages in thread
From: Ihor Radchenko @ 2022-11-02 6:53 UTC (permalink / raw)
To: Christian Köstlin; +Cc: emacs-orgmode
Christian Köstlin <christian.koestlin@gmail.com> writes:
> The new async feature of python seems to be implemented in a way
> that can break with high CPU load. This patch still runs the 3 tests in
> question, but does not fail the complete testsuite if any of those fail.
Thanks for the patch, but I'm afraid that it will drive the tests useless.
After the patch, even if the tests fail for a valid reason, it will not
be visible upon running make test, AFAIU.
Maybe we can instead provide some variable that can turn-on ignoring
some problematic tests only during CI tests?
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 6:53 ` Ihor Radchenko
@ 2022-11-02 11:15 ` Christian Köstlin
2022-11-02 13:55 ` Jack Kamm
2022-11-03 7:14 ` Ihor Radchenko
0 siblings, 2 replies; 9+ messages in thread
From: Christian Köstlin @ 2022-11-02 11:15 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
On Wed, Nov 2, 2022 at 7:52 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
> > The new async feature of python seems to be implemented in a way
> > that can break with high CPU load. This patch still runs the 3 tests in
> > question, but does not fail the complete testsuite if any of those fail.
>
> Thanks for the patch, but I'm afraid that it will drive the tests useless.
> After the patch, even if the tests fail for a valid reason, it will not
> be visible upon running make test, AFAIU.
I think we know of the async problem, and you reported that as well,
so I would expect
if someone is able to fix it, he will also move the tests to "sharp" again.
With my patch the tests are still failing but the testsuite continues
to run and returns
with 0 as status code (one can see the failed test though).
> Maybe we can instead provide some variable that can turn-on ignoring
> some problematic tests only during CI tests?
Yes ... that would also be possible e.g. skip the test or test nothing
if it's running on the ci.
(I am talking only about the 3 tests!).
Kind regards,
Christian
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 11:15 ` Christian Köstlin
@ 2022-11-02 13:55 ` Jack Kamm
2022-11-02 14:12 ` Ihor Radchenko
2022-11-03 7:14 ` Ihor Radchenko
1 sibling, 1 reply; 9+ messages in thread
From: Jack Kamm @ 2022-11-02 13:55 UTC (permalink / raw)
To: Christian Köstlin, Ihor Radchenko; +Cc: emacs-orgmode
Christian Köstlin <christian.koestlin@gmail.com> writes:
> I think we know of the async problem, and you reported that as well,
> so I would expect
> if someone is able to fix it, he will also move the tests to "sharp" again.
Could you provide some references about the async problem, either how to
reproduce it or the underlying cause? I tried searching emacs-orgmode
and emacs-devel, but couldn't find a report of the problem.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 13:55 ` Jack Kamm
@ 2022-11-02 14:12 ` Ihor Radchenko
2022-11-02 14:17 ` Ihor Radchenko
0 siblings, 1 reply; 9+ messages in thread
From: Ihor Radchenko @ 2022-11-02 14:12 UTC (permalink / raw)
To: Jack Kamm; +Cc: Christian Köstlin, emacs-orgmode
Jack Kamm <jackkamm@gmail.com> writes:
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
>> I think we know of the async problem, and you reported that as well,
>> so I would expect
>> if someone is able to fix it, he will also move the tests to "sharp" again.
>
> Could you provide some references about the async problem, either how to
> reproduce it or the underlying cause? I tried searching emacs-orgmode
> and emacs-devel, but couldn't find a report of the problem.
https://lists.sr.ht/~bzg/org-build-failures/%3CCNRV9YIXSILF.2AF0TMJG974H4%40cirno%3E
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 14:12 ` Ihor Radchenko
@ 2022-11-02 14:17 ` Ihor Radchenko
0 siblings, 0 replies; 9+ messages in thread
From: Ihor Radchenko @ 2022-11-02 14:17 UTC (permalink / raw)
To: Jack Kamm; +Cc: Christian Köstlin, emacs-orgmode
Ihor Radchenko <yantar92@posteo.net> writes:
> Jack Kamm <jackkamm@gmail.com> writes:
>
>> Christian Köstlin <christian.koestlin@gmail.com> writes:
>>
>>> I think we know of the async problem, and you reported that as well,
>>> so I would expect
>>> if someone is able to fix it, he will also move the tests to "sharp" again.
>>
>> Could you provide some references about the async problem, either how to
>> reproduce it or the underlying cause? I tried searching emacs-orgmode
>> and emacs-devel, but couldn't find a report of the problem.
Also, https://lists.sr.ht/~bzg/org-build-failures/%3CCNZYPX9R5ABN.VI1C53TZJSTI%40cirno2%3E
BTW, Christian, failures of tests in the patch are valid failures that
revealed a real problem.
See https://orgmode.org/list/87bkprid1d.fsf@localhost
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-02 11:15 ` Christian Köstlin
2022-11-02 13:55 ` Jack Kamm
@ 2022-11-03 7:14 ` Ihor Radchenko
2022-11-03 21:15 ` Christian Köstlin
1 sibling, 1 reply; 9+ messages in thread
From: Ihor Radchenko @ 2022-11-03 7:14 UTC (permalink / raw)
To: Christian Köstlin; +Cc: emacs-orgmode
Christian Köstlin <christian.koestlin@gmail.com> writes:
> With my patch the tests are still failing but the testsuite continues
> to run and returns
> with 0 as status code (one can see the failed test though).
FYI, I am using automated scripts to run make test locally on multiple
Emacs versions. 0 status code would do no good for my testing.
>> Maybe we can instead provide some variable that can turn-on ignoring
>> some problematic tests only during CI tests?
> Yes ... that would also be possible e.g. skip the test or test nothing
> if it's running on the ci.
> (I am talking only about the 3 tests!).
More like just 1 test with underscore :)
The 3 tests in the patch are real failures because ob-python does not
fully support python-mode.el.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-03 7:14 ` Ihor Radchenko
@ 2022-11-03 21:15 ` Christian Köstlin
2022-11-04 4:19 ` Ihor Radchenko
0 siblings, 1 reply; 9+ messages in thread
From: Christian Köstlin @ 2022-11-03 21:15 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
Hi Ihor,
I somehow seem to remember that the async tests were also broken with
python.el (python-mode.el not installed).
at least sometimes? or is this only the underscore test?
If it's only the underscore test, I would suggest not installing
python-mode for ci and ignoring the really flaky tests (i think the
underscore one is a candidate here).
I somehow got the async things mixed up with the flaky tests, which is
not true for python-mode.el installed.
what do you think?
I really would like to see python tests pass ...
Kind regards,
Christian
On Thu, Nov 3, 2022 at 8:13 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
> > With my patch the tests are still failing but the testsuite continues
> > to run and returns
> > with 0 as status code (one can see the failed test though).
>
> FYI, I am using automated scripts to run make test locally on multiple
> Emacs versions. 0 status code would do no good for my testing.
>
> >> Maybe we can instead provide some variable that can turn-on ignoring
> >> some problematic tests only during CI tests?
> > Yes ... that would also be possible e.g. skip the test or test nothing
> > if it's running on the ci.
> > (I am talking only about the 3 tests!).
>
> More like just 1 test with underscore :)
> The 3 tests in the patch are real failures because ob-python does not
> fully support python-mode.el.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Ignore flaky ob-python tests
2022-11-03 21:15 ` Christian Köstlin
@ 2022-11-04 4:19 ` Ihor Radchenko
0 siblings, 0 replies; 9+ messages in thread
From: Ihor Radchenko @ 2022-11-04 4:19 UTC (permalink / raw)
To: Christian Köstlin; +Cc: emacs-orgmode
Christian Köstlin <christian.koestlin@gmail.com> writes:
> I somehow seem to remember that the async tests were also broken with
> python.el (python-mode.el not installed).
> at least sometimes?
I only recall the underscore test.
Note that I debugged the async tests with your rake builds, and I have
identified that async tests are failing because of python-mode
inclusion. (python-mode is not supported by async ob-python code).
> or is this only the underscore test?
> If it's only the underscore test, I would suggest not installing
> python-mode for ci and ignoring the really flaky tests (i think the
> underscore one is a candidate here).
Yes. Please, go ahead and remove python-mode from the CI tests.
We are likely going to deprecate python-mode support. See
https://orgmode.org/list/87r0yk7bx8.fsf@localhost
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-11-04 4:18 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-02 0:41 [PATCH] Ignore flaky ob-python tests Christian Köstlin
2022-11-02 6:53 ` Ihor Radchenko
2022-11-02 11:15 ` Christian Köstlin
2022-11-02 13:55 ` Jack Kamm
2022-11-02 14:12 ` Ihor Radchenko
2022-11-02 14:17 ` Ihor Radchenko
2022-11-03 7:14 ` Ihor Radchenko
2022-11-03 21:15 ` Christian Köstlin
2022-11-04 4:19 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).