emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [HELP] Request for patches to improve Org test coverage
@ 2021-11-24 13:11 Ihor Radchenko
  2021-11-24 14:27 ` Max Nikulin
  2021-11-24 14:36 ` Daniele Pizzolli
  0 siblings, 2 replies; 5+ messages in thread
From: Ihor Radchenko @ 2021-11-24 13:11 UTC (permalink / raw)
  To: emacs-orgmode

Dear Fellow Orgers,

Despite being not directly visible to users, Org mode testing suite is
critical to maintain overall codebase stability. It helps to avoid
unintentional breakages as we keep adding new features to Org.

Yet, writing tests is probably not the most exciting part for
volunteer contributors. Not many new patches also include tests and the
core Org maintainers do not have a bandwidth to supply every new
significant patch with tests.

I am writing this help request in a hope that someone interested and/or
experienced with writing tests can help improving Org tests without a
need to spend a lot of time doing something unfamiliar (e.g. I
personally wrote my very first test as a patch for Org, not knowing much
about code testing beforehand).

Some of the possible improvements to Org test suite are listed below:

- Tests on bugfix branch fail to run using Emacs 26.
  Not because of main Org code, but because test-org re-defines
  decode-time in a way not supported by older Emacs versions. (This has
  been fixed on main and simply need to be ported back to bugfix).

- Tests always run in a single process and cannot benefit from make -j
  Supporting parallel execution of multiple test sets would make Org
  development much faster by reducing make test runtime.

- Testing native-compiled Org is missing. Having it would be nice.

- Tests covering state logging are missing. I wrote an example patch how
  to write such tests, but never got around for more
  https://orgmode.org/list/87tul1v11c.fsf@localhost

- Most of the tests are written assuming default values of customised
  variables. We have very basic test coverage for non-standard settings.
  A way to run all possible tests under most common user customisations
  would improve the test coverage a lot.

- Generally, more and better tests would be welcome.

Best,
Ihor


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [HELP] Request for patches to improve Org test coverage
  2021-11-24 13:11 [HELP] Request for patches to improve Org test coverage Ihor Radchenko
@ 2021-11-24 14:27 ` Max Nikulin
  2021-11-25 12:13   ` Ihor Radchenko
  2021-11-24 14:36 ` Daniele Pizzolli
  1 sibling, 1 reply; 5+ messages in thread
From: Max Nikulin @ 2021-11-24 14:27 UTC (permalink / raw)
  To: emacs-orgmode

On 24/11/2021 20:11, Ihor Radchenko wrote:
> 
> - Generally, more and better tests would be welcome.

There are some tests (test-org/org-sort-remove-invisible) lost in a 
lengthy thread
https://list.orgmode.org/orgmode/s5p88r$go9$1@ciao.gmane.io/
There is even updates.orgmode.org entry for them: [Patch] tests for 
org-remove-invisible

org-protocol tests are not run by "make test" and require additional 
variables to be set. Maybe it is reasonable to use separate emacs 
process due to some specific requirements as emacs-server.



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [HELP] Request for patches to improve Org test coverage
  2021-11-24 13:11 [HELP] Request for patches to improve Org test coverage Ihor Radchenko
  2021-11-24 14:27 ` Max Nikulin
@ 2021-11-24 14:36 ` Daniele Pizzolli
  2021-11-25 12:32   ` Ihor Radchenko
  1 sibling, 1 reply; 5+ messages in thread
From: Daniele Pizzolli @ 2021-11-24 14:36 UTC (permalink / raw)
  To: emacs-orgmode


Ihor Radchenko writes:

> Dear Fellow Orgers,
>
> Despite being not directly visible to users, Org mode testing suite is
> critical to maintain overall codebase stability. It helps to avoid
> unintentional breakages as we keep adding new features to Org.

[]

> - Generally, more and better tests would be welcome.

Hello,

Thanks for calling for tests improvement with such rationale.

I submitted my small test patch for a bug in babel/shell early this year
(unfortunately I was not able to fix the bug). I still think that the
test is valuable for inclusion.  If not, please give feedback.

See the original mail at: <87h7jhus7k.fsf@spx.local.examples>

Best,
Daniele


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [HELP] Request for patches to improve Org test coverage
  2021-11-24 14:27 ` Max Nikulin
@ 2021-11-25 12:13   ` Ihor Radchenko
  0 siblings, 0 replies; 5+ messages in thread
From: Ihor Radchenko @ 2021-11-25 12:13 UTC (permalink / raw)
  To: Max Nikulin; +Cc: emacs-orgmode

Max Nikulin <manikulin@gmail.com> writes:

> On 24/11/2021 20:11, Ihor Radchenko wrote:
>> 
>> - Generally, more and better tests would be welcome.
>
> There are some tests (test-org/org-sort-remove-invisible) lost in a 
> lengthy thread
> https://list.orgmode.org/orgmode/s5p88r$go9$1@ciao.gmane.io/
> There is even updates.orgmode.org entry for them: [Patch] tests for 
> org-remove-invisible

I replied in the thread.

> org-protocol tests are not run by "make test" and require additional 
> variables to be set. Maybe it is reasonable to use separate emacs 
> process due to some specific requirements as emacs-server.

Another good direction of improvement.

Running different test files in the same process is generally not a good
idea. It may cause unintentional interactions and prevent parallel make.

Best,
Ihor


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [HELP] Request for patches to improve Org test coverage
  2021-11-24 14:36 ` Daniele Pizzolli
@ 2021-11-25 12:32   ` Ihor Radchenko
  0 siblings, 0 replies; 5+ messages in thread
From: Ihor Radchenko @ 2021-11-25 12:32 UTC (permalink / raw)
  To: Daniele Pizzolli; +Cc: emacs-orgmode

Daniele Pizzolli <dan@toel.it> writes:

> I submitted my small test patch for a bug in babel/shell early this year
> (unfortunately I was not able to fix the bug). I still think that the
> test is valuable for inclusion.  If not, please give feedback.
>
> See the original mail at: <87h7jhus7k.fsf@spx.local.examples>

Your patch is very much appreciated :) However it is more than 15LOC.
May I know if you have the FSF copyright assignment? You are not listed
as a contributor at https://orgmode.org/worg/contributors.html

Best,
Ihor



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-11-25 12:31 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-24 13:11 [HELP] Request for patches to improve Org test coverage Ihor Radchenko
2021-11-24 14:27 ` Max Nikulin
2021-11-25 12:13   ` Ihor Radchenko
2021-11-24 14:36 ` Daniele Pizzolli
2021-11-25 12:32   ` Ihor Radchenko

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).