Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz <Stromeko@nexgo.de>]

Eric Schulte <eric.schulte@gmx.com> writes:
> Moving forward on this point, many of the existing tests explicitly
> `require' new Org-mode functionality (mainly language support for
> testing code blocks execution).  I do not think that tests should ever
> be activating new packages changing a users global environment.

Thanks.

I've had to deactivate the inclusion of ./testing/lisp/contrib into the
test flow due to a recent removal of said directory by Nicolas (he's
probably doing that patch himself) and now get this:

lisp/org-mode> make test-dirty |& grep failed
   failed    5/109  ob-exp/export-from-a-temp-buffer
   failed   17/109  org-missing-dependency/test-ob-C
   failed   18/109  org-missing-dependency/test-ob-R
   failed   19/109  org-missing-dependency/test-ob-awk
   failed   20/109  org-missing-dependency/test-ob-fortran
   failed   21/109  org-missing-dependency/test-ob-lilypond
   failed   22/109  org-missing-dependency/test-ob-maxima
   failed   23/109  org-missing-dependency/test-ob-octave
   failed   24/109  org-missing-dependency/test-ob-python
   failed   25/109  org-missing-dependency/test-org-element
   failed   26/109  org-missing-dependency/test-org-export

This is expected, as my local configuration doesn't have any of these
packages activated.

lisp/org-mode> make test-dirty | & grep FAILED
   FAILED   27/109  test-ob-exp/org-babel-exp-src-blocks/w-no-file
   FAILED   62/109  test-org-babel/get-src-block-info-body
   FAILED   63/109  test-org-babel/get-src-block-info-language
   FAILED   64/109  test-org-babel/get-src-block-info-tangle

This is unexpected and I don't really see where they are coming from.
The backtrace always starts at (regexp-quote org-test-file-ob-anchor).

> For this reason I have just pushed up a commit which changes all
>
>   (require 'org-foo)
>
> to
>
>   (unless (featurep 'org-foo)
>     (signal 'missing-test-dependency "Org support for doing foo."))
>
> so that those tests simple aren't run on the users system.  Please let
> me know if anyone thinks this is a mistake and we can discuss.  The only
> drawback I see is that batch-mode scripts will have to explicitly
> activate the features which they would like to test, by evaluating forms
> like (require 'org-foo) before the call to the test suite -- and I would
> argue that being explicit about such things is a benefit.

Well, currently all testing runs under my user settings, which is not
ideal for testing anyway.  I suppose it would not be difficult to just
inject a different (minimal) startup file specifically for testing to
separate these issues.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada