From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Schulte Subject: Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user Date: Wed, 29 Feb 2012 13:52:19 -0700 Message-ID: <87r4xd8irg.fsf@gmx.com> References: <87ipit5onv.fsf@Rainer.invalid> <87linlbokm.fsf@gmx.com> <87hay9z9mz.fsf@Rainer.invalid> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([208.118.235.92]:37802) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S2qVk-0005OS-48 for emacs-orgmode@gnu.org; Wed, 29 Feb 2012 15:52:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S2qVh-0001F8-P6 for emacs-orgmode@gnu.org; Wed, 29 Feb 2012 15:52:31 -0500 Received: from mailout-us.gmx.com ([74.208.5.67]:44321) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1S2qVh-0001EY-Hp for emacs-orgmode@gnu.org; Wed, 29 Feb 2012 15:52:29 -0500 In-Reply-To: <87hay9z9mz.fsf@Rainer.invalid> (Achim Gratz's message of "Wed, 29 Feb 2012 21:07:16 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Achim Gratz Cc: emacs-orgmode@gnu.org Achim Gratz writes: > Eric Schulte writes: >> Moving forward on this point, many of the existing tests explicitly >> `require' new Org-mode functionality (mainly language support for >> testing code blocks execution). I do not think that tests should ever >> be activating new packages changing a users global environment. > > Thanks. > > I've had to deactivate the inclusion of ./testing/lisp/contrib into the > test flow due to a recent removal of said directory by Nicolas (he's > probably doing that patch himself) and now get this: > Yes, I talked with Nicolas this morning, and at my urging he moved everything from ./testing/lisp/contrib into ./testing/lisp and removed the contrib directory. I think moving forward we are best served by a single test directory, especially given that we already have good methods of conditionally loading only those tests appropriate for the users environment. > > lisp/org-mode> make test-dirty |& grep failed > failed 5/109 ob-exp/export-from-a-temp-buffer > failed 17/109 org-missing-dependency/test-ob-C > failed 18/109 org-missing-dependency/test-ob-R > failed 19/109 org-missing-dependency/test-ob-awk > failed 20/109 org-missing-dependency/test-ob-fortran > failed 21/109 org-missing-dependency/test-ob-lilypond > failed 22/109 org-missing-dependency/test-ob-maxima > failed 23/109 org-missing-dependency/test-ob-octave > failed 24/109 org-missing-dependency/test-ob-python > failed 25/109 org-missing-dependency/test-org-element > failed 26/109 org-missing-dependency/test-org-export > > This is expected, as my local configuration doesn't have any of these > packages activated. > Yes, I assume that the lower case "failed" means expected failure. > > lisp/org-mode> make test-dirty | & grep FAILED > FAILED 27/109 test-ob-exp/org-babel-exp-src-blocks/w-no-file > FAILED 62/109 test-org-babel/get-src-block-info-body > FAILED 63/109 test-org-babel/get-src-block-info-language > FAILED 64/109 test-org-babel/get-src-block-info-tangle > > This is unexpected and I don't really see where they are coming from. > The backtrace always starts at (regexp-quote org-test-file-ob-anchor). > My guess is that when I removed require statements for org-html and org-ascii from from test-org-html and test-org-exp, they removed some functionality not loaded by default in the test scripts. These should probably be added into the Makefile. > >> For this reason I have just pushed up a commit which changes all >> >> (require 'org-foo) >> >> to >> >> (unless (featurep 'org-foo) >> (signal 'missing-test-dependency "Org support for doing foo.")) >> >> so that those tests simple aren't run on the users system. Please let >> me know if anyone thinks this is a mistake and we can discuss. The only >> drawback I see is that batch-mode scripts will have to explicitly >> activate the features which they would like to test, by evaluating forms >> like (require 'org-foo) before the call to the test suite -- and I would >> argue that being explicit about such things is a benefit. > > Well, currently all testing runs under my user settings, which is not > ideal for testing anyway. I suppose it would not be difficult to just > inject a different (minimal) startup file specifically for testing to > separate these issues. > Yea, I also run most tests from my normal Emacs session, which is part of the motivation for removing these embedded require statements. Cheers, > > > Regards, > Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/