From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Schulte Subject: Re: [test] Mark tests with missing dependencies as "expected to fail" Date: Sun, 13 Nov 2011 12:31:20 -0700 Message-ID: <8762inzunr.fsf@gmail.com> References: <87fwiqnclu.wl%dmaus@ictsoc.de> <87fwiq2rsl.fsf@gmail.com> <87zkg0ov13.wl%dmaus@ictsoc.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([140.186.70.92]:51893) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RPfm4-0001S4-Iu for emacs-orgmode@gnu.org; Sun, 13 Nov 2011 14:31:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RPfm1-0003xc-24 for emacs-orgmode@gnu.org; Sun, 13 Nov 2011 14:31:28 -0500 Received: from mail-pz0-f47.google.com ([209.85.210.47]:60595) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RPfm0-0003x6-PK for emacs-orgmode@gnu.org; Sun, 13 Nov 2011 14:31:25 -0500 Received: by pzk1 with SMTP id 1so8707569pzk.6 for ; Sun, 13 Nov 2011 11:31:23 -0800 (PST) In-Reply-To: <87zkg0ov13.wl%dmaus@ictsoc.de> (David Maus's message of "Sun, 13 Nov 2011 17:18:48 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: David Maus Cc: emacs-orgmode@gnu.org --=-=-= Content-Type: text/plain David Maus writes: > Hi Eric, > > At Tue, 18 Oct 2011 10:22:34 -0600, > Eric Schulte wrote: >> Hi David, >> >> I agree it would be preferable to note that not all tests are run when >> dependencies are missing, although I don't think it is extremely >> important. I think some version of the above would be worthwhile if it >> could be done in a file-wide manner (as are the current dependency >> checks) and wouldn't require duplicating the dependency check or >> changing every test form individually. Perhaps a file-local-variable >> could be used to expect failures for every form defined in the file? > > I tried the approach with a file-local variable but it didn't work > out: A macro can be expanded at any time, i.e. looks like there is no > way to obtain a reference to the buffer where the macro is defined at > expansion time. > > But finally came up with this one: > Nice macro, The only downside I see is the requirement to wrap every single deftest form which (to me) is too much overhead for too little payoff. How about the following which will register a failing test for each file of tests not loaded due to missing dependencies. Best -- Eric --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=0001-Indicate-tests-with-missing-dependencies-by-adding-a.patch >From a84259bba32a69ae0b54dc577f1ba19b23625714 Mon Sep 17 00:00:00 2001 From: Eric Schulte Date: Sun, 13 Nov 2011 12:30:25 -0700 Subject: [PATCH] Indicate tests with missing dependencies by adding a expected failing test * testing/lisp/test-ob-R.el (featurep): Signal missing dependencies with an error rather than a throw. * testing/org-test.el (missing-test-dependency): Define the error signal for missing dependencies. (org-test-for-executable): Signal missing dependencies with an error rather than a throw. (org-test-load): Define an expected failing test for each file with missing dependencies. --- testing/lisp/test-ob-R.el | 2 +- testing/org-test.el | 19 +++++++++++++++---- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el index bc637ff..bb9783f 100644 --- a/testing/lisp/test-ob-R.el +++ b/testing/lisp/test-ob-R.el @@ -8,7 +8,7 @@ (org-test-for-executable "R") (unless (featurep 'ess) - (throw 'missing-test-dependency "ESS")) + (signal 'missing-test-dependency "ESS")) (let ((load-path (cons (expand-file-name ".." (file-name-directory diff --git a/testing/org-test.el b/testing/org-test.el index 7d2f7e7..0f9cf1a 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -102,6 +102,10 @@ org-test searches this directory up the directory tree.") ;;; Functions for writing tests +(put 'missing-test-dependency + 'error-conditions + '(error missing-test-dependency)) + (defun org-test-for-executable (exe) "Throw an error if EXE is not available. This can be used at the top of code-block-language specific test @@ -111,7 +115,7 @@ executable." (lambda (acc dir) (or acc (file-exists-p (expand-file-name exe dir)))) exec-path :initial-value nil) - (throw 'missing-test-dependency exe))) + (signal 'missing-test-dependency (list exe)))) (defun org-test-buffer (&optional file) "TODO: Setup and return a buffer to work with. @@ -275,10 +279,17 @@ otherwise place the point at the beginning of the inserted text." (lambda (path) (if (file-directory-p path) (rld path) - (catch 'missing-test-dependency - (when (string-match "^[A-Za-z].*\\.el$" + (condition-case err + (when (string-match "^[A-Za-z].*\\.el$" (file-name-nondirectory path)) - (load-file path))))) + (load-file path)) + (missing-test-dependency + (let ((name (intern + (concat "org-missing-dependency/" + (file-name-nondirectory + (file-name-sans-extension path)))))) + (eval `(ert-deftest ,name () + :expected-result :failed (should nil)))))))) (directory-files base 'full "^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$")))) (rld (expand-file-name "lisp" org-test-dir)) -- 1.7.4.1 --=-=-= Content-Type: text/plain > > #+begin_src emacs-lisp > (defmacro org-test-with-dependencies (dependencies &rest body) > "Mark `ert-deftest' forms in BODY with a expected result > depending on DEPENDENCIES. DEPENDENCIES is an alist with a > human-readable name of the dependency as key. The second element > of each member should be a form that evaluates to a non-nil value > if the dependency is met. > > All `ert-deftest' forms in BODY are marked as expected to pass if > all dependencies are met. Otherwise the expected result is set to > `:failed' and the test's bodies modified to signal an error with > an error message indicating the first failing dependency." > (macrolet ((define-dependencies () > `(cond > ,@(mapcar (lambda (dependency) > `((not ,(second dependency)) ,(first dependency))) dependencies)))) > (let ((missing-dependency (define-dependencies))) > `(progn > ,@(mapcar (lambda (sexp) > (if (and (consp sexp) > (eq (first sexp) 'ert-deftest)) > (let* ((docstring (if (stringp (fourth sexp)) (list (fourth sexp)))) > (deftest-body (nthcdr (if docstring 4 3) sexp))) > `(,@(append (list (first sexp) (second sexp) (third sexp)) docstring) > :expected-result ,@(if missing-dependency > `(:failed (error "Missing dependency: %s" ,missing-dependency)) > '(:passed)) > ,@deftest-body)) > sexp)) body))))) > #+end_src > > You wrap it around ert-deftest forms, e.g. > > #+begin_src emacs-lisp > (org-test-with-dependencies (("ESS" (featurep 'ess))) > (ert-deftest test-ob-R/simple-session () > (org-test-with-temp-text > "#+begin_src R :session R\n paste(\"Yep!\")\n#+end_src\n" > (should (string= "Yep!" (org-babel-execute-src-block)))))) > #+end_src > > #+begin_quote > ELISP> (macroexpand '(org-test-with-dependencies (("ESS" (featurep 'ess))) (ert-deftest foo () t))) > (progn > (ert-deftest foo nil :expected-result :failed > (error "Missing dependency: %s" "ESS") > t)) > > ELISP> > #end_quote > > If this is acceptable I'd push it and start to adjust the test > definitions. > > Best, > -- David > -- > OpenPGP... 0x99ADB83B5A4478E6 > Jabber.... dmjena@jabber.org > Email..... dmaus@ictsoc.de -- Eric Schulte http://cs.unm.edu/~eschulte/ --=-=-=--