Re: [bug] regression tests broken

[Eric Schulte <schulte.eric@gmail.com>]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Martyn Jago <martyn.jago@btinternet.com> writes:
> Hi Eric 
>
> [...]
>
> Further to the problem of tests failing under sandboxed conditions
> unexpectedly, I've made a few observations which I want to pass on.
>
> * ID
>
> Currently the ID searching command `org-id-find' has global scope to
> find ID's all over the host system. Since developers frequently have > 1
> Orgs on their system, this can lead to confusion.
>

Customizing the `org-id-locations-file' can be used to isolate ID lists
between different instances of Emacs.  Only the files in the
`org-id-files' list should be scanned for IDs, so it shouldn't be too
difficult to limit the scope of those files used in testing to only the
example files in the testing directory.

>
> For instance, the test `ob-C/table' in in test-ob-C.el appears to
> contain an orphan ID, which fails,

Confirmed, I don't have this ID located on my system either.  Perhaps
the author of this test file can help here.  Note that this test is
marked as expected to fail, so it should not affect the test suite as a
whole.

> but may well pass on the devs system, since the ID probably exists
> somewhere (does this test also require to be called test-ob-C/table)?.
>

No, the existing name is fine, see the ERT test match string in
`org-test-run-all-tests' to see which tests will be picked up by the
Org-mode test suite..

>
> Unfortunately `org-id-find' seems to be heavily dependent on org-mode.el
> forms, and I suspect it would require a lot of time for me to attempt to
> restrict it's scope to `the current Org system in use, so I will leave
> it to someone with more experience.
>
> Another by-product is that sandboxed test systems are not guaranteed to
> pick up the _correct_ file containing the search ID (this plagued me
> this-morning).
>

Given that all of these IDs are long hash tags I doubt there is any risk
of ID duplication unless there are multiple checkouts of Org-mode being
included in the files Org-mode scans for IDs.

>
> * `org-test-load' attempts to load symlinks (Emacs interlocking
>files).
>
> Currently `org-test-load' will happily attempt to load an interlocking
> symbolic link (such as .#test-ob-exp.el ->
> martyn@88-96-171-138.59787). This results in a failure to run any tests,
> and is particularly annoying during test development. Below is a patch
> that fixes this problem for me.
>

Great, I've just applied this patch.

Thanks -- Eric

>
> From 04dd7358295caef0d42a3fe93f4979ccd3b5c630 Mon Sep 17 00:00:00 2001
> From: Martyn Jago <martyn.jago@btinternet.com>
> Date: Sun, 13 Nov 2011 14:43:34 +0000
> Subject: [PATCH] Avoid loading (and failing) symbolic links (interlocking files)
>  * testing/org-test.el: During test development various interlocking
>  files may be present in testing/lisp directory (since they are being
>  edited by emacs). Currently org-test-load will attempt to load these
>  and fail.
>
> ---
>  testing/org-test.el |    5 +++--
>  1 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/testing/org-test.el b/testing/org-test.el
> index 57b7252..ea8cae4 100644
> --- a/testing/org-test.el
> +++ b/testing/org-test.el
> @@ -274,9 +274,10 @@ otherwise place the point at the beginning of the inserted text."
>  	      (mapc
>  	       (lambda (path)
>  		 (if (file-directory-p path)
> -		   (rld path)
> +		     (rld path)
>  		   (catch 'missing-test-dependency
> -		     (load-file path))))
> +		     (when (string-match "^[A-Za-z].*\\.el$" path)
> +		       load-file path))))
>  	       (directory-files base 'full
>  				"^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$"))))
>      (rld (expand-file-name "lisp" org-test-dir))

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/