[bug] regression tests broken

[bug] regression tests broken

[Martyn Jago]

Hi 

All automated regression tests broken since 7720786 commit

http://martynjago.com:3333

Best, Martyn

Re: [bug] regression tests broken

[Eric Schulte]

Hi Martyn,

I have all tests passing in a branch in the Org-mode git repo (which I
plan to merge into master soon).  Could you try a run of this on the
standard-code-block-syntax branch?

As a side note, how automated is your crusecontrol.rb setup?  Would you
recommend this as a way to run automated tests on the Org-mode server
after every git commit?

Thanks -- Eric

Martyn Jago <martyn.jago@btinternet.com> writes:

> Hi 
>
> All automated regression tests broken since 7720786 commit
>
> http://martynjago.com:3333
>
> Best, Martyn
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [bug] regression tests broken

[Martyn Jago]

Hi Eric

Eric Schulte <schulte.eric@gmail.com> writes:

> Hi Martyn,
>
> I have all tests passing in a branch in the Org-mode git repo (which I
> plan to merge into master soon).  Could you try a run of this on the
> standard-code-block-syntax branch?

I've added the standard-code-block-syntax branch to my test server, and
have configured notification so that you will be informed via email on
failure. Unfortunately, the branch is currently failing, as are all
other builds including my test-branch. I will try to take a look
tomorrow on why three tests are failing.

> As a side note, how automated is your crusecontrol.rb setup?  Would you
> recommend this as a way to run automated tests on the Org-mode server
> after every git commit?

It is completely autonomous, git-pulling, cleaning, building, building
docs, and running tests within 10 minutes or so of a fresh commit to the
Org repository. Builds are serialized.

I would certainly recommend it as an easy way of setting up a test
server, since everything you need is built-in, such as email
notification (notifying the list is of course a possibility), RSS, and
rapid backtrace investigation etc. 

I have come across other gnu projects that use cruisecontrolrb.

Your server would need Ruby installed (I use v1.9.2), but installation
should be straight-forward since I have already written the scripts,
which is fairly simple anyway.

Hope that helps

Best Martyn

> Thanks -- Eric
>
> Martyn Jago <martyn.jago@btinternet.com> writes:
>
>> Hi 
>>
>> All automated regression tests broken since 7720786 commit
>>
>> http://martynjago.com:3333
>>
>> Best, Martyn
>>
>>

Re: [bug] regression tests broken

[Martyn Jago]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

Martyn Jago <martyn.jago@btinternet.com> writes:
Hi Eric 

[...]

Further to the problem of tests failing under sandboxed conditions
unexpectedly, I've made a few observations which I want to pass on.

* ID

Currently the ID searching command `org-id-find' has global scope to
find ID's all over the host system. Since developers frequently have > 1
Orgs on their system, this can lead to confusion.

For instance, the test `ob-C/table' in in test-ob-C.el appears to
contain an orphan ID, which fails, but may well pass on the devs system,
since the ID probably exists somewhere (does this test also require to
be called test-ob-C/table)?.

Unfortunately `org-id-find' seems to be heavily dependent on org-mode.el
forms, and I suspect it would require a lot of time for me to attempt to
restrict it's scope to `the current Org system in use, so I will leave
it to someone with more experience.

Another by-product is that sandboxed test systems are not guaranteed to
pick up the _correct_ file containing the search ID (this plagued me
this-morning).

* `org-test-load' attempts to load symlinks (Emacs interlocking files).

Currently `org-test-load' will happily attempt to load an interlocking
symbolic link (such as .#test-ob-exp.el ->
martyn@88-96-171-138.59787). This results in a failure to run any tests,
and is particularly annoying during test development. Below is a patch
that fixes this problem for me.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Avoid-loading-and-failing-symbolic-links-interlockin.patch --]
[-- Type: text/x-patch, Size: 1180 bytes --]

From 04dd7358295caef0d42a3fe93f4979ccd3b5c630 Mon Sep 17 00:00:00 2001
From: Martyn Jago <martyn.jago@btinternet.com>
Date: Sun, 13 Nov 2011 14:43:34 +0000
Subject: [PATCH] Avoid loading (and failing) symbolic links (interlocking files)
 * testing/org-test.el: During test development various interlocking
 files may be present in testing/lisp directory (since they are being
 edited by emacs). Currently org-test-load will attempt to load these
 and fail.

---
 testing/org-test.el |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/testing/org-test.el b/testing/org-test.el
index 57b7252..ea8cae4 100644
--- a/testing/org-test.el
+++ b/testing/org-test.el
@@ -274,9 +274,10 @@ otherwise place the point at the beginning of the inserted text."
 	      (mapc
 	       (lambda (path)
 		 (if (file-directory-p path)
-		   (rld path)
+		     (rld path)
 		   (catch 'missing-test-dependency
-		     (load-file path))))
+		     (when (string-match "^[A-Za-z].*\\.el$" path)
+		       load-file path))))
 	       (directory-files base 'full
 				"^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$"))))
     (rld (expand-file-name "lisp" org-test-dir))
-- 
1.7.3.4


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]


I'm still looking at a potential variable passing problem during html
export by inline calls, and will submit something as soon as I can.

Best, Martyn

Re: [bug] regression tests broken

[Martyn Jago]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

Hi Eric

Oops, the patch was broken - this one fixes the problem...

[...]

> Currently `org-test-load' will happily attempt to load an interlocking
> symbolic link (such as .#test-ob-exp.el ->
> martyn@88-96-171-138.59787). This results in a failure to run any tests,
> and is particularly annoying during test development. Below is a patch
> that fixes this problem for me.
>


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Fix-don-t-load-symlinks-Emacs-interlocking-files --]
[-- Type: text/x-patch, Size: 1191 bytes --]

From b868e208dd3ece7c10818ed73f2cf0dc86dac94b Mon Sep 17 00:00:00 2001
From: Martyn Jago <martyn.jago@btinternet.com>
Date: Sun, 13 Nov 2011 16:03:51 +0000
Subject: [PATCH] Fix don't load symlinks (Emacs interlocking files)
 * testing/org-test.el: During test development various interlocking
  files may be present in testing/lisp directory (since they are being
  edited by emacs). Currently org-test-load will attempt to load these

---
 testing/org-test.el |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/testing/org-test.el b/testing/org-test.el
index ea8cae4..7d2f7e7 100644
--- a/testing/org-test.el
+++ b/testing/org-test.el
@@ -276,8 +276,9 @@ otherwise place the point at the beginning of the inserted text."
 		 (if (file-directory-p path)
 		     (rld path)
 		   (catch 'missing-test-dependency
-		     (when (string-match "^[A-Za-z].*\\.el$" path)
-		       load-file path))))
+		     (when (string-match "^[A-Za-z].*\\.el$"
+					 (file-name-nondirectory path))
+		       (load-file path)))))
 	       (directory-files base 'full
 				"^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$"))))
     (rld (expand-file-name "lisp" org-test-dir))
-- 
1.7.3.4


[-- Attachment #3: Type: text/plain, Size: 22 bytes --]


Best, Martyn

[...]

Re: [bug] regression tests broken

[Eric Schulte]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Martyn Jago <martyn.jago@btinternet.com> writes:
> Hi Eric 
>
> [...]
>
> Further to the problem of tests failing under sandboxed conditions
> unexpectedly, I've made a few observations which I want to pass on.
>
> * ID
>
> Currently the ID searching command `org-id-find' has global scope to
> find ID's all over the host system. Since developers frequently have > 1
> Orgs on their system, this can lead to confusion.
>

Customizing the `org-id-locations-file' can be used to isolate ID lists
between different instances of Emacs.  Only the files in the
`org-id-files' list should be scanned for IDs, so it shouldn't be too
difficult to limit the scope of those files used in testing to only the
example files in the testing directory.

>
> For instance, the test `ob-C/table' in in test-ob-C.el appears to
> contain an orphan ID, which fails,

Confirmed, I don't have this ID located on my system either.  Perhaps
the author of this test file can help here.  Note that this test is
marked as expected to fail, so it should not affect the test suite as a
whole.

> but may well pass on the devs system, since the ID probably exists
> somewhere (does this test also require to be called test-ob-C/table)?.
>

No, the existing name is fine, see the ERT test match string in
`org-test-run-all-tests' to see which tests will be picked up by the
Org-mode test suite..

>
> Unfortunately `org-id-find' seems to be heavily dependent on org-mode.el
> forms, and I suspect it would require a lot of time for me to attempt to
> restrict it's scope to `the current Org system in use, so I will leave
> it to someone with more experience.
>
> Another by-product is that sandboxed test systems are not guaranteed to
> pick up the _correct_ file containing the search ID (this plagued me
> this-morning).
>

Given that all of these IDs are long hash tags I doubt there is any risk
of ID duplication unless there are multiple checkouts of Org-mode being
included in the files Org-mode scans for IDs.

>
> * `org-test-load' attempts to load symlinks (Emacs interlocking
>files).
>
> Currently `org-test-load' will happily attempt to load an interlocking
> symbolic link (such as .#test-ob-exp.el ->
> martyn@88-96-171-138.59787). This results in a failure to run any tests,
> and is particularly annoying during test development. Below is a patch
> that fixes this problem for me.
>

Great, I've just applied this patch.

Thanks -- Eric

>
> From 04dd7358295caef0d42a3fe93f4979ccd3b5c630 Mon Sep 17 00:00:00 2001
> From: Martyn Jago <martyn.jago@btinternet.com>
> Date: Sun, 13 Nov 2011 14:43:34 +0000
> Subject: [PATCH] Avoid loading (and failing) symbolic links (interlocking files)
>  * testing/org-test.el: During test development various interlocking
>  files may be present in testing/lisp directory (since they are being
>  edited by emacs). Currently org-test-load will attempt to load these
>  and fail.
>
> ---
>  testing/org-test.el |    5 +++--
>  1 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/testing/org-test.el b/testing/org-test.el
> index 57b7252..ea8cae4 100644
> --- a/testing/org-test.el
> +++ b/testing/org-test.el
> @@ -274,9 +274,10 @@ otherwise place the point at the beginning of the inserted text."
>  	      (mapc
>  	       (lambda (path)
>  		 (if (file-directory-p path)
> -		   (rld path)
> +		     (rld path)
>  		   (catch 'missing-test-dependency
> -		     (load-file path))))
> +		     (when (string-match "^[A-Za-z].*\\.el$" path)
> +		       load-file path))))
>  	       (directory-files base 'full
>  				"^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$"))))
>      (rld (expand-file-name "lisp" org-test-dir))

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [bug] regression tests broken

[Eric Schulte]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Hi Eric
>
> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Hi Martyn,
>>
>> I have all tests passing in a branch in the Org-mode git repo (which I
>> plan to merge into master soon).  Could you try a run of this on the
>> standard-code-block-syntax branch?
>
> I've added the standard-code-block-syntax branch to my test server, and
> have configured notification so that you will be informed via email on
> failure. Unfortunately, the branch is currently failing, as are all
> other builds including my test-branch. I will try to take a look
> tomorrow on why three tests are failing.
>

I had to take special care that previous versions of Org-mode weren't
loaded when running tests on this branch.  The problem being that if
some defvar forms are defined by a previous instillation of Org-mode
their new values may not be overwritten with the latest versions in this
branch.

From looking at your test output it seems the only two tests which are
failing are the most recent two related to nested code blocks.  Again
these are passing on my system so I'm not sure what the cause could be.

>
>> As a side note, how automated is your crusecontrol.rb setup?  Would you
>> recommend this as a way to run automated tests on the Org-mode server
>> after every git commit?
>
> It is completely autonomous, git-pulling, cleaning, building, building
> docs, and running tests within 10 minutes or so of a fresh commit to the
> Org repository. Builds are serialized.
>
> I would certainly recommend it as an easy way of setting up a test
> server, since everything you need is built-in, such as email
> notification (notifying the list is of course a possibility), RSS, and
> rapid backtrace investigation etc. 
>
> I have come across other gnu projects that use cruisecontrolrb.
>
> Your server would need Ruby installed (I use v1.9.2), but installation
> should be straight-forward since I have already written the scripts,
> which is fairly simple anyway.
>
> Hope that helps
>

This sounds like a very nice setup.  It would be great if you can work
with me and Jason to see if we can set this up on the Org-mode server.

Thanks -- Eric

>
> Best Martyn
>
>> Thanks -- Eric
>>
>> Martyn Jago <martyn.jago@btinternet.com> writes:
>>
>>> Hi 
>>>
>>> All automated regression tests broken since 7720786 commit
>>>
>>> http://martynjago.com:3333
>>>
>>> Best, Martyn
>>>
>>>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [bug] regression tests broken

[Martyn Jago]

Hi Eric

Eric Schulte <schulte.eric@gmail.com> writes:


[...]

> This sounds like a very nice setup.  It would be great if you can work
> with me and Jason to see if we can set this up on the Org-mode server.

No problem. If it is a Linux server, I can work on setting up a fresh
instance on Linux prior the actual deployment (currently on OSX).

Best, Martyn

> Thanks -- Eric
>
>>
>> Best Martyn
>>
>>> Thanks -- Eric
>>>
>>> Martyn Jago <martyn.jago@btinternet.com> writes:
>>>
>>>> Hi 
>>>>
>>>> All automated regression tests broken since 7720786 commit
>>>>
>>>> http://martynjago.com:3333
>>>>
>>>> Best, Martyn
>>>>
>>>>
>>
>>