FYI: Org mode testing framework, Emacs 23 and 22

FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Just FYI: I've tried to get the testing framework running on Emacs 23
and Emacs 22 and succeeded to 50%:

 - copying ert.el and ert-x.el from Emacs24 Git repository at
   git://git.savannah.gnu.org/emacs.git with master on
   231bffa3e6c37164fa40ad18bd27249cc7704e30 and installing jump.el is
   sufficient to get the test suit running again

 - not so much luck with Emacs22: Recent ERT requires `special-mode',
   defined in simple.el from 23.x upwards.

Emacs 22 version used was: GNU Emacs 22.3.2 (i686-pc-linux-gnu) of 2011-05-28 on x60s

Not having the test suit running with Emacs22 is a not-so-good because
Org mode claims compatibility with Emacs22:

,---- [ http://orgmode.org/]
| This package works on GNU Emacs 24, 23 and 22, and (with minor
| restrictions) on XEmacs 21. Emacs 22.2 ships with Org-mode version
| 4.67d, Emacs 23.1 with 6.21b. The latest Emacs development version
| usually contains a fairly recent version, but may lag a bit behind the
| website release.
`----

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

Hi David,

David Maus <dmaus@ictsoc.de> writes:

> Just FYI: I've tried to get the testing framework running on Emacs 23
> and Emacs 22 and succeeded to 50%:
>
>  - copying ert.el and ert-x.el from Emacs24 Git repository at
>    git://git.savannah.gnu.org/emacs.git with master on
>    231bffa3e6c37164fa40ad18bd27249cc7704e30 and installing jump.el is
>    sufficient to get the test suit running again
>
>  - not so much luck with Emacs22: Recent ERT requires `special-mode',
>    defined in simple.el from 23.x upwards.
>

Would it be possible to load a more recent version of simple.el in
Emacs22 to define special mode, and then see if ert loads successfully?

Also, did all tests pass successfully in Emacs23?

Currently on my system using Emacs24 all tests pass but those related to
fortran code block support, all of which are failing.

Once we have all tests running for Emacs>=22 it would be very nice (and
not too difficult) to add a post-update hook to the Org-mode git
repository which could run the test suite and record the results for
every version of Emacs with something like the following

emacs23 -Q --batch -l $ORGTESTSTUFF --eval "
(progn (org-test-run-all-tests)
       (htmlize-buffer)
       (write-file "commit-id-23.html"))"

Best -- Eric

>
> Emacs 22 version used was: GNU Emacs 22.3.2 (i686-pc-linux-gnu) of 2011-05-28 on x60s
>
> Not having the test suit running with Emacs22 is a not-so-good because
> Org mode claims compatibility with Emacs22:
>
> ,---- [ http://orgmode.org/]
> | This package works on GNU Emacs 24, 23 and 22, and (with minor
> | restrictions) on XEmacs 21. Emacs 22.2 ships with Org-mode version
> | 4.67d, Emacs 23.1 with 6.21b. The latest Emacs development version
> | usually contains a fairly recent version, but may lag a bit behind the
> | website release.
> `----
>
> Best,
>   -- David
> --
> OpenPGP... 0x99ADB83B5A4478E6
> Jabber.... dmjena@jabber.org
> Email..... dmaus@ictsoc.de

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

At Fri, 02 Sep 2011 07:50:02 -0600,
Eric Schulte wrote:
>
> Hi David,
>
> David Maus <dmaus@ictsoc.de> writes:
>
> > Just FYI: I've tried to get the testing framework running on Emacs 23
> > and Emacs 22 and succeeded to 50%:
> >
> >  - copying ert.el and ert-x.el from Emacs24 Git repository at
> >    git://git.savannah.gnu.org/emacs.git with master on
> >    231bffa3e6c37164fa40ad18bd27249cc7704e30 and installing jump.el is
> >    sufficient to get the test suit running again
> >
> >  - not so much luck with Emacs22: Recent ERT requires `special-mode',
> >    defined in simple.el from 23.x upwards.
> >
>
> Would it be possible to load a more recent version of simple.el in
> Emacs22 to define special mode, and then see if ert loads successfully?

Next on my list.

>
> Also, did all tests pass successfully in Emacs23?
>
> Currently on my system using Emacs24 all tests pass but those related to
> fortran code block support, all of which are failing.

The test result with failing tests for Emacs23 is attached.

,----
| Passed: 101
| Failed: 8 (8 unexpected)
| Total:  109/109
|
| Started at:   2011-09-11 19:53:46+0200
| Finished.
| Finished at:  2011-09-11 19:53:51+0200
`----

7 of the 8 are fortran-tests, the 8th is
`org-babel-get-inline-src-block-matches'.

>
> Once we have all tests running for Emacs>=22 it would be very nice (and
> not too difficult) to add a post-update hook to the Org-mode git
> repository which could run the test suite and record the results for
> every version of Emacs with something like the following
>
> emacs23 -Q --batch -l $ORGTESTSTUFF --eval "
> (progn (org-test-run-all-tests)
>        (htmlize-buffer)
>        (write-file "commit-id-23.html"))"

+1

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

>
> The test result with failing tests for Emacs23 is attached.
>
> ,----
> | Passed: 101
> | Failed: 8 (8 unexpected)
> | Total:  109/109
> |
> | Started at:   2011-09-11 19:53:46+0200
> | Finished.
> | Finished at:  2011-09-11 19:53:51+0200
> `----
>
> 7 of the 8 are fortran-tests, the 8th is
> `org-babel-get-inline-src-block-matches'.
>

I would imagine that the fortran tests are failing because you do not
have the gfortran executable in your path.  I had to apt-get install
fortran for these tests to pass.

The eighth test should not be passing.  I've getting all tests passing
on Emacs24.

>
>>
>> Once we have all tests running for Emacs>=22 it would be very nice (and
>> not too difficult) to add a post-update hook to the Org-mode git
>> repository which could run the test suite and record the results for
>> every version of Emacs with something like the following
>>
>> emacs23 -Q --batch -l $ORGTESTSTUFF --eval "
>> (progn (org-test-run-all-tests)
>>        (htmlize-buffer)
>>        (write-file "commit-id-23.html"))"
>
> +1
>

I suppose the next step here would be to talk to Bastien about setting
up such a system on the org-mode server.

Cheers -- Eric

>
> Best,
>   -- David
> --
> OpenPGP... 0x99ADB83B5A4478E6
> Jabber.... dmjena@jabber.org
> Email..... dmaus@ictsoc.de

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

At Sun, 11 Sep 2011 15:02:08 -0600,
Eric Schulte wrote:
>
> >
> > The test result with failing tests for Emacs23 is attached.
> >
> > ,----
> > | Passed: 101
> > | Failed: 8 (8 unexpected)
> > | Total:  109/109
> > |
> > | Started at:   2011-09-11 19:53:46+0200
> > | Finished.
> > | Finished at:  2011-09-11 19:53:51+0200
> > `----
> >
> > 7 of the 8 are fortran-tests, the 8th is
> > `org-babel-get-inline-src-block-matches'.
> >
>
> I would imagine that the fortran tests are failing because you do not
> have the gfortran executable in your path.  I had to apt-get install
> fortran for these tests to pass.

m( indeed. After installing gfortran all but one run trough:

,----
| Selector: "\\(org\\|ob\\)"
| Passed: 108
| Failed: 1 (1 unexpected)
| Total:  109/109
|
| Started at:   2011-09-13 05:45:38+0200
| Finished.
| Finished at:  2011-09-13 05:45:45+0200
|
| F test-org-babel/org-babel-get-inline-src-block-matches
|    (void-function left-char)
`----

This is with

GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0) of 2010-12-11 on raven, modified by Debian

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

David Maus <dmaus@ictsoc.de> writes:

> At Sun, 11 Sep 2011 15:02:08 -0600,
> Eric Schulte wrote:
>>
>> >
>> > The test result with failing tests for Emacs23 is attached.
>> >
>> > ,----
>> > | Passed: 101
>> > | Failed: 8 (8 unexpected)
>> > | Total:  109/109
>> > |
>> > | Started at:   2011-09-11 19:53:46+0200
>> > | Finished.
>> > | Finished at:  2011-09-11 19:53:51+0200
>> > `----
>> >
>> > 7 of the 8 are fortran-tests, the 8th is
>> > `org-babel-get-inline-src-block-matches'.
>> >
>>
>> I would imagine that the fortran tests are failing because you do not
>> have the gfortran executable in your path.  I had to apt-get install
>> fortran for these tests to pass.
>
> m( indeed. After installing gfortran all but one run trough:
>
> ,----
> | Selector: "\\(org\\|ob\\)"
> | Passed: 108
> | Failed: 1 (1 unexpected)
> | Total:  109/109
> |
> | Started at:   2011-09-13 05:45:38+0200
> | Finished.
> | Finished at:  2011-09-13 05:45:45+0200
> |
> | F test-org-babel/org-babel-get-inline-src-block-matches
> |    (void-function left-char)
> `----
>

This test has been updated so that it no longer calls the left-char
function.  Please update to the latest git head and this test should
pass.

So am I right in thinking that as of right this commit [1] we are
passing all tests on Emacs 22 through Emacs24?

Thanks -- Eric

>
> This is with
>
> GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0) of
> 2010-12-11 on raven, modified by Debian
>
> Best,
>   -- David
> --
> OpenPGP... 0x99ADB83B5A4478E6
> Jabber.... dmjena@jabber.org
> Email..... dmaus@ictsoc.de


Footnotes: 
[1]  dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

At Tue, 13 Sep 2011 08:48:49 -0600,
Eric Schulte wrote:
> This test has been updated so that it no longer calls the left-char
> function.  Please update to the latest git head and this test should
> pass.

All tests pass on Emacs 23.

>
> So am I right in thinking that as of right this commit [1] we are
> passing all tests on Emacs 22 through Emacs24?

No Emacs23 through Emacs24: Backporting simple.el is scheduled for the
upcoming weekend.

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]

At Tue, 13 Sep 2011 08:48:49 -0600,
Eric Schulte wrote:
> 
> So am I right in thinking that as of right this commit [1] we are
> passing all tests on Emacs 22 through Emacs24?
> 
> Thanks -- Eric
> 
> >
> > This is with
> >
> > GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0) of
> > 2010-12-11 on raven, modified by Debian
> >
> > Best,
> >   -- David
> > --
> > OpenPGP... 0x99ADB83B5A4478E6
> > Jabber.... dmjena@jabber.org
> > Email..... dmaus@ictsoc.de
> 
> 
> Footnotes: 
> [1]  dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55
> 
> -- 
> Eric Schulte
> http://cs.unm.edu/~eschulte/

For Emacs22 the only thing we need from simple.el is the definition of
special-mode; going to factor it out and trim it to Emacs22.

With HEAD at dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55 and a fix for
the check for featurep 'org (37db5deea5ef75186bb7413b196fa0c96e5bdfb9)
I got:


Selector: "\\(org\\|ob\\)"
Passed: 99
Failed: 10 (10 unexpected)
Total:  109/109

Started at:   2011-09-16 06:56:59+0200
Finished.
Finished at:  2011-09-16 06:57:04+0200

......FF...................................................FF..F.F.......FF.................F..............F.

F ob-fortran/list-var
    Test real array input
    (void-function characterp)

F ob-fortran/list-var-from-table
    Test real array from a table
    (void-function characterp)

F test-ob-exp/org-babel-exp-src-blocks/w-no-file
    Testing export from buffers which are not visiting any file.
    (wrong-type-argument stringp nil)

F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
    Testing export without any headlines in the org-mode file.
    (wrong-type-argument stringp nil)

F test-ob-lob/export-lob-lines
    Test the export of a variety of library babel call lines.
    (wrong-type-argument stringp nil)

F test-ob-sh/dont-error-on-empty-results
    Was throwing an elisp error when shell blocks threw errors and
    (void-function org-babel-execute:sh)

F test-org-babel/inline-src-blocks
    (error "No org-babel-execute function for sh!")

F test-org-babel/inline-src_blk-default-results-replace-line-1
    (error "No org-babel-execute function for sh!")

F test-org-exp/stripping-commas
    Test the stripping of commas from within blocks during export.
    (wrong-type-argument stringp nil)

F test-org/org-link-unescape-ascii-extended-char
    Unescape old style percent escaped character.
    (ert-test-failed
     ((should
       (string= "àâçèéêîôùû"
		(org-link-unescape "%E0%E2%E7%E8%E9%EA%EE%F4%F9%FB")))
      :form
      (string= "àâçèéêîôùû" "\340\342\347\350\351\352\356\364\371\373")
      :value nil))


Best,
  -- David
-- 
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

Hi David,

>
> For Emacs22 the only thing we need from simple.el is the definition of
> special-mode; going to factor it out and trim it to Emacs22.
>

Sounds great, thanks.

>
> With HEAD at dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55 and a fix for
> the check for featurep 'org (37db5deea5ef75186bb7413b196fa0c96e5bdfb9)
> I got:
>

Thanks for compiling this information.

>
>
> Selector: "\\(org\\|ob\\)"
> Passed: 99
> Failed: 10 (10 unexpected)
> Total:  109/109
>
> Started at:   2011-09-16 06:56:59+0200
> Finished.
> Finished at:  2011-09-16 06:57:04+0200
>
> ......FF...................................................FF..F.F.......FF.................F..............F.
>
> F ob-fortran/list-var
>     Test real array input
>     (void-function characterp)
>
> F ob-fortran/list-var-from-table
>     Test real array from a table
>     (void-function characterp)
>

I've replaced the use of `characterp' in ob-fortran with integerp, which
should be backwards-compatible with emacs22.

>
> F test-ob-exp/org-babel-exp-src-blocks/w-no-file
>     Testing export from buffers which are not visiting any file.
>     (wrong-type-argument stringp nil)
>
> F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
>     Testing export without any headlines in the org-mode file.
>     (wrong-type-argument stringp nil)
>
> F test-ob-lob/export-lob-lines
>     Test the export of a variety of library babel call lines.
>     (wrong-type-argument stringp nil)
>

I have no idea what could be going wrong with these executions, would it
be difficult to generate backtraces for these failures?

>
> F test-ob-sh/dont-error-on-empty-results
>     Was throwing an elisp error when shell blocks threw errors and
>     (void-function org-babel-execute:sh)
>
> F test-org-babel/inline-src-blocks
>     (error "No org-babel-execute function for sh!")
>
> F test-org-babel/inline-src_blk-default-results-replace-line-1
>     (error "No org-babel-execute function for sh!")
>

It looks like you need to load ob-sh.el.

>
> F test-org-exp/stripping-commas
>     Test the stripping of commas from within blocks during export.
>     (wrong-type-argument stringp nil)
>
> F test-org/org-link-unescape-ascii-extended-char
>     Unescape old style percent escaped character.
>     (ert-test-failed
>      ((should
>        (string= "àâçèéêîôùû"
> 		(org-link-unescape "%E0%E2%E7%E8%E9%EA%EE%F4%F9%FB")))
>       :form
>       (string= "àâçèéêîôùû" "\340\342\347\350\351\352\356\364\371\373")
>       :value nil))
>

Hmm, I don't know anything about possible differences in string handling
between Emacs22 and more modern Emacsen, but I wouldn't be surprised if
support for accented characters was less complete in Emacs22.  Perhaps
we should just skip this test on Emacs22.

Cheers -- Eric

>
>
> Best,
>   -- David

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1.1: Type: text/plain, Size: 2242 bytes --]

At Sat, 17 Sep 2011 14:56:09 -0600,
Eric Schulte wrote:
>
> Hi David,
>
> >
> > For Emacs22 the only thing we need from simple.el is the definition of
> > special-mode; going to factor it out and trim it to Emacs22.
> >
>
> Sounds great, thanks.

Done. Conditionally defined in org-test.el when Emacs version = 22.

>
> >
> > With HEAD at dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55 and a fix for
> > the check for featurep 'org (37db5deea5ef75186bb7413b196fa0c96e5bdfb9)
> > I got:
> >
>
> Thanks for compiling this information.
>
> >
> >
> > Selector: "\\(org\\|ob\\)"
> > Passed: 99
> > Failed: 10 (10 unexpected)
> > Total:  109/109
> >
> > Started at:   2011-09-16 06:56:59+0200
> > Finished.
> > Finished at:  2011-09-16 06:57:04+0200
> >
> > ......FF...................................................FF..F.F.......FF.................F..............F.
> >
> > F ob-fortran/list-var
> >     Test real array input
> >     (void-function characterp)
> >
> > F ob-fortran/list-var-from-table
> >     Test real array from a table
> >     (void-function characterp)
> >
>
> I've replaced the use of `characterp' in ob-fortran with integerp, which
> should be backwards-compatible with emacs22.

These errors are gone.

>
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-file
> >     Testing export from buffers which are not visiting any file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
> >     Testing export without any headlines in the org-mode file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-lob/export-lob-lines
> >     Test the export of a variety of library babel call lines.
> >     (wrong-type-argument stringp nil)
> >
>
> I have no idea what could be going wrong with these executions, would it
> be difficult to generate backtraces for these failures?

No, its really easy (as expected w/ Emacs): Section on interactive
debugging ERT mentions "b" to create the backtrace.

In case of w-no-file and w-no-headers `org-todo-line-regexp' is bound
to nil, in export-lob-lines `org-maybe-keyword-time-regexp' is bound
to nil. Backtraces are attached.

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #1.2: test-ob-exp--org-babel-exp-src-blocks--w-no-headers.backtrace.gz --]
[-- Type: application/octet-stream, Size: 30134 bytes --]

[-- Attachment #1.3: test-ob-exp--org-babel-exp-src-blocks--w-no-file.backtrace.gz --]
[-- Type: application/octet-stream, Size: 30347 bytes --]

[-- Attachment #1.4: test-ob-lob--export-lob-lines.backtrace.gz --]
[-- Type: application/octet-stream, Size: 36557 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

>
>>
>> >
>> > F test-ob-exp/org-babel-exp-src-blocks/w-no-file
>> >     Testing export from buffers which are not visiting any file.
>> >     (wrong-type-argument stringp nil)
>> >
>> > F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
>> >     Testing export without any headlines in the org-mode file.
>> >     (wrong-type-argument stringp nil)
>> >
>> > F test-ob-lob/export-lob-lines
>> >     Test the export of a variety of library babel call lines.
>> >     (wrong-type-argument stringp nil)
>> >
>>
>> I have no idea what could be going wrong with these executions, would it
>> be difficult to generate backtraces for these failures?
>
> No, its really easy (as expected w/ Emacs): Section on interactive
> debugging ERT mentions "b" to create the backtrace.
>
> In case of w-no-file and w-no-headers `org-todo-line-regexp' is bound
> to nil, in export-lob-lines `org-maybe-keyword-time-regexp' is bound
> to nil. Backtraces are attached.
>

So,

Could it be that for some reason using Emacs22 all variable whose values
are set in `org-set-regexps-and-options' immediately following

  ;; Compute the regular expressions and other local variables

in org.el are somehow not having their values set?  I assume that
Org-mode simply wouldn't work on Emacs22 if this was generally the case,
so could there be something about the way that you are loading Org-mode?

Maybe explicitly adding a call to this function to the Emacs22
configuration would fix these test errors?

Hope this is helpful, please let me know if I'm miss understanding the
test results.

Thanks -- Eric

>
> Best,
>   -- David
> --
> OpenPGP... 0x99ADB83B5A4478E6
> Jabber.... dmjena@jabber.org
> Email..... dmaus@ictsoc.de
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 2061 bytes --]

At Thu, 22 Sep 2011 07:44:42 -0600,
Eric Schulte wrote:
>
> >
> >>
> >> >
> >> > F test-ob-exp/org-babel-exp-src-blocks/w-no-file
> >> >     Testing export from buffers which are not visiting any file.
> >> >     (wrong-type-argument stringp nil)
> >> >
> >> > F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
> >> >     Testing export without any headlines in the org-mode file.
> >> >     (wrong-type-argument stringp nil)
> >> >
> >> > F test-ob-lob/export-lob-lines
> >> >     Test the export of a variety of library babel call lines.
> >> >     (wrong-type-argument stringp nil)
> >> >
> >>
> >> I have no idea what could be going wrong with these executions, would it
> >> be difficult to generate backtraces for these failures?
> >
> > No, its really easy (as expected w/ Emacs): Section on interactive
> > debugging ERT mentions "b" to create the backtrace.
> >
> > In case of w-no-file and w-no-headers `org-todo-line-regexp' is bound
> > to nil, in export-lob-lines `org-maybe-keyword-time-regexp' is bound
> > to nil. Backtraces are attached.
> >
>
> So,
>
> Could it be that for some reason using Emacs22 all variable whose values
> are set in `org-set-regexps-and-options' immediately following
>
>   ;; Compute the regular expressions and other local variables
>
> in org.el are somehow not having their values set?

Nope, turns out that .org is not in Emacs22's auto-mode-alist and thus
the examples files are not opened in Org mode. Pushed a fix, turn on
Org mode before running the test body.

Now I get a beautiful:

,----
| Selector: "\\(org\\|ob\\)"
| Passed: 111
| Failed: 0
| Total:  111/111
|
| Started at:   2011-09-25 09:26:09+0200
| Finished.
| Finished at:  2011-09-25 09:26:13+0200
|
| ...............................................................................................................
`----

for GNU Emacs 22.3.2 (i686-pc-linux-gnu) of 2011-05-28 on x60s with
Org-mode version 7.7 (release_7.7.323.g07d11) \o/

Best,
  -- David
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

At Sat, 17 Sep 2011 14:56:09 -0600,
Eric Schulte wrote:
>
> >
> > F test-ob-sh/dont-error-on-empty-results
> >     Was throwing an elisp error when shell blocks threw errors and
> >     (void-function org-babel-execute:sh)
> >
> > F test-org-babel/inline-src-blocks
> >     (error "No org-babel-execute function for sh!")
> >
> > F test-org-babel/inline-src_blk-default-results-replace-line-1
> >     (error "No org-babel-execute function for sh!")
> >
>
> It looks like you need to load ob-sh.el.

Yes, (require 'ob-sh) is sufficient to remove these three from the
list of failing tests.

Best,
  -- David

--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 3354 bytes --]

At Sat, 17 Sep 2011 14:56:09 -0600,
Eric Schulte wrote:
> 
> Hi David,
> 
> >
> > For Emacs22 the only thing we need from simple.el is the definition of
> > special-mode; going to factor it out and trim it to Emacs22.
> >
> 
> Sounds great, thanks.
> 
> >
> > With HEAD at dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55 and a fix for
> > the check for featurep 'org (37db5deea5ef75186bb7413b196fa0c96e5bdfb9)
> > I got:
> >
> 
> Thanks for compiling this information.
> 
> >
> >
> > Selector: "\\(org\\|ob\\)"
> > Passed: 99
> > Failed: 10 (10 unexpected)
> > Total:  109/109
> >
> > Started at:   2011-09-16 06:56:59+0200
> > Finished.
> > Finished at:  2011-09-16 06:57:04+0200
> >
> > ......FF...................................................FF..F.F.......FF.................F..............F.
> >
> > F ob-fortran/list-var
> >     Test real array input
> >     (void-function characterp)
> >
> > F ob-fortran/list-var-from-table
> >     Test real array from a table
> >     (void-function characterp)
> >
> 
> I've replaced the use of `characterp' in ob-fortran with integerp, which
> should be backwards-compatible with emacs22.
> 
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-file
> >     Testing export from buffers which are not visiting any file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
> >     Testing export without any headlines in the org-mode file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-lob/export-lob-lines
> >     Test the export of a variety of library babel call lines.
> >     (wrong-type-argument stringp nil)
> >
> 
> I have no idea what could be going wrong with these executions, would it
> be difficult to generate backtraces for these failures?
> 
> >
> > F test-ob-sh/dont-error-on-empty-results
> >     Was throwing an elisp error when shell blocks threw errors and
> >     (void-function org-babel-execute:sh)
> >
> > F test-org-babel/inline-src-blocks
> >     (error "No org-babel-execute function for sh!")
> >
> > F test-org-babel/inline-src_blk-default-results-replace-line-1
> >     (error "No org-babel-execute function for sh!")
> >
> 
> It looks like you need to load ob-sh.el.
> 
> >
> > F test-org-exp/stripping-commas
> >     Test the stripping of commas from within blocks during export.
> >     (wrong-type-argument stringp nil)

And last not least: In this case it is `org-todo-line-regexp' bound to
nil, too.

> >
> > F test-org/org-link-unescape-ascii-extended-char
> >     Unescape old style percent escaped character.
> >     (ert-test-failed
> >      ((should
> >        (string= "àâçèéêîôùû"
> > 		(org-link-unescape "%E0%E2%E7%E8%E9%EA%EE%F4%F9%FB")))
> >       :form
> >       (string= "àâçèéêîôùû" "\340\342\347\350\351\352\356\364\371\373")
> >       :value nil))
> >
> 
> Hmm, I don't know anything about possible differences in string handling
> between Emacs22 and more modern Emacsen, but I wouldn't be surprised if
> support for accented characters was less complete in Emacs22.  Perhaps
> we should just skip this test on Emacs22.

I'll dig into the link escaping failure.

Best,
  -- David
-- 
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1: Type: text/plain, Size: 3720 bytes --]

At Sat, 17 Sep 2011 14:56:09 -0600,
Eric Schulte wrote:
> 
> Hi David,
> 
> >
> > For Emacs22 the only thing we need from simple.el is the definition of
> > special-mode; going to factor it out and trim it to Emacs22.
> >
> 
> Sounds great, thanks.
> 
> >
> > With HEAD at dbf0e6d5bcbe94c8ee57d68889d3c25bf9cdef55 and a fix for
> > the check for featurep 'org (37db5deea5ef75186bb7413b196fa0c96e5bdfb9)
> > I got:
> >
> 
> Thanks for compiling this information.
> 
> >
> >
> > Selector: "\\(org\\|ob\\)"
> > Passed: 99
> > Failed: 10 (10 unexpected)
> > Total:  109/109
> >
> > Started at:   2011-09-16 06:56:59+0200
> > Finished.
> > Finished at:  2011-09-16 06:57:04+0200
> >
> > ......FF...................................................FF..F.F.......FF.................F..............F.
> >
> > F ob-fortran/list-var
> >     Test real array input
> >     (void-function characterp)
> >
> > F ob-fortran/list-var-from-table
> >     Test real array from a table
> >     (void-function characterp)
> >
> 
> I've replaced the use of `characterp' in ob-fortran with integerp, which
> should be backwards-compatible with emacs22.
> 
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-file
> >     Testing export from buffers which are not visiting any file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-exp/org-babel-exp-src-blocks/w-no-headers
> >     Testing export without any headlines in the org-mode file.
> >     (wrong-type-argument stringp nil)
> >
> > F test-ob-lob/export-lob-lines
> >     Test the export of a variety of library babel call lines.
> >     (wrong-type-argument stringp nil)
> >
> 
> I have no idea what could be going wrong with these executions, would it
> be difficult to generate backtraces for these failures?
> 
> >
> > F test-ob-sh/dont-error-on-empty-results
> >     Was throwing an elisp error when shell blocks threw errors and
> >     (void-function org-babel-execute:sh)
> >
> > F test-org-babel/inline-src-blocks
> >     (error "No org-babel-execute function for sh!")
> >
> > F test-org-babel/inline-src_blk-default-results-replace-line-1
> >     (error "No org-babel-execute function for sh!")
> >
> 
> It looks like you need to load ob-sh.el.
> 
> >
> > F test-org-exp/stripping-commas
> >     Test the stripping of commas from within blocks during export.
> >     (wrong-type-argument stringp nil)
> >
> > F test-org/org-link-unescape-ascii-extended-char
> >     Unescape old style percent escaped character.
> >     (ert-test-failed
> >      ((should
> >        (string= "àâçèéêîôùû"
> > 		(org-link-unescape "%E0%E2%E7%E8%E9%EA%EE%F4%F9%FB")))
> >       :form
> >       (string= "àâçèéêîôùû" "\340\342\347\350\351\352\356\364\371\373")
> >       :value nil))
> >
> 
> Hmm, I don't know anything about possible differences in string handling
> between Emacs22 and more modern Emacsen, but I wouldn't be surprised if
> support for accented characters was less complete in Emacs22.  Perhaps
> we should just skip this test on Emacs22.

Nope, this test is fixed. Had to explicitely tell Emacs that the
expected output string is latin-1; i.e. the input is extended ASCII
which is covered by latin-1 encoding. The test failed because
org-link-unescape returned the raw byte sequence.

I am aware that there might be a problem with opening links that
contain extended ASCII characters (=Org's old escaping choice) and I'm
going to test this next. 

The test is fine, though. The expected output is a latin-1 encoded
string.

Best,
  -- David
-- 
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Bastien]

Hi Eric and David,

Eric Schulte <schulte.eric@gmail.com> writes:

> I suppose the next step here would be to talk to Bastien about setting
> up such a system on the org-mode server.

For the record, I'm all for a server-side setup that will test
latest Org on a regular basis.

Jason, depending on our server resources, would that be feasible
to run regular (cron'ed) tests?  What is the best frequency?

Thanks,

-- 
 Bastien

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Sebastien Vauban]

Hi Bastien,

Bastien wrote:
> Hi Eric and David,
> Eric Schulte <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> I suppose the next step here would be to talk to Bastien about setting
>> up such a system on the org-mode server.
>
> For the record, I'm all for a server-side setup that will test
> latest Org on a regular basis.
>
> Jason, depending on our server resources, would that be feasible
> to run regular (cron'ed) tests?  What is the best frequency?

Wouldn't the best frequency be: at every commit?

That's not for the minute (maybe even less) it takes to run the 102 tests, I
guess...

Best regards,
  Seb

-- 
Sebastien Vauban

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

"Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> writes:

> Hi Bastien,
>
> Bastien wrote:
>> Hi Eric and David,
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> I suppose the next step here would be to talk to Bastien about setting
>>> up such a system on the org-mode server.
>>
>> For the record, I'm all for a server-side setup that will test
>> latest Org on a regular basis.
>>

Fantastic.

>>
>> Jason, depending on our server resources, would that be feasible
>> to run regular (cron'ed) tests?  What is the best frequency?
>
> Wouldn't the best frequency be: at every commit?
>
> That's not for the minute (maybe even less) it takes to run the 102 tests, I
> guess...
>

$ time emacs -Q --batch \
    -l ~/.emacs.d/src/org/testing/org-test.el \
    --eval "(setq org-confirm-babel-evaluate nil)" \
    -f org-test-run-batch-tests 2>/dev/null 1>/dev/null \
4.00s user 0.44s system 56% cpu 7.834 total

So at 8 seconds (on my machine) it shouldn't put too much load on the
server to run after every commit, we would just want to be sure that
multiple instances of the test suite are not run at once.  A check to
only run the suite if it is not currently running shouldn't be difficult
to run.

I can work with Jason offline to figure out how to set this up.

Cheers -- Eric

--
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Bastien]

Hello,

Eric Schulte <schulte.eric@gmail.com> writes:

> $ time emacs -Q --batch \
>     -l ~/.emacs.d/src/org/testing/org-test.el \
>     --eval "(setq org-confirm-babel-evaluate nil)" \
>     -f org-test-run-batch-tests 2>/dev/null 1>/dev/null \
> 4.00s user 0.44s system 56% cpu 7.834 total
>
> So at 8 seconds (on my machine) it shouldn't put too much load on the
> server to run after every commit, we would just want to be sure that
> multiple instances of the test suite are not run at once.  A check to
> only run the suite if it is not currently running shouldn't be difficult
> to run.
>
> I can work with Jason offline to figure out how to set this up.

I blindly trust you and Jason for finding the best solution here.

If having tests after each commit does not create unwanted complexity, 
let's go with it.  

Thanks,

-- 
 Bastien

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Sebastien Vauban]

Hi Eric,

Eric Schulte wrote:
> "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> Bastien wrote:
>>> Jason, depending on our server resources, would that be feasible
>>> to run regular (cron'ed) tests?  What is the best frequency?
>>
>> Wouldn't the best frequency be: at every commit?
>>
>> That's not for the minute (maybe even less) it takes to run the 102 tests,
>> I guess...
>
> At 8 seconds (on my machine) it shouldn't put too much load on the server to
> run after every commit, we would just want to be sure that multiple
> instances of the test suite are not run at once. A check to only run the
> suite if it is not currently running shouldn't be difficult to run.

For my information, why do you need to test that 2 suites don't run at the
same time?  They only write to temp buffers, no?  Can they conflict?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Brian Wightman]

On Mon, Oct 24, 2011 at 7:12 AM, Sebastien Vauban
<wxhgmqzgwmuf@spammotel.com> wrote:
> For my information, why do you need to test that 2 suites don't run at the
> same time?  They only write to temp buffers, no?  Can they conflict?

If they do conflict and only one set of tests should run at a time,
shouldn't the second set of tests be queued up so it runs when the
first is complete?  If this isn't done, I could see a situation where
at least one commit remains untested until the next commit.

my $0.02;
Brian

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Martyn Jago]

Hi

Brian Wightman <MidLifeXis@wightmanfam.org> writes:

> On Mon, Oct 24, 2011 at 7:12 AM, Sebastien Vauban
> <wxhgmqzgwmuf@spammotel.com> wrote:
>> For my information, why do you need to test that 2 suites don't run at the
>> same time?  They only write to temp buffers, no?  Can they conflict?
>
> If they do conflict and only one set of tests should run at a time,
> shouldn't the second set of tests be queued up so it runs when the
> first is complete?  If this isn't done, I could see a situation where
> at least one commit remains untested until the next commit.
>
> my $0.02;
> Brian

I would think it would make sense to ensure the newly committed code
also builds correctly (including info), and why not test back to Emacs
22 also (since that is how this thread came about)? I have set up
something similar, and doing all this takes less than 1 minute on my
machine.

Incidentally, I added info on running tests on Emacs 22 and 23 to Worg at 

http://orgmode.org/worg/org-tests/index.html

Best, Martyn

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Hi
>
> Brian Wightman <MidLifeXis@wightmanfam.org> writes:
>
>> On Mon, Oct 24, 2011 at 7:12 AM, Sebastien Vauban
>> <wxhgmqzgwmuf@spammotel.com> wrote:
>>> For my information, why do you need to test that 2 suites don't run at the
>>> same time?  They only write to temp buffers, no?  Can they conflict?
>>
>> If they do conflict and only one set of tests should run at a time,
>> shouldn't the second set of tests be queued up so it runs when the
>> first is complete?  If this isn't done, I could see a situation where
>> at least one commit remains untested until the next commit.
>>
>> my $0.02;
>> Brian
>
> I would think it would make sense to ensure the newly committed code
> also builds correctly (including info), and why not test back to Emacs
> 22 also (since that is how this thread came about)? I have set up
> something similar, and doing all this takes less than 1 minute on my
> machine.
>
> Incidentally, I added info on running tests on Emacs 22 and 23 to Worg at 
>
> http://orgmode.org/worg/org-tests/index.html
>

These are good points, so the best solution would have two parts, the
first should en-queue each new commit as it is received, and the second
should pop commits off of the queue, run the test suite on that commit
and write the results to an html file.

My first thought for a shell-script solution would use a mkfifo named
pipe, something like the following...

,----[setup]
| mkfifo /tmp/org-commits-to-test
`----

,----[in a post-commit hook]
| echo $COMMIT >> /tmp/org-commits-to-test
`----

,----[in another constantly-running shell script]
| while true; do
|   for i in $(cat /tmp/org-commits-to-test);do
|     cd /testing/repo/location
|     git checkout $i
|     git reset --hard HEAD
|     emacs --batch ...
|   done
| done
`----

But I imagine there is likely a better solution.

Best -- Eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Samuel Wales]

Dunno if this is off-topic for this thread -- but one thing I think
would be really great in the testing framework is the ability to test
visibility.  Looking for ellipses in the wrong places, headlines not
showing, etc.  I have constant issues with folding and elision in Org.
 Is this covered?

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com
===
Bigotry against people with serious diseases is still bigotry.

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Eric Schulte]

Samuel Wales <samologist@gmail.com> writes:

> Dunno if this is off-topic for this thread -- but one thing I think
> would be really great in the testing framework is the ability to test
> visibility.  Looking for ellipses in the wrong places, headlines not
> showing, etc.  I have constant issues with folding and elision in Org.
>  Is this covered?

I was talking more about the test framework rather than specific tests
(such as tests for visibility), but I agree that tests of the visibility
of elements would be useful -- although I don't know off the top of my
head what elisp primitives would be most useful for testing visibility.

Best -- Eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: FYI: Org mode testing framework, Emacs 23 and 22

[David Maus]

[-- Attachment #1.1: Type: text/plain, Size: 106 bytes --]

...Attachment...
--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #1.2: org-test-emacs23.txt --]
[-- Type: text/plain, Size: 1728 bytes --]

Selector: "\\(org\\|ob\\)"
Passed: 101
Failed: 8 (8 unexpected)
Total:  109/109

Started at:   2011-09-11 19:53:46+0200
Finished.
Finished at:  2011-09-11 19:53:51+0200

...FFFFF.FF.........................................................................F........................

F ob-fortran/command-arguments
    Test real array from a table
    (wrong-type-argument number-or-marker-p "")

F ob-fortran/fortran-var-program
    Test a fortran variable
    (wrong-type-argument number-or-marker-p "")

F ob-fortran/input-var
    Test :var
    (wrong-type-argument number-or-marker-p "")

F ob-fortran/list-var
    Test real array input
    (ert-test-failed
     ((should
       (equal "1.00 2.00 3.00"
	      (org-babel-execute-src-block)))
      :form
      (equal "1.00 2.00 3.00" "")
      :value nil :explanation
      (arrays-of-different-length 14 0 "1.00 2.00 3.00" "" first-mismatch-at 0)))

F ob-fortran/list-var-from-table
    Test real array from a table
    (ert-test-failed
     ((should
       (equal "1.00 2.00"
	      (org-babel-execute-src-block)))
      :form
      (equal "1.00 2.00" "")
      :value nil :explanation
      (arrays-of-different-length 9 0 "1.00 2.00" "" first-mismatch-at 0)))

F ob-fortran/preprosessor-var
    Test preprocessed fortran
    (wrong-type-argument number-or-marker-p "")

F ob-fortran/simple-program
    Test of hello world program.
    (ert-test-failed
     ((should
       (equal "Hello world"
	      (org-babel-execute-src-block)))
      :form
      (equal "Hello world" "")
      :value nil :explanation
      (arrays-of-different-length 11 0 "Hello world" "" first-mismatch-at 0)))

F test-org-babel/org-babel-get-inline-src-block-matches
    (void-function left-char)


[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: FYI: Org mode testing framework, Emacs 23 and 22

[Martyn Jago]

Hi David

David Maus <dmaus@ictsoc.de> writes:


[...]

> F test-org-babel/org-babel-get-inline-src-block-matches
>     (void-function left-char)

I've pushed a patch to the mailing-list which should fix the above
test in Emacs 22. I will take a look at the fortran related tests when I get a
chance.

Best, Martyn