* Test framework needed
@ 2011-03-30 13:01 Rainer M Krug
2011-03-30 13:46 ` Eric Abrahamsen
0 siblings, 1 reply; 15+ messages in thread
From: Rainer M Krug @ 2011-03-30 13:01 UTC (permalink / raw)
To: emacs-orgmode
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi
I was bitten again from an unintended regression in org-mode, and that
the second time in two weeks.
I am probably not the right person to suggest this, but I think it is
time to introduce a test framework for org-mode, to ensure that the
(without doubt useful) approach to develop org-mode does not lead to
regressions.
org is, as I see it, complex and is getting broader and broader - time
management, task management, literate programming, recipes (I like the
idea), all have different (possibly contradicting?) requirements. And I
think it is getting more and more difficult for new contributors to have
the overview to make sure that something they are contributing actually
does not break something else.
As I think it would be *really bad* idea to split org, I think that it
needs to be assured that no regressions are introduced.
Unfortunately I have no idea about elisp and can therefore not
contribute to the framework itself, but I guess all of us could
contribute small org files as test cases and expected resulting outputs
(exports, tangling, agenda files, ...). If the framework could
accommodate those, everybody could help to bring together a (definitely
not perfect) test framework, that would bring org at least one step
closer to error-free growth.
I have to add I really love org and babel and have no idea what I would
use instead for literal programming in R.
Cheers,
Rainer
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa
Tel: +33 - (0)9 53 10 27 44
Cell: +27 - (0)8 39 47 90 42
Fax (SA): +27 - (0)8 65 16 27 82
Fax (D) : +49 - (0)3 21 21 25 22 44
Fax (FR): +33 - (0)9 58 10 27 44
email: Rainer@krugs.de
Skype: RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2TKbwACgkQoYgNqgF2egpQJwCdG7C+3Ft+xGyLBiwY6aBwMLaC
TIIAn0RD3ypbvkfE1RAL4FK3ZwmnERdE
=GlJt
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed
2011-03-30 13:01 Test framework needed Rainer M Krug
@ 2011-03-30 13:46 ` Eric Abrahamsen
2011-03-30 13:56 ` Rainer M Krug
0 siblings, 1 reply; 15+ messages in thread
From: Eric Abrahamsen @ 2011-03-30 13:46 UTC (permalink / raw)
To: emacs-orgmode
On Wed, Mar 30 2011, Rainer M Krug wrote:
> Hi
>
> I was bitten again from an unintended regression in org-mode, and that
> the second time in two weeks.
>
> I am probably not the right person to suggest this, but I think it is
> time to introduce a test framework for org-mode, to ensure that the
> (without doubt useful) approach to develop org-mode does not lead to
> regressions.
This would be the page to start with, though the most likely candidate
(Elisp Regression Testing) is only available in Emacs trunk at the
moment…
http://www.emacswiki.org/emacs/UnitTesting
E
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 13:46 ` Eric Abrahamsen
@ 2011-03-30 13:56 ` Rainer M Krug
2011-03-30 14:11 ` Eric Abrahamsen
2011-03-30 14:18 ` Christian Egli
0 siblings, 2 replies; 15+ messages in thread
From: Rainer M Krug @ 2011-03-30 13:56 UTC (permalink / raw)
To: Eric Abrahamsen; +Cc: emacs-orgmode
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/03/11 15:46, Eric Abrahamsen wrote:
> On Wed, Mar 30 2011, Rainer M Krug wrote:
>
>> Hi
>>
>> I was bitten again from an unintended regression in org-mode, and that
>> the second time in two weeks.
>>
>> I am probably not the right person to suggest this, but I think it is
>> time to introduce a test framework for org-mode, to ensure that the
>> (without doubt useful) approach to develop org-mode does not lead to
>> regressions.
>
> This would be the page to start with, though the most likely candidate
> (Elisp Regression Testing) is only available in Emacs trunk at the
> moment…
>
> http://www.emacswiki.org/emacs/UnitTesting
Am I right in assuming, that all of the possible test frameworks would
require org files and the expected output (tengle, export to ...,
agenda, ...)? In this case, would it make sense to start collecting
those, as they can easily be user contributed, consequently representing
a cross section of the use cases (even not intended use cases)?
Rainer
>
> E
>
>
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa
Tel: +33 - (0)9 53 10 27 44
Cell: +27 - (0)8 39 47 90 42
Fax (SA): +27 - (0)8 65 16 27 82
Fax (D) : +49 - (0)3 21 21 25 22 44
Fax (FR): +33 - (0)9 58 10 27 44
email: Rainer@krugs.de
Skype: RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2TNpwACgkQoYgNqgF2egpnWACeMFmP1p7mKsMXsELXcFSEqW99
mMkAnivsDT7zmguzphqziXzoKdx5fqz3
=AWd5
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed
2011-03-30 13:56 ` Rainer M Krug
@ 2011-03-30 14:11 ` Eric Abrahamsen
2011-03-30 14:22 ` Rainer M Krug
2011-03-30 14:26 ` MidLifeXis at PerlMonks
2011-03-30 14:18 ` Christian Egli
1 sibling, 2 replies; 15+ messages in thread
From: Eric Abrahamsen @ 2011-03-30 14:11 UTC (permalink / raw)
To: emacs-orgmode
On Wed, Mar 30 2011, Rainer M Krug wrote:
> On 30/03/11 15:46, Eric Abrahamsen wrote:
>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>>
>>> Hi
>>>
>>> I was bitten again from an unintended regression in org-mode, and that
>>> the second time in two weeks.
>>>
>>> I am probably not the right person to suggest this, but I think it is
>>> time to introduce a test framework for org-mode, to ensure that the
>>> (without doubt useful) approach to develop org-mode does not lead to
>>> regressions.
>>
>> This would be the page to start with, though the most likely candidate
>> (Elisp Regression Testing) is only available in Emacs trunk at the
>> moment…
>>
>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?
Yup, what you would need is some org source files that exercise all of
the possible export options (for testing export, for example), including
weird edge cases, and then ERT (if that's what we ended up using) would
provide handy functions for making sure the export output matches
expectations. The excellent gentleman who created the ODT exporter,
whose name currently escapes me, has already created test files for his
exporter—that would be a perfect place to start.
Covering all of org's various functions would end up being a bit of a
PITA, though you're quite right that it's an excellent idea, and will
become more and more necessary.
E
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed
2011-03-30 13:56 ` Rainer M Krug
2011-03-30 14:11 ` Eric Abrahamsen
@ 2011-03-30 14:18 ` Christian Egli
2011-03-30 14:30 ` Rainer M Krug
1 sibling, 1 reply; 15+ messages in thread
From: Christian Egli @ 2011-03-30 14:18 UTC (permalink / raw)
To: emacs-orgmode
Rainer M Krug <r.m.krug@gmail.com> writes:
>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?
Before you go too far with this; Orgmode already contains a unit test
suite. Look at the README in the testing directory
(http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)
--
Christian Egli
Swiss Library for the Blind, Visually Impaired and Print Disabled
Grubenstrasse 12, CH-8045 Zürich, Switzerland
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 14:11 ` Eric Abrahamsen
@ 2011-03-30 14:22 ` Rainer M Krug
2011-03-30 14:26 ` MidLifeXis at PerlMonks
1 sibling, 0 replies; 15+ messages in thread
From: Rainer M Krug @ 2011-03-30 14:22 UTC (permalink / raw)
To: Eric Abrahamsen; +Cc: emacs-orgmode
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/03/11 16:11, Eric Abrahamsen wrote:
> On Wed, Mar 30 2011, Rainer M Krug wrote:
>
>> On 30/03/11 15:46, Eric Abrahamsen wrote:
>>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>>>
>>>> Hi
>>>>
>>>> I was bitten again from an unintended regression in org-mode, and that
>>>> the second time in two weeks.
>>>>
>>>> I am probably not the right person to suggest this, but I think it is
>>>> time to introduce a test framework for org-mode, to ensure that the
>>>> (without doubt useful) approach to develop org-mode does not lead to
>>>> regressions.
>>>
>>> This would be the page to start with, though the most likely candidate
>>> (Elisp Regression Testing) is only available in Emacs trunk at the
>>> moment…
>>>
>>> http://www.emacswiki.org/emacs/UnitTesting
>>
>> Am I right in assuming, that all of the possible test frameworks would
>> require org files and the expected output (tengle, export to ...,
>> agenda, ...)? In this case, would it make sense to start collecting
>> those, as they can easily be user contributed, consequently representing
>> a cross section of the use cases (even not intended use cases)?
>
> Yup, what you would need is some org source files that exercise all of
> the possible export options (for testing export, for example), including
> weird edge cases, and then ERT (if that's what we ended up using) would
> provide handy functions for making sure the export output matches
> expectations. The excellent gentleman who created the ODT exporter,
> whose name currently escapes me, has already created test files for his
> exporter—that would be a perfect place to start.
>
> Covering all of org's various functions would end up being a bit of a
> PITA, though you're quite right that it's an excellent idea, and will
> become more and more necessary.
So would there be a possibility of "normal org users" (if there is such
a thing ...) to contribute to this?
What would be needed? Any specific structure of the org files?
Rainer
>
> E
>
>
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa
Tel: +33 - (0)9 53 10 27 44
Cell: +27 - (0)8 39 47 90 42
Fax (SA): +27 - (0)8 65 16 27 82
Fax (D) : +49 - (0)3 21 21 25 22 44
Fax (FR): +33 - (0)9 58 10 27 44
email: Rainer@krugs.de
Skype: RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2TPJgACgkQoYgNqgF2egoWKwCeMjWgggD7JMhVTrQTHe3f7n6s
VhgAn2CG25hOa1Q4RPufarreQVYlezHm
=18s2
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 14:11 ` Eric Abrahamsen
2011-03-30 14:22 ` Rainer M Krug
@ 2011-03-30 14:26 ` MidLifeXis at PerlMonks
1 sibling, 0 replies; 15+ messages in thread
From: MidLifeXis at PerlMonks @ 2011-03-30 14:26 UTC (permalink / raw)
To: emacs-orgmode
As a heavy Perl user, writing /automated/ tests is a large part of my dev work.
I would suggest / plea / encourage that whatever framework is used can be
automated. If it cannot be run as part of an automated process it is not going
to be run. Also consider a set of testing platforms (emacs version, supporting
versions of other .el modules, OS version, external software). There are many
dependencies that org has - being able to automate this testing is a must.
Just my $0.02.
Brian / MidLifeXis
----- Original Message ----
From: Eric Abrahamsen <eric@ericabrahamsen.net>
To: emacs-orgmode@gnu.org
Sent: Wed, March 30, 2011 9:11:23 AM
Subject: [O] Re: Test framework needed
On Wed, Mar 30 2011, Rainer M Krug wrote:
> On 30/03/11 15:46, Eric Abrahamsen wrote:
>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>>
>>> Hi
>>>
>>> I was bitten again from an unintended regression in org-mode, and that
>>> the second time in two weeks.
>>>
>>> I am probably not the right person to suggest this, but I think it is
>>> time to introduce a test framework for org-mode, to ensure that the
>>> (without doubt useful) approach to develop org-mode does not lead to
>>> regressions.
>>
>> This would be the page to start with, though the most likely candidate
>> (Elisp Regression Testing) is only available in Emacs trunk at the
>> moment…
>>
>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?
Yup, what you would need is some org source files that exercise all of
the possible export options (for testing export, for example), including
weird edge cases, and then ERT (if that's what we ended up using) would
provide handy functions for making sure the export output matches
expectations. The excellent gentleman who created the ODT exporter,
whose name currently escapes me, has already created test files for his
exporter—that would be a perfect place to start.
Covering all of org's various functions would end up being a bit of a
PITA, though you're quite right that it's an excellent idea, and will
become more and more necessary.
E
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 14:18 ` Christian Egli
@ 2011-03-30 14:30 ` Rainer M Krug
2011-03-30 15:13 ` Manuel Giraud
2011-03-30 21:42 ` Eric Schulte
0 siblings, 2 replies; 15+ messages in thread
From: Rainer M Krug @ 2011-03-30 14:30 UTC (permalink / raw)
To: Christian Egli; +Cc: emacs-orgmode
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/03/11 16:18, Christian Egli wrote:
> Rainer M Krug <r.m.krug@gmail.com> writes:
>
>>> http://www.emacswiki.org/emacs/UnitTesting
>>
>> Am I right in assuming, that all of the possible test frameworks would
>> require org files and the expected output (tengle, export to ...,
>> agenda, ...)? In this case, would it make sense to start collecting
>> those, as they can easily be user contributed, consequently representing
>> a cross section of the use cases (even not intended use cases)?
>
> Before you go too far with this; Orgmode already contains a unit test
> suite. Look at the README in the testing directory
> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)
>
But it does not look as if it is used very often... There are not many
test org files, and I did not se anything which compares the resulting
exported / tangle file with an expected output?
Please correct me if I am missing something.
This suite should actually be updated with effectively each patch which
introduces new features and run after each patch.
So is it only necessary to add meat to this framework?
Rainer
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa
Tel: +33 - (0)9 53 10 27 44
Cell: +27 - (0)8 39 47 90 42
Fax (SA): +27 - (0)8 65 16 27 82
Fax (D) : +49 - (0)3 21 21 25 22 44
Fax (FR): +33 - (0)9 58 10 27 44
email: Rainer@krugs.de
Skype: RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2TPp4ACgkQoYgNqgF2egpBZQCfQN+g3J2+BiyKaWzh6o847mZS
PR4An0aW0Di6vfgzwsJgwfK4FajL65WP
=9/CW
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 14:30 ` Rainer M Krug
@ 2011-03-30 15:13 ` Manuel Giraud
2011-03-30 20:14 ` Aankhen
2011-03-30 21:42 ` Eric Schulte
1 sibling, 1 reply; 15+ messages in thread
From: Manuel Giraud @ 2011-03-30 15:13 UTC (permalink / raw)
To: Rainer M Krug; +Cc: emacs-orgmode, Christian Egli
Rainer M Krug <r.m.krug@gmail.com> writes:
> But it does not look as if it is used very often... There are not many
> test org files, and I did not se anything which compares the resulting
> exported / tangle file with an expected output?
It could be useful to have this kind of framework for org but I think it
is difficult problem when it comes to org export. For example, is
removing the "validate xhtml" button from an html export make this
export invalid? I guess not.
> Please correct me if I am missing something.
>
> This suite should actually be updated with effectively each patch which
> introduces new features and run after each patch.
Which renders this framework far less automatic. I think that having a
set of org files against which one could try any export and *see* that
the results are almost correct would be enough.
--
Manuel Giraud
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 15:13 ` Manuel Giraud
@ 2011-03-30 20:14 ` Aankhen
2011-03-30 21:39 ` Eric Schulte
0 siblings, 1 reply; 15+ messages in thread
From: Aankhen @ 2011-03-30 20:14 UTC (permalink / raw)
To: Manuel Giraud, Rainer M Krug, emacs-orgmode, Christian Egli
On Wed, Mar 30, 2011 at 20:43, Manuel Giraud
<manuel.giraud@univ-nantes.fr> wrote:
> Rainer M Krug <r.m.krug@gmail.com> writes:
> [snip]
>
>> Please correct me if I am missing something.
>>
>> This suite should actually be updated with effectively each patch which
>> introduces new features and run after each patch.
>
> Which renders this framework far less automatic. I think that having a
> set of org files against which one could try any export and *see* that
> the results are almost correct would be enough.
I think the “automated” part refers to running the tests. What you
suggest—having a set of files that you could manually export to verify
the results—wouldn’t be of much help, IMHO. First, it’d require a lot
more time than executing a single command and checking the summary at
the end. Second, it’d be very error-prone.
A comprehensive automated test suite gives people writing patches an
easier way to perform regression testing and catch any unintended
consequences. On the other hand, it /does/ take a lot of effort to
keep it in sync with the codebase… maybe we need a Test Fairy. ;-)
Aankhen
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 20:14 ` Aankhen
@ 2011-03-30 21:39 ` Eric Schulte
0 siblings, 0 replies; 15+ messages in thread
From: Eric Schulte @ 2011-03-30 21:39 UTC (permalink / raw)
To: Aankhen; +Cc: Christian Egli, emacs-orgmode, Manuel Giraud, Rainer M Krug
Aankhen <aankhen@gmail.com> writes:
> On Wed, Mar 30, 2011 at 20:43, Manuel Giraud
> <manuel.giraud@univ-nantes.fr> wrote:
>> Rainer M Krug <r.m.krug@gmail.com> writes:
>> [snip]
>>
>>> Please correct me if I am missing something.
>>>
>>> This suite should actually be updated with effectively each patch which
>>> introduces new features and run after each patch.
>>
>> Which renders this framework far less automatic. I think that having a
>> set of org files against which one could try any export and *see* that
>> the results are almost correct would be enough.
>
I think an "automatic" solution would involve running a batch Emacs
process which loads up Org-mode and then loads and runs the existing
test suite. This would be both easily automatable and would allow
testing of various versions of Emacs with relative ease.
Best -- Eric
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 14:30 ` Rainer M Krug
2011-03-30 15:13 ` Manuel Giraud
@ 2011-03-30 21:42 ` Eric Schulte
2011-03-31 0:19 ` Suvayu Ali
1 sibling, 1 reply; 15+ messages in thread
From: Eric Schulte @ 2011-03-30 21:42 UTC (permalink / raw)
To: Rainer M Krug; +Cc: emacs-orgmode, Christian Egli
Rainer M Krug <r.m.krug@gmail.com> writes:
> On 30/03/11 16:18, Christian Egli wrote:
>> Rainer M Krug <r.m.krug@gmail.com> writes:
>>
>>>> http://www.emacswiki.org/emacs/UnitTesting
>>>
>>> Am I right in assuming, that all of the possible test frameworks would
>>> require org files and the expected output (tengle, export to ...,
>>> agenda, ...)? In this case, would it make sense to start collecting
>>> those, as they can easily be user contributed, consequently representing
>>> a cross section of the use cases (even not intended use cases)?
>>
>> Before you go too far with this; Orgmode already contains a unit test
>> suite. Look at the README in the testing directory
>> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)
>>
>
> But it does not look as if it is used very often... There are not many
> test org files, and I did not se anything which compares the resulting
> exported / tangle file with an expected output?
>
> Please correct me if I am missing something.
>
You are correct that the existing test suite needs some attention and
some more tests. Just a general commitment to convert problem reports
from the mailing list into unit tests should be a step in the right
direction.
However the existing test suite (while under populated) is the result of
multiple previous discussion of this nature on the mailing list, and I
think that abandoning the existing infrastructure would be a step in the
wrong direction.
>
> This suite should actually be updated with effectively each patch which
> introduces new features and run after each patch.
>
Agreed, in a perfect world...
>
> So is it only necessary to add meat to this framework?
>
Yes, I believe the best way forward would be to add tests to the
existing framework.
Best -- Eric
>
> Rainer
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-30 21:42 ` Eric Schulte
@ 2011-03-31 0:19 ` Suvayu Ali
2011-03-31 3:40 ` Eric Schulte
0 siblings, 1 reply; 15+ messages in thread
From: Suvayu Ali @ 2011-03-31 0:19 UTC (permalink / raw)
To: emacs-orgmode
On Wed, 30 Mar 2011 15:42:19 -0600
"Eric Schulte" <schulte.eric@gmail.com> wrote:
> >
> > This suite should actually be updated with effectively each patch
> > which introduces new features and run after each patch.
> >
>
> Agreed, in a perfect world...
>
> >
> > So is it only necessary to add meat to this framework?
> >
>
> Yes, I believe the best way forward would be to add tests to the
> existing framework.
I have a possibly completely useless idea regarding "automatically"
checking regressions. As far I understand the problem now is its not
very feasible to do automated tests with what ever test suite we have
(or will have after the improvements) and see the exported results for
each patch, as at some step it involves human intervention (as in, was
the export good).
So maybe we can have a directory on the Worg website (not part of the
Worg git repo) where every week or so the test suites will publish with
what ever the org-mode head is at the time for all the supported
formats. Then us "puny lisp illiterate" users can check up on it over
the course of the week and report back to the list if there is a
problem.
Since this way people can look at the export formats they are
interested in, none of the formats get treated like a step child
either. Would that be feasible? Or did I completely misunderstand the
problem at hand?
--
Suvayu
Open source is the future. It sets us free.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-31 0:19 ` Suvayu Ali
@ 2011-03-31 3:40 ` Eric Schulte
2011-03-31 7:15 ` Rainer M Krug
0 siblings, 1 reply; 15+ messages in thread
From: Eric Schulte @ 2011-03-31 3:40 UTC (permalink / raw)
To: Suvayu Ali; +Cc: emacs-orgmode
Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:
> On Wed, 30 Mar 2011 15:42:19 -0600
> "Eric Schulte" <schulte.eric@gmail.com> wrote:
>
>> >
>> > This suite should actually be updated with effectively each patch
>> > which introduces new features and run after each patch.
>> >
>>
>> Agreed, in a perfect world...
>>
>> >
>> > So is it only necessary to add meat to this framework?
>> >
>>
>> Yes, I believe the best way forward would be to add tests to the
>> existing framework.
>
> I have a possibly completely useless idea regarding "automatically"
> checking regressions. As far I understand the problem now is its not
> very feasible to do automated tests with what ever test suite we have
> (or will have after the improvements) and see the exported results for
> each patch, as at some step it involves human intervention (as in, was
> the export good).
>
I would disagree that we need user interaction in the test suite. There
are already fully automated tests which (e.g., export to some backend
like html or tex and then programatically check for properties of the
exported results.
It is certainly likely that I am missing something, but I can't think of
a situation or a feature of Org-mode which could not be tested under the
current setup (mainly due to the fact that *every* user action in Emacs
reduces to a series of function calls which could be programatically
recreated).
>
> So maybe we can have a directory on the Worg website (not part of the
> Worg git repo) where every week or so the test suites will publish with
> what ever the org-mode head is at the time for all the supported
> formats. Then us "puny lisp illiterate" users can check up on it over
> the course of the week and report back to the list if there is a
> problem.
>
> Since this way people can look at the export formats they are
> interested in, none of the formats get treated like a step child
> either. Would that be feasible? Or did I completely misunderstand the
> problem at hand?
I'd think that a better way for contributing to the test suite in a
non-lisp manner would be to submit test cases, e.g. "this block of
Org-mode text should export to this but sometimes instead exports to
this", or "when I press this key sequence in this place in this org-mode
text I expect x to happen to the text".
We could even potentially leverage the existing Emacs macro system to
build a *record* method so that users could semi-automatically record
their actions allowing an interactive method of recording tests (or
submitting a re-creatable bug report). Or at least recording enough
information so that someone with a little bit more elisp-fu could wrap
the recorded actions into a unit test.
Hope this is helpful -- Eric
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed
2011-03-31 3:40 ` Eric Schulte
@ 2011-03-31 7:15 ` Rainer M Krug
0 siblings, 0 replies; 15+ messages in thread
From: Rainer M Krug @ 2011-03-31 7:15 UTC (permalink / raw)
To: Eric Schulte; +Cc: emacs-orgmode
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 31/03/11 05:40, Eric Schulte wrote:
> Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:
>
>> On Wed, 30 Mar 2011 15:42:19 -0600
>> "Eric Schulte" <schulte.eric@gmail.com> wrote:
>>
>>>>
>>>> This suite should actually be updated with effectively each patch
>>>> which introduces new features and run after each patch.
>>>>
>>>
>>> Agreed, in a perfect world...
>>>
>>>>
>>>> So is it only necessary to add meat to this framework?
>>>>
>>>
>>> Yes, I believe the best way forward would be to add tests to the
>>> existing framework.
>>
>> I have a possibly completely useless idea regarding "automatically"
>> checking regressions. As far I understand the problem now is its not
>> very feasible to do automated tests with what ever test suite we have
>> (or will have after the improvements) and see the exported results for
>> each patch, as at some step it involves human intervention (as in, was
>> the export good).
Good or not good is subjective - but *consistent* is not - and
consistency is important. Even if I do not like the default LaTeX output
from org, I can tweak it, but there is a problem if there are unexpected
changes in the export, which break my customizations or make it
difficult to recreate old documents, especially if these changes are not
documented.
>>
>
> I would disagree that we need user interaction in the test suite. There
> are already fully automated tests which (e.g., export to some backend
> like html or tex and then programatically check for properties of the
> exported results.
Exactly - the tests is R work this way: you have some code which is
executed, and then the resulting *output* is redirected into a file.
these results are then compared to a reference output, and if they are
not *identical* an error is raised.
Similar could be done in org: export to LaTeX should always result in
the same output, unless a change is intended (e.g. additional headers,
improvements, ...). So one could compare the resulting .tex file with a
reference .tex file for this test automatically, without user intervention.
>
> It is certainly likely that I am missing something, but I can't think of
> a situation or a feature of Org-mode which could not be tested under the
> current setup (mainly due to the fact that *every* user action in Emacs
> reduces to a series of function calls which could be programatically
> recreated).
>
>>
>> So maybe we can have a directory on the Worg website (not part of the
>> Worg git repo) where every week or so the test suites will publish with
>> what ever the org-mode head is at the time for all the supported
>> formats. Then us "puny lisp illiterate" users can check up on it over
>> the course of the week and report back to the list if there is a
>> problem.
>>
>> Since this way people can look at the export formats they are
>> interested in, none of the formats get treated like a step child
>> either. Would that be feasible? Or did I completely misunderstand the
>> problem at hand?
>
> I'd think that a better way for contributing to the test suite in a
> non-lisp manner would be to submit test cases, e.g. "this block of
> Org-mode text should export to this but sometimes instead exports to
> this", or "when I press this key sequence in this place in this org-mode
> text I expect x to happen to the text".
Correct - this is what we would, in addition to programmatic tests of
individual functions, need. I would actually say that the exports /
tangling / agendas / ... are the possibly the more important test cases,
as they 1) only show in a later stage of ones project, and 2) errors in
functions are easily detected by users and reported - and fizxed quite
quickly and finally 3) I guess an export / ... includes quite a lot of
functions, which are therefore tested as well (kind off...).
>
> We could even potentially leverage the existing Emacs macro system to
> build a *record* method so that users could semi-automatically record
> their actions allowing an interactive method of recording tests (or
> submitting a re-creatable bug report). Or at least recording enough
> information so that someone with a little bit more elisp-fu could wrap
> the recorded actions into a unit test.
That would be brilliant. Like the error reporting:
atach the current buffer, record what was done and *the individual
configuration of org / emacs* and finally email / upload it to an
address, where it is automatically added to other submitted test cases,
might bring us a long way closer to an very useful test base. I am
actually ot aware of any other test framework, which let's "normal"
users submit test cases via email / internet - I think that would be a
very useful addition.
>
> Hope this is helpful -- Eric
Most definitely,
Rainer
>
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)
Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa
Tel: +33 - (0)9 53 10 27 44
Cell: +27 - (0)8 39 47 90 42
Fax (SA): +27 - (0)8 65 16 27 82
Fax (D) : +49 - (0)3 21 21 25 22 44
Fax (FR): +33 - (0)9 58 10 27 44
email: Rainer@krugs.de
Skype: RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2UKhoACgkQoYgNqgF2egr8LACdEUpds2WnA9LYRHugLlQ9jrK5
dvoAn3A9qu3F5eWB5OFgYJlrgHds8BHW
=6OYS
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2011-03-31 7:15 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-03-30 13:01 Test framework needed Rainer M Krug
2011-03-30 13:46 ` Eric Abrahamsen
2011-03-30 13:56 ` Rainer M Krug
2011-03-30 14:11 ` Eric Abrahamsen
2011-03-30 14:22 ` Rainer M Krug
2011-03-30 14:26 ` MidLifeXis at PerlMonks
2011-03-30 14:18 ` Christian Egli
2011-03-30 14:30 ` Rainer M Krug
2011-03-30 15:13 ` Manuel Giraud
2011-03-30 20:14 ` Aankhen
2011-03-30 21:39 ` Eric Schulte
2011-03-30 21:42 ` Eric Schulte
2011-03-31 0:19 ` Suvayu Ali
2011-03-31 3:40 ` Eric Schulte
2011-03-31 7:15 ` Rainer M Krug
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).