* Test framework needed @ 2011-03-30 13:01 Rainer M Krug 2011-03-30 13:46 ` Eric Abrahamsen 0 siblings, 1 reply; 15+ messages in thread From: Rainer M Krug @ 2011-03-30 13:01 UTC (permalink / raw) To: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I was bitten again from an unintended regression in org-mode, and that the second time in two weeks. I am probably not the right person to suggest this, but I think it is time to introduce a test framework for org-mode, to ensure that the (without doubt useful) approach to develop org-mode does not lead to regressions. org is, as I see it, complex and is getting broader and broader - time management, task management, literate programming, recipes (I like the idea), all have different (possibly contradicting?) requirements. And I think it is getting more and more difficult for new contributors to have the overview to make sure that something they are contributing actually does not break something else. As I think it would be *really bad* idea to split org, I think that it needs to be assured that no regressions are introduced. Unfortunately I have no idea about elisp and can therefore not contribute to the framework itself, but I guess all of us could contribute small org files as test cases and expected resulting outputs (exports, tangling, agenda files, ...). If the framework could accommodate those, everybody could help to bring together a (definitely not perfect) test framework, that would bring org at least one step closer to error-free growth. I have to add I really love org and babel and have no idea what I would use instead for literal programming in R. Cheers, Rainer - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel: +33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TKbwACgkQoYgNqgF2egpQJwCdG7C+3Ft+xGyLBiwY6aBwMLaC TIIAn0RD3ypbvkfE1RAL4FK3ZwmnERdE =GlJt -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed 2011-03-30 13:01 Test framework needed Rainer M Krug @ 2011-03-30 13:46 ` Eric Abrahamsen 2011-03-30 13:56 ` Rainer M Krug 0 siblings, 1 reply; 15+ messages in thread From: Eric Abrahamsen @ 2011-03-30 13:46 UTC (permalink / raw) To: emacs-orgmode On Wed, Mar 30 2011, Rainer M Krug wrote: > Hi > > I was bitten again from an unintended regression in org-mode, and that > the second time in two weeks. > > I am probably not the right person to suggest this, but I think it is > time to introduce a test framework for org-mode, to ensure that the > (without doubt useful) approach to develop org-mode does not lead to > regressions. This would be the page to start with, though the most likely candidate (Elisp Regression Testing) is only available in Emacs trunk at the moment… http://www.emacswiki.org/emacs/UnitTesting E ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 13:46 ` Eric Abrahamsen @ 2011-03-30 13:56 ` Rainer M Krug 2011-03-30 14:11 ` Eric Abrahamsen 2011-03-30 14:18 ` Christian Egli 0 siblings, 2 replies; 15+ messages in thread From: Rainer M Krug @ 2011-03-30 13:56 UTC (permalink / raw) To: Eric Abrahamsen; +Cc: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/03/11 15:46, Eric Abrahamsen wrote: > On Wed, Mar 30 2011, Rainer M Krug wrote: > >> Hi >> >> I was bitten again from an unintended regression in org-mode, and that >> the second time in two weeks. >> >> I am probably not the right person to suggest this, but I think it is >> time to introduce a test framework for org-mode, to ensure that the >> (without doubt useful) approach to develop org-mode does not lead to >> regressions. > > This would be the page to start with, though the most likely candidate > (Elisp Regression Testing) is only available in Emacs trunk at the > moment… > > http://www.emacswiki.org/emacs/UnitTesting Am I right in assuming, that all of the possible test frameworks would require org files and the expected output (tengle, export to ..., agenda, ...)? In this case, would it make sense to start collecting those, as they can easily be user contributed, consequently representing a cross section of the use cases (even not intended use cases)? Rainer > > E > > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel: +33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TNpwACgkQoYgNqgF2egpnWACeMFmP1p7mKsMXsELXcFSEqW99 mMkAnivsDT7zmguzphqziXzoKdx5fqz3 =AWd5 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed 2011-03-30 13:56 ` Rainer M Krug @ 2011-03-30 14:11 ` Eric Abrahamsen 2011-03-30 14:22 ` Rainer M Krug 2011-03-30 14:26 ` MidLifeXis at PerlMonks 2011-03-30 14:18 ` Christian Egli 1 sibling, 2 replies; 15+ messages in thread From: Eric Abrahamsen @ 2011-03-30 14:11 UTC (permalink / raw) To: emacs-orgmode On Wed, Mar 30 2011, Rainer M Krug wrote: > On 30/03/11 15:46, Eric Abrahamsen wrote: >> On Wed, Mar 30 2011, Rainer M Krug wrote: >> >>> Hi >>> >>> I was bitten again from an unintended regression in org-mode, and that >>> the second time in two weeks. >>> >>> I am probably not the right person to suggest this, but I think it is >>> time to introduce a test framework for org-mode, to ensure that the >>> (without doubt useful) approach to develop org-mode does not lead to >>> regressions. >> >> This would be the page to start with, though the most likely candidate >> (Elisp Regression Testing) is only available in Emacs trunk at the >> moment… >> >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Yup, what you would need is some org source files that exercise all of the possible export options (for testing export, for example), including weird edge cases, and then ERT (if that's what we ended up using) would provide handy functions for making sure the export output matches expectations. The excellent gentleman who created the ODT exporter, whose name currently escapes me, has already created test files for his exporter—that would be a perfect place to start. Covering all of org's various functions would end up being a bit of a PITA, though you're quite right that it's an excellent idea, and will become more and more necessary. E ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 14:11 ` Eric Abrahamsen @ 2011-03-30 14:22 ` Rainer M Krug 2011-03-30 14:26 ` MidLifeXis at PerlMonks 1 sibling, 0 replies; 15+ messages in thread From: Rainer M Krug @ 2011-03-30 14:22 UTC (permalink / raw) To: Eric Abrahamsen; +Cc: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/03/11 16:11, Eric Abrahamsen wrote: > On Wed, Mar 30 2011, Rainer M Krug wrote: > >> On 30/03/11 15:46, Eric Abrahamsen wrote: >>> On Wed, Mar 30 2011, Rainer M Krug wrote: >>> >>>> Hi >>>> >>>> I was bitten again from an unintended regression in org-mode, and that >>>> the second time in two weeks. >>>> >>>> I am probably not the right person to suggest this, but I think it is >>>> time to introduce a test framework for org-mode, to ensure that the >>>> (without doubt useful) approach to develop org-mode does not lead to >>>> regressions. >>> >>> This would be the page to start with, though the most likely candidate >>> (Elisp Regression Testing) is only available in Emacs trunk at the >>> moment… >>> >>> http://www.emacswiki.org/emacs/UnitTesting >> >> Am I right in assuming, that all of the possible test frameworks would >> require org files and the expected output (tengle, export to ..., >> agenda, ...)? In this case, would it make sense to start collecting >> those, as they can easily be user contributed, consequently representing >> a cross section of the use cases (even not intended use cases)? > > Yup, what you would need is some org source files that exercise all of > the possible export options (for testing export, for example), including > weird edge cases, and then ERT (if that's what we ended up using) would > provide handy functions for making sure the export output matches > expectations. The excellent gentleman who created the ODT exporter, > whose name currently escapes me, has already created test files for his > exporter—that would be a perfect place to start. > > Covering all of org's various functions would end up being a bit of a > PITA, though you're quite right that it's an excellent idea, and will > become more and more necessary. So would there be a possibility of "normal org users" (if there is such a thing ...) to contribute to this? What would be needed? Any specific structure of the org files? Rainer > > E > > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel: +33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TPJgACgkQoYgNqgF2egoWKwCeMjWgggD7JMhVTrQTHe3f7n6s VhgAn2CG25hOa1Q4RPufarreQVYlezHm =18s2 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 14:11 ` Eric Abrahamsen 2011-03-30 14:22 ` Rainer M Krug @ 2011-03-30 14:26 ` MidLifeXis at PerlMonks 1 sibling, 0 replies; 15+ messages in thread From: MidLifeXis at PerlMonks @ 2011-03-30 14:26 UTC (permalink / raw) To: emacs-orgmode As a heavy Perl user, writing /automated/ tests is a large part of my dev work. I would suggest / plea / encourage that whatever framework is used can be automated. If it cannot be run as part of an automated process it is not going to be run. Also consider a set of testing platforms (emacs version, supporting versions of other .el modules, OS version, external software). There are many dependencies that org has - being able to automate this testing is a must. Just my $0.02. Brian / MidLifeXis ----- Original Message ---- From: Eric Abrahamsen <eric@ericabrahamsen.net> To: emacs-orgmode@gnu.org Sent: Wed, March 30, 2011 9:11:23 AM Subject: [O] Re: Test framework needed On Wed, Mar 30 2011, Rainer M Krug wrote: > On 30/03/11 15:46, Eric Abrahamsen wrote: >> On Wed, Mar 30 2011, Rainer M Krug wrote: >> >>> Hi >>> >>> I was bitten again from an unintended regression in org-mode, and that >>> the second time in two weeks. >>> >>> I am probably not the right person to suggest this, but I think it is >>> time to introduce a test framework for org-mode, to ensure that the >>> (without doubt useful) approach to develop org-mode does not lead to >>> regressions. >> >> This would be the page to start with, though the most likely candidate >> (Elisp Regression Testing) is only available in Emacs trunk at the >> moment… >> >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Yup, what you would need is some org source files that exercise all of the possible export options (for testing export, for example), including weird edge cases, and then ERT (if that's what we ended up using) would provide handy functions for making sure the export output matches expectations. The excellent gentleman who created the ODT exporter, whose name currently escapes me, has already created test files for his exporter—that would be a perfect place to start. Covering all of org's various functions would end up being a bit of a PITA, though you're quite right that it's an excellent idea, and will become more and more necessary. E ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Test framework needed 2011-03-30 13:56 ` Rainer M Krug 2011-03-30 14:11 ` Eric Abrahamsen @ 2011-03-30 14:18 ` Christian Egli 2011-03-30 14:30 ` Rainer M Krug 1 sibling, 1 reply; 15+ messages in thread From: Christian Egli @ 2011-03-30 14:18 UTC (permalink / raw) To: emacs-orgmode Rainer M Krug <r.m.krug@gmail.com> writes: >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Before you go too far with this; Orgmode already contains a unit test suite. Look at the README in the testing directory (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) -- Christian Egli Swiss Library for the Blind, Visually Impaired and Print Disabled Grubenstrasse 12, CH-8045 Zürich, Switzerland ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 14:18 ` Christian Egli @ 2011-03-30 14:30 ` Rainer M Krug 2011-03-30 15:13 ` Manuel Giraud 2011-03-30 21:42 ` Eric Schulte 0 siblings, 2 replies; 15+ messages in thread From: Rainer M Krug @ 2011-03-30 14:30 UTC (permalink / raw) To: Christian Egli; +Cc: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/03/11 16:18, Christian Egli wrote: > Rainer M Krug <r.m.krug@gmail.com> writes: > >>> http://www.emacswiki.org/emacs/UnitTesting >> >> Am I right in assuming, that all of the possible test frameworks would >> require org files and the expected output (tengle, export to ..., >> agenda, ...)? In this case, would it make sense to start collecting >> those, as they can easily be user contributed, consequently representing >> a cross section of the use cases (even not intended use cases)? > > Before you go too far with this; Orgmode already contains a unit test > suite. Look at the README in the testing directory > (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) > But it does not look as if it is used very often... There are not many test org files, and I did not se anything which compares the resulting exported / tangle file with an expected output? Please correct me if I am missing something. This suite should actually be updated with effectively each patch which introduces new features and run after each patch. So is it only necessary to add meat to this framework? Rainer - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel: +33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TPp4ACgkQoYgNqgF2egpBZQCfQN+g3J2+BiyKaWzh6o847mZS PR4An0aW0Di6vfgzwsJgwfK4FajL65WP =9/CW -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 14:30 ` Rainer M Krug @ 2011-03-30 15:13 ` Manuel Giraud 2011-03-30 20:14 ` Aankhen 2011-03-30 21:42 ` Eric Schulte 1 sibling, 1 reply; 15+ messages in thread From: Manuel Giraud @ 2011-03-30 15:13 UTC (permalink / raw) To: Rainer M Krug; +Cc: emacs-orgmode, Christian Egli Rainer M Krug <r.m.krug@gmail.com> writes: > But it does not look as if it is used very often... There are not many > test org files, and I did not se anything which compares the resulting > exported / tangle file with an expected output? It could be useful to have this kind of framework for org but I think it is difficult problem when it comes to org export. For example, is removing the "validate xhtml" button from an html export make this export invalid? I guess not. > Please correct me if I am missing something. > > This suite should actually be updated with effectively each patch which > introduces new features and run after each patch. Which renders this framework far less automatic. I think that having a set of org files against which one could try any export and *see* that the results are almost correct would be enough. -- Manuel Giraud ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 15:13 ` Manuel Giraud @ 2011-03-30 20:14 ` Aankhen 2011-03-30 21:39 ` Eric Schulte 0 siblings, 1 reply; 15+ messages in thread From: Aankhen @ 2011-03-30 20:14 UTC (permalink / raw) To: Manuel Giraud, Rainer M Krug, emacs-orgmode, Christian Egli On Wed, Mar 30, 2011 at 20:43, Manuel Giraud <manuel.giraud@univ-nantes.fr> wrote: > Rainer M Krug <r.m.krug@gmail.com> writes: > [snip] > >> Please correct me if I am missing something. >> >> This suite should actually be updated with effectively each patch which >> introduces new features and run after each patch. > > Which renders this framework far less automatic. I think that having a > set of org files against which one could try any export and *see* that > the results are almost correct would be enough. I think the “automated” part refers to running the tests. What you suggest—having a set of files that you could manually export to verify the results—wouldn’t be of much help, IMHO. First, it’d require a lot more time than executing a single command and checking the summary at the end. Second, it’d be very error-prone. A comprehensive automated test suite gives people writing patches an easier way to perform regression testing and catch any unintended consequences. On the other hand, it /does/ take a lot of effort to keep it in sync with the codebase… maybe we need a Test Fairy. ;-) Aankhen ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 20:14 ` Aankhen @ 2011-03-30 21:39 ` Eric Schulte 0 siblings, 0 replies; 15+ messages in thread From: Eric Schulte @ 2011-03-30 21:39 UTC (permalink / raw) To: Aankhen; +Cc: Christian Egli, emacs-orgmode, Manuel Giraud, Rainer M Krug Aankhen <aankhen@gmail.com> writes: > On Wed, Mar 30, 2011 at 20:43, Manuel Giraud > <manuel.giraud@univ-nantes.fr> wrote: >> Rainer M Krug <r.m.krug@gmail.com> writes: >> [snip] >> >>> Please correct me if I am missing something. >>> >>> This suite should actually be updated with effectively each patch which >>> introduces new features and run after each patch. >> >> Which renders this framework far less automatic. I think that having a >> set of org files against which one could try any export and *see* that >> the results are almost correct would be enough. > I think an "automatic" solution would involve running a batch Emacs process which loads up Org-mode and then loads and runs the existing test suite. This would be both easily automatable and would allow testing of various versions of Emacs with relative ease. Best -- Eric ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 14:30 ` Rainer M Krug 2011-03-30 15:13 ` Manuel Giraud @ 2011-03-30 21:42 ` Eric Schulte 2011-03-31 0:19 ` Suvayu Ali 1 sibling, 1 reply; 15+ messages in thread From: Eric Schulte @ 2011-03-30 21:42 UTC (permalink / raw) To: Rainer M Krug; +Cc: emacs-orgmode, Christian Egli Rainer M Krug <r.m.krug@gmail.com> writes: > On 30/03/11 16:18, Christian Egli wrote: >> Rainer M Krug <r.m.krug@gmail.com> writes: >> >>>> http://www.emacswiki.org/emacs/UnitTesting >>> >>> Am I right in assuming, that all of the possible test frameworks would >>> require org files and the expected output (tengle, export to ..., >>> agenda, ...)? In this case, would it make sense to start collecting >>> those, as they can easily be user contributed, consequently representing >>> a cross section of the use cases (even not intended use cases)? >> >> Before you go too far with this; Orgmode already contains a unit test >> suite. Look at the README in the testing directory >> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) >> > > But it does not look as if it is used very often... There are not many > test org files, and I did not se anything which compares the resulting > exported / tangle file with an expected output? > > Please correct me if I am missing something. > You are correct that the existing test suite needs some attention and some more tests. Just a general commitment to convert problem reports from the mailing list into unit tests should be a step in the right direction. However the existing test suite (while under populated) is the result of multiple previous discussion of this nature on the mailing list, and I think that abandoning the existing infrastructure would be a step in the wrong direction. > > This suite should actually be updated with effectively each patch which > introduces new features and run after each patch. > Agreed, in a perfect world... > > So is it only necessary to add meat to this framework? > Yes, I believe the best way forward would be to add tests to the existing framework. Best -- Eric > > Rainer ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-30 21:42 ` Eric Schulte @ 2011-03-31 0:19 ` Suvayu Ali 2011-03-31 3:40 ` Eric Schulte 0 siblings, 1 reply; 15+ messages in thread From: Suvayu Ali @ 2011-03-31 0:19 UTC (permalink / raw) To: emacs-orgmode On Wed, 30 Mar 2011 15:42:19 -0600 "Eric Schulte" <schulte.eric@gmail.com> wrote: > > > > This suite should actually be updated with effectively each patch > > which introduces new features and run after each patch. > > > > Agreed, in a perfect world... > > > > > So is it only necessary to add meat to this framework? > > > > Yes, I believe the best way forward would be to add tests to the > existing framework. I have a possibly completely useless idea regarding "automatically" checking regressions. As far I understand the problem now is its not very feasible to do automated tests with what ever test suite we have (or will have after the improvements) and see the exported results for each patch, as at some step it involves human intervention (as in, was the export good). So maybe we can have a directory on the Worg website (not part of the Worg git repo) where every week or so the test suites will publish with what ever the org-mode head is at the time for all the supported formats. Then us "puny lisp illiterate" users can check up on it over the course of the week and report back to the list if there is a problem. Since this way people can look at the export formats they are interested in, none of the formats get treated like a step child either. Would that be feasible? Or did I completely misunderstand the problem at hand? -- Suvayu Open source is the future. It sets us free. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-31 0:19 ` Suvayu Ali @ 2011-03-31 3:40 ` Eric Schulte 2011-03-31 7:15 ` Rainer M Krug 0 siblings, 1 reply; 15+ messages in thread From: Eric Schulte @ 2011-03-31 3:40 UTC (permalink / raw) To: Suvayu Ali; +Cc: emacs-orgmode Suvayu Ali <fatkasuvayu+linux@gmail.com> writes: > On Wed, 30 Mar 2011 15:42:19 -0600 > "Eric Schulte" <schulte.eric@gmail.com> wrote: > >> > >> > This suite should actually be updated with effectively each patch >> > which introduces new features and run after each patch. >> > >> >> Agreed, in a perfect world... >> >> > >> > So is it only necessary to add meat to this framework? >> > >> >> Yes, I believe the best way forward would be to add tests to the >> existing framework. > > I have a possibly completely useless idea regarding "automatically" > checking regressions. As far I understand the problem now is its not > very feasible to do automated tests with what ever test suite we have > (or will have after the improvements) and see the exported results for > each patch, as at some step it involves human intervention (as in, was > the export good). > I would disagree that we need user interaction in the test suite. There are already fully automated tests which (e.g., export to some backend like html or tex and then programatically check for properties of the exported results. It is certainly likely that I am missing something, but I can't think of a situation or a feature of Org-mode which could not be tested under the current setup (mainly due to the fact that *every* user action in Emacs reduces to a series of function calls which could be programatically recreated). > > So maybe we can have a directory on the Worg website (not part of the > Worg git repo) where every week or so the test suites will publish with > what ever the org-mode head is at the time for all the supported > formats. Then us "puny lisp illiterate" users can check up on it over > the course of the week and report back to the list if there is a > problem. > > Since this way people can look at the export formats they are > interested in, none of the formats get treated like a step child > either. Would that be feasible? Or did I completely misunderstand the > problem at hand? I'd think that a better way for contributing to the test suite in a non-lisp manner would be to submit test cases, e.g. "this block of Org-mode text should export to this but sometimes instead exports to this", or "when I press this key sequence in this place in this org-mode text I expect x to happen to the text". We could even potentially leverage the existing Emacs macro system to build a *record* method so that users could semi-automatically record their actions allowing an interactive method of recording tests (or submitting a re-creatable bug report). Or at least recording enough information so that someone with a little bit more elisp-fu could wrap the recorded actions into a unit test. Hope this is helpful -- Eric ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Re: Test framework needed 2011-03-31 3:40 ` Eric Schulte @ 2011-03-31 7:15 ` Rainer M Krug 0 siblings, 0 replies; 15+ messages in thread From: Rainer M Krug @ 2011-03-31 7:15 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/03/11 05:40, Eric Schulte wrote: > Suvayu Ali <fatkasuvayu+linux@gmail.com> writes: > >> On Wed, 30 Mar 2011 15:42:19 -0600 >> "Eric Schulte" <schulte.eric@gmail.com> wrote: >> >>>> >>>> This suite should actually be updated with effectively each patch >>>> which introduces new features and run after each patch. >>>> >>> >>> Agreed, in a perfect world... >>> >>>> >>>> So is it only necessary to add meat to this framework? >>>> >>> >>> Yes, I believe the best way forward would be to add tests to the >>> existing framework. >> >> I have a possibly completely useless idea regarding "automatically" >> checking regressions. As far I understand the problem now is its not >> very feasible to do automated tests with what ever test suite we have >> (or will have after the improvements) and see the exported results for >> each patch, as at some step it involves human intervention (as in, was >> the export good). Good or not good is subjective - but *consistent* is not - and consistency is important. Even if I do not like the default LaTeX output from org, I can tweak it, but there is a problem if there are unexpected changes in the export, which break my customizations or make it difficult to recreate old documents, especially if these changes are not documented. >> > > I would disagree that we need user interaction in the test suite. There > are already fully automated tests which (e.g., export to some backend > like html or tex and then programatically check for properties of the > exported results. Exactly - the tests is R work this way: you have some code which is executed, and then the resulting *output* is redirected into a file. these results are then compared to a reference output, and if they are not *identical* an error is raised. Similar could be done in org: export to LaTeX should always result in the same output, unless a change is intended (e.g. additional headers, improvements, ...). So one could compare the resulting .tex file with a reference .tex file for this test automatically, without user intervention. > > It is certainly likely that I am missing something, but I can't think of > a situation or a feature of Org-mode which could not be tested under the > current setup (mainly due to the fact that *every* user action in Emacs > reduces to a series of function calls which could be programatically > recreated). > >> >> So maybe we can have a directory on the Worg website (not part of the >> Worg git repo) where every week or so the test suites will publish with >> what ever the org-mode head is at the time for all the supported >> formats. Then us "puny lisp illiterate" users can check up on it over >> the course of the week and report back to the list if there is a >> problem. >> >> Since this way people can look at the export formats they are >> interested in, none of the formats get treated like a step child >> either. Would that be feasible? Or did I completely misunderstand the >> problem at hand? > > I'd think that a better way for contributing to the test suite in a > non-lisp manner would be to submit test cases, e.g. "this block of > Org-mode text should export to this but sometimes instead exports to > this", or "when I press this key sequence in this place in this org-mode > text I expect x to happen to the text". Correct - this is what we would, in addition to programmatic tests of individual functions, need. I would actually say that the exports / tangling / agendas / ... are the possibly the more important test cases, as they 1) only show in a later stage of ones project, and 2) errors in functions are easily detected by users and reported - and fizxed quite quickly and finally 3) I guess an export / ... includes quite a lot of functions, which are therefore tested as well (kind off...). > > We could even potentially leverage the existing Emacs macro system to > build a *record* method so that users could semi-automatically record > their actions allowing an interactive method of recording tests (or > submitting a re-creatable bug report). Or at least recording enough > information so that someone with a little bit more elisp-fu could wrap > the recorded actions into a unit test. That would be brilliant. Like the error reporting: atach the current buffer, record what was done and *the individual configuration of org / emacs* and finally email / upload it to an address, where it is automatically added to other submitted test cases, might bring us a long way closer to an very useful test base. I am actually ot aware of any other test framework, which let's "normal" users submit test cases via email / internet - I think that would be a very useful addition. > > Hope this is helpful -- Eric Most definitely, Rainer > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel: +33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2UKhoACgkQoYgNqgF2egr8LACdEUpds2WnA9LYRHugLlQ9jrK5 dvoAn3A9qu3F5eWB5OFgYJlrgHds8BHW =6OYS -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2011-03-31 7:15 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-03-30 13:01 Test framework needed Rainer M Krug 2011-03-30 13:46 ` Eric Abrahamsen 2011-03-30 13:56 ` Rainer M Krug 2011-03-30 14:11 ` Eric Abrahamsen 2011-03-30 14:22 ` Rainer M Krug 2011-03-30 14:26 ` MidLifeXis at PerlMonks 2011-03-30 14:18 ` Christian Egli 2011-03-30 14:30 ` Rainer M Krug 2011-03-30 15:13 ` Manuel Giraud 2011-03-30 20:14 ` Aankhen 2011-03-30 21:39 ` Eric Schulte 2011-03-30 21:42 ` Eric Schulte 2011-03-31 0:19 ` Suvayu Ali 2011-03-31 3:40 ` Eric Schulte 2011-03-31 7:15 ` Rainer M Krug
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).