[RFC] testing the org exporters

[RFC] testing the org exporters

[Jambunathan K]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]


I have been using the attached test.org file as part of my org-odt
related work.

The usual steps I follow for testing are quite simple:
1. Visit the file
2. Export it to xhtml or odt
3. Visually make sure that everything is OK.

I would like to add this file to the testing dir of the trunk. This file
would be a good starting point for not only validating but also
documenting (the quirks?) of various exporters.

At the moment, I am not very inclined to have it integrated with
ert[1]. Based on my experience, the standalone test.org file has proven
it's worth multiple times over during the development phase.

What do you think? I am happy to hear feedback on 

1. what to put in that test.org 
2. where to put the test.org 
3. how to have it integrated with test framework.



[-- Attachment #2: tests.tar --]
[-- Type: archive/tar, Size: 40960 bytes --]

[-- Attachment #3: Type: text/plain, Size: 79 bytes --]



Footnotes: 
[1]  This is mostly because I don't know nothing about ert.

-- 

Re: [RFC] testing the org exporters

[Eric Schulte]

Jambunathan K <kjambunathan@gmail.com> writes:

> I have been using the attached test.org file as part of my org-odt
> related work.
>
> The usual steps I follow for testing are quite simple:
> 1. Visit the file
> 2. Export it to xhtml or odt
> 3. Visually make sure that everything is OK.
>
> I would like to add this file to the testing dir of the trunk. This file
> would be a good starting point for not only validating but also
> documenting (the quirks?) of various exporters.
>
> At the moment, I am not very inclined to have it integrated with
> ert[1]. Based on my experience, the standalone test.org file has proven
> it's worth multiple times over during the development phase.
>
> What do you think? I am happy to hear feedback on 
>
> 1. what to put in that test.org 
> 2. where to put the test.org 
> 3. how to have it integrated with test framework.
>
> Footnotes: 
> [1]  This is mostly because I don't know nothing about ert.

Hi Jambunathan,

I would very much like to see this test file [1] integrated into the testing
repository of org-mode in (testing/examples) _and_ integrated into
Org-mode's ert test framework.

ERT is easy to use, and will more than repay any time spent applying it
to your test file through the ability to quickly programmatically and
reproducibly evaluate export results.  There are many tests [2] which
already implement a
1. visit example file
2. export to backend
3. confirm properties of backend (e.g., does or does not contain some string)

Please try running the Org-mode test suite and look at the examples in
testing/lisp.  I see no reason not to take this final step and begin
writing ERT tests which can serve as both documentation of issues and
insurance against regression.

Best -- Eric

Footnotes: 
[1]  I am not sure about including the org-mode unicorn png image in the
     repository as that could add alot of weight... perhaps .gitignore
     that file, point to a version of the file online, or generate an
     image on the fly with ditaa as part of the test suite

[2]  for example those in testing/lisp/test-ob-exp.el

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [RFC] testing the org exporters

[Martyn Jago]

Hi Jambunathan

>> I have been using the attached test.org file as part of my org-odt
>> related work.
>>
>> The usual steps I follow for testing are quite simple:
>> 1. Visit the file
>> 2. Export it to xhtml or odt
>> 3. Visually make sure that everything is OK.
>>
>> I would like to add this file to the testing dir of the trunk. This file
>> would be a good starting point for not only validating but also
>> documenting (the quirks?) of various exporters.
>>
>> At the moment, I am not very inclined to have it integrated with
>> ert[1]. Based on my experience, the standalone test.org file has proven
>> it's worth multiple times over during the development phase.
>>
>> What do you think? I am happy to hear feedback on 
>>
>> 1. what to put in that test.org 
>> 2. where to put the test.org 
>> 3. how to have it integrated with test framework.
>>
>> Footnotes: 
>> [1]  This is mostly because I don't know nothing about ert.
>
>Hi Jambunathan,
>
>I would very much like to see this test file [1] integrated into the testing
>repository of org-mode in (testing/examples) _and_ integrated into
>Org-mode's ert test framework.
>
>ERT is easy to use, and will more than repay any time spent applying it
>to your test file through the ability to quickly programmatically and
>reproducibly evaluate export results.  There are many tests [2] which
>already implement a
>1. visit example file
>2. export to backend
>3. confirm properties of backend (e.g., does or does not 
>contain some string)
>
>Please try running the Org-mode test suite and look at the examples in
>testing/lisp.  I see no reason not to take this final step and begin
>writing ERT tests which can serve as both documentation of issues and
>insurance against regression.


I also think that this is a great idea. 

In fact I have recently discovered that the LilyPond 
(Music Score Engravement) community (gnu) do something like this to 
ensure their high-quality rendering is not regressively broken or degraded. 
AFAICR they generate visual snippets by continuous integration, and have 
a team of (non-programmer but heavy user) contributers analyse the visual
results in order to report any regressions to the development team. These
tests are performed weekly, and are intended to take no more than 20 minutes
per week of each quality-assurance members time. Sign-offs happen once per
week.

I also agree with you that this could aid in providing highly maintained
visual snippets (with simple source) for the provision of examples for
documentation purposes. And even where ert can be configured to do the
actual testing, it may be nice to generate the visual result for 
documentation purposes.

Perhaps this visual output could live in worg?

Being a user of ert I would be happy helping to set this up if the community
thought it would be of use.

( just my 10 pence ;)

Regards

Martyn