Re: Re: How you can help

[Ben Alexander <bva@alexanderonline.org>]

On 2008-Oct-23, at 16:49, Richard Riley wrote:

> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> Bernt Hansen <bernt@norang.ca> writes:
>>> Running a minimal emacs should suppress custom config files:
>>>         emacs -q -l yourtest.el
>>
>> Added this one to the Clippboard section on new org-tests/index.org  
>> in
>> Worg.git. (this section will be temporary...)
>
> Something like the above should only be a link (at most) to the emacs
> manual. Reproducing standard info is bad in the long run in case  
> things
> in the base product (emacs) change for example.
>
>>
>>> Some kind of regression testing framework would be awesome.  Org- 
>>> mode is
>>> large enough that this is almost a necessity to keep things stable  
>>> and
>>> bug-free.
>>
>> It's big and feature-RICH.
>
> The nature of OSS means that the community using the product keep it
> stable and bug free. I dont think the efforts to produce meaningful
> regression tests would be beneficial in an ever morphing product like
> org-mode. Clearly my humble opinion on that one :-;
>
I'm a bit clue-free on what the words mean.  I think of regression  
tests as ways of defining the way "Things Should Work(TM)" so if  
someone commits as fix to org-mode that works fine on Debian, but  
breaks on Windows, there's a way for someone on Windows to easily  
(brainlessly) run the test and report back to the developer.

I don't think of org-mode as 'ever morphing'.  There are some features  
that are stable, and should remain so.  A regression test would be  
valuable, presumably, for developers working on a new feature (don't  
screw up important existing functionality!)
>
> I must say I am dubious about this. It means, for the tests to be
> meaningful, that the output must be  a fixed format in base org. I  
> doubt
> this will ever be the case. The presentation will fluctuate while the
> core information (dates, schedules periods etc) will remain pretty  
> much
> constant.
>
I agree that the presentation is fluid and it was hard for me to  
imagine how to test it in an automated way.  But I disagree (I think)  
in that having a test on the presentation is extremely powerful.

I'm thinking that one example of how to write such a test could be  
copied when someone is trying to explain what they want to happen, and  
how it is not happening right now.

Perhaps such a test would have a very limited useful life.  I'd like  
to think that it could live and be useful for a very long time (that's  
why thinking about configuration/user customization issues in a test  
is important).  But even if the test is only useful until the bug is  
fixed (or the feature implemented) , so be it.

> The majority of bugs that I see are often down to people misusing or
> using things in the base which are not fully explored. No amount of
> regression testing can cover things like that unless the regression
> tests include everyones customisations.
>
Not everyone's customizations, just every customization org-mode has.   
I mean, if the feature is there, it should work.  If the feature is  
important, it should work in the same way on every platform.

The trick -- and I hope it's either solved or easy -- is exactly how  
to show that "the feature works the same way as it should, right now,  
on this platform"

If I understand you properly, you're just saying that the user  
customization creates so much variability in the output, there isn't  
anyway you could capture all the possible presentations in a single  
test.  Well, yeah, so user customizations need to be left out of the  
test case.

Except that in the scenario I'm thinking of where someone (a mere  
mortal using org-mode, with limited lisp knowledge) is trying to  
report a bug, user customization will be critical.  My first thought  
was asking org-mode to dump all the customization variables it has  
into a buffer (as buffer-local variables maybe?) would help someone  
else recreate the bug. (By starting org-mode with NO customizations,  
except what's found in the test case)

Or are you suggesting that the source of presentation variation is due  
mostly to OS, Emacs version, or something else beyond the scope of our  
tests, so they couldn't be replicated on another machine?

> Do I think regression testing is important? Yes - in certain
> environments. But every time Carsten, you, myself or anyone else fires
> up org-mode we are already doing just that.
>
>
> -- 
> Inventor:  A person who makes an ingenious arrangement of wheels,  
> levers and springs, and believes it civilization.  ~Ambrose Bierce,  
> The Devil's Dictionary