Re: Re: How you can help

[Richard Riley <rileyrgdev@googlemail.com>]

Sebastian Rose <sebastian_rose@gmx.de> writes:

> Hi Richard,
>
>
> Richard Riley <rileyrgdev@googlemail.com> writes:
>>> Added this one to the Clippboard section on new org-tests/index.org in
>>> Worg.git. (this section will be temporary...)
>>
>> Something like the above should only be a link (at most) to the emacs
>> manual. Reproducing standard info is bad in the long run in case things
>> in the base product (emacs) change for example.
>
>
> On the long run, yes. I just added this section, to start the page. And
> since it's just STARTing... this was in the first or second reaction
> that came in. Feel free to edit ('go wild' ;-))!
>
> I wouldn't bother to have several pages like this one will be (it's
> supposed to be an index on the long run), each covering another way of
> testing.
>
> I just meant to take _ACT_. Scepticism is a good thing, as long as it
> doesn't stop you from doing.

Of course not. 

>
>
>>>
>>>> Some kind of regression testing framework would be awesome.  Org-mode is
>>>> large enough that this is almost a necessity to keep things stable and
>>>> bug-free.
>>>
>>> It's big and feature-RICH.
>>
>> The nature of OSS means that the community using the product keep it
>> stable and bug free. I dont think the efforts to produce meaningful
>> regression tests would be beneficial in an ever morphing product like
>> org-mode. Clearly my humble opinion on that one :-;
>
>
> We do test our software by using it. But the bug in the HTML-exporter
> that Carsten has fixed two days ago, was introduced in early September
> and would be in 6.10, which is supposed to be in the emacs 23 release.
>
> A very simple test plan would have revealed it.

The question is : define simple. I am not against test plans, but the
nature of such means a process in place where such test plans are
carried out. Carsten knows his SW inside out and would find it hard to
apply test cases every time a bug is fixed for time alone. Bugs happen
in SW and the nature (excuse the repetition) of OSS is that we, the
users, are kind of the testers and users.

>>>> Maybe something can be put together from the git testing framework and
>>>> use of emacs -batch to process test org files and verify the output is
>>>> as expected (with diff or some other tool).
>>>
>>>
>>> Hey, diff is a good idea!!
>>>
>>> I didn't take the verification of the output into account yet :-)
>>>
>>> I just pushed a change of Worgs start page, and added a directory
>>> 'org-tests'. I've placed an index.org there, which now is just a
>>> collection of ideas (I'm on my day job, so I can't really work on it
>>> now).
>>>
>>> Don't know how often the git repo is published.
>>> Bernt and Ben, are you 'worgers' allready?
>>>
>>> Do you think it makes sense to add snippets and ideas to the new page in
>>> Worg? I think while the list great to exchange ideas, it's good to have
>>> a place, where all those ideas are destilled to one-liners.
>>
>> I must say I am dubious about this. It means, for the tests to be
>> meaningful, that the output must be  a fixed format in base org.
>
>
> Why? If the test bails out 'ERROR', I will have to look for the
> reason. If the format changed in a legal way => adjust the test.


What is an error? And for whom? That the categories are one more tab to
the right? I know I am probably sounding a bit of a killjoy here but
having been involved in closed source SW development with dedicated QA
and testing teams it was hard enough then. Maybe I am just being a bit
of an old curmudgeon here :-(

>
>
>> I doubt
>> this will ever be the case. The presentation will fluctuate while the
>> core information (dates, schedules periods etc) will remain pretty much
>> constant.
>>
>> The majority of bugs that I see are often down to people misusing or
>> using things in the base which are not fully explored. No amount of
>> regression testing can cover things like that unless the regression
>> tests include everyones customisations.
>
> Yes, because Carsten add features en masse :-)
> I see the testing differently. In the first place we need THINK of
> testing.

I think we all do. And the main contributors do too. But it is
unrealistic to expect a full regression test for each and every release
unless there is someone willing to step up to the mark and take the time
to perform them. I personally feel that enough take the "cutting edge"
and report back before the "official" cuts. We are all beat testers. Its
the nature of being an emacs and OSS user in the first place :-;

>
> New Org-revision out?
> Ahh, OK, I have to the HTML-exports, I want to be working.
>
> To do this, I need several different setups, several different data
> directories (Org-files) and an easy way to test, that doesn't eat my
> time and gives a result. The quality may vary, but ERRORs will be
> detected for shure.
>
> Not one or mounths later - immediately.

If you don't find it immediately then how often do you use the feature?
Now I know I probably sound negative. I assure I am not.  Maybe the
earlier word "sceptic" is more applicable ..

>
> If someone installs emacs 23, tries to export to HTML and gets an
> error.... 

I have been there too. In fact you found a bug that I left hanging for
ages because I found a work around ( dont publish from anywhere other
than the root of the project) - so thanks for your perseverance on that.

I think the best thing people can do is keep a stable version and also
test the CVS (oops) GIT version too very now and again.

>
>
>> Do I think regression testing is important? Yes - in certain
>> environments. But every time Carsten, you, myself or anyone else fires
>> up org-mode we are already doing just that.
>
> Yes, but we can do better, easier and more complete.

Possibly. I look forward to seeing proposed solutions  and would gladly
lend some time to applying them.

-- 
God never made his work for man to mend.
~John Dryden