Testing

* Testing
@ 2010-03-01  8:54 Martin Pohlack
  2010-03-20  0:18 ` A testing framework (was: Testing) Martin Pohlack
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Pohlack @ 2010-03-01  8:54 UTC (permalink / raw)
  To: org-mode

Hi All,

I have been using org-mode for a while now and am quite happy with the
features it provides.  Occasionally, I update to the current git head
to get some fancy new features ;-).

I am, however, more and more reluctant to do so out of fear of loosing
parts of my data due to newly introduced bugs.

Just to be clear, I'm not writing here to blame anyone for bugs.  I
understand that they are probably inevitable on the path to new
features.

In the last time I often found long-corrupted entries in rarely
expanded subtrees of my org files.  That kind of scares me as it
greatly reduces the trust I can have in “my system”.

For example, I started using mrd-org-smartwrap.el for wrapping long
lines.  As it turned out (much later) some irregularities in this mode
distort the parsing of headline / entry boundaries.  Consequently, I
corrupted some entries in my trees without noticing it for a long time.
One particularly nasty effect was that recurring items where finally
marked as DONE with mrd-org-smartwrap active, instead of jumping to
the next recurrence.

What I want to propose here is to have a collection of test cases
which could be used to verify org-mode's integrity with regard to
certain features.  This email is meant to start a discussion on this
topic.

I currently envision each test-case to have five parts:

1. An input file: This file is the basis for a test and contains the
   initial state of an org-mode file.
2. A sequence of actions to operate on the input file.
3. An expected output file: This file resembles the transformed input
   file after the action sequence.  The actual output file can be
   compared to this expected file.
4. An expected visual representation of the org-mode buffer in emacs
   after the action sequence (like a screenshot but character based).
   This is required to capture visibility related bugs.  Those would
   not be visible from the actual output file alone.
5. An environment with an init.el / .emacs to customize org-mode for a
   certain test run.

So 1., 2., and 5. are the input of the test process.  The test succeeds
if the captured visual representation equals 4 and if the captured
output file equals 3.

Let me give you an example:

1.
---------->8---------->8---------->8---------->8---------->8----------
* TODO test
  SCHEDULED: <2010-03-01 Mo +1w>
---------->8---------->8---------->8---------->8---------->8----------

2.
---------->8---------->8---------->8---------->8---------->8----------
(beginning-of-buffer)
(org-todo 'done)
(save-buffer)
---------->8---------->8---------->8---------->8---------->8----------

3.
---------->8---------->8---------->8---------->8---------->8----------
* TODO test
  SCHEDULED: <2010-03-08 Mo +1w>
  - State "WAIT"       from "TODO"       [2010-03-01 Mo 09:45]
  :PROPERTIES:
  :LAST_REPEAT: [2010-03-01 Mo 09:45]
  :END:
---------->8---------->8---------->8---------->8---------->8----------

4.
---------->8---------->8---------->8---------->8---------->8----------
* TODO test
  SCHEDULED: <2010-03-08 Mo +1w>
  - State "WAIT"       from "TODO"       [2010-03-01 Mo 09:45]
  :PROPERTIES:…
---------->8---------->8---------->8---------->8---------->8----------

5.
(My current setup emitted for brevity.)

1. and 3. may have to be extended to a set of files if more complex
features should be tested like querying several files or export to PDF
/ ….

3. and 4. may need some wild-card representation for the timestamps in
order to make the equality comparison portable.

I imagine that bug reports could be accompanied by such test cases.
An example, the expected output, and a sequence of actions should
belong there anyways :-).

What do you guys think?

Cheers,
Martin Pohlack

^ permalink raw reply	[flat|nested] 4+ messages in thread