From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martin Pohlack Subject: Testing Date: Mon, 01 Mar 2010 09:54:51 +0100 Message-ID: <4B8B80DB.8040804@os.inf.tu-dresden.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Nm1Nn-0004mi-Iy for emacs-orgmode@gnu.org; Mon, 01 Mar 2010 03:53:43 -0500 Received: from [140.186.70.92] (port=47034 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Nm1Nm-0004mO-G7 for emacs-orgmode@gnu.org; Mon, 01 Mar 2010 03:53:43 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Nm1Nl-0005f0-CH for emacs-orgmode@gnu.org; Mon, 01 Mar 2010 03:53:42 -0500 Received: from os.inf.tu-dresden.de ([141.76.48.99]:34664) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Nm1Nl-0005eg-2b for emacs-orgmode@gnu.org; Mon, 01 Mar 2010 03:53:41 -0500 Received: from [93.218.187.242] (helo=[192.168.2.100]) by os.inf.tu-dresden.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.71) id 1Nm1Ni-0006z8-Js for emacs-orgmode@gnu.org; Mon, 01 Mar 2010 09:53:38 +0100 List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: org-mode Hi All, I have been using org-mode for a while now and am quite happy with the features it provides. Occasionally, I update to the current git head to get some fancy new features ;-). I am, however, more and more reluctant to do so out of fear of loosing parts of my data due to newly introduced bugs. Just to be clear, I'm not writing here to blame anyone for bugs. I understand that they are probably inevitable on the path to new features. In the last time I often found long-corrupted entries in rarely expanded subtrees of my org files. That kind of scares me as it greatly reduces the trust I can have in “my system”. For example, I started using mrd-org-smartwrap.el for wrapping long lines. As it turned out (much later) some irregularities in this mode distort the parsing of headline / entry boundaries. Consequently, I corrupted some entries in my trees without noticing it for a long time. One particularly nasty effect was that recurring items where finally marked as DONE with mrd-org-smartwrap active, instead of jumping to the next recurrence. What I want to propose here is to have a collection of test cases which could be used to verify org-mode's integrity with regard to certain features. This email is meant to start a discussion on this topic. I currently envision each test-case to have five parts: 1. An input file: This file is the basis for a test and contains the initial state of an org-mode file. 2. A sequence of actions to operate on the input file. 3. An expected output file: This file resembles the transformed input file after the action sequence. The actual output file can be compared to this expected file. 4. An expected visual representation of the org-mode buffer in emacs after the action sequence (like a screenshot but character based). This is required to capture visibility related bugs. Those would not be visible from the actual output file alone. 5. An environment with an init.el / .emacs to customize org-mode for a certain test run. So 1., 2., and 5. are the input of the test process. The test succeeds if the captured visual representation equals 4 and if the captured output file equals 3. Let me give you an example: 1. ---------->8---------->8---------->8---------->8---------->8---------- * TODO test SCHEDULED: <2010-03-01 Mo +1w> ---------->8---------->8---------->8---------->8---------->8---------- 2. ---------->8---------->8---------->8---------->8---------->8---------- (beginning-of-buffer) (org-todo 'done) (save-buffer) ---------->8---------->8---------->8---------->8---------->8---------- 3. ---------->8---------->8---------->8---------->8---------->8---------- * TODO test SCHEDULED: <2010-03-08 Mo +1w> - State "WAIT" from "TODO" [2010-03-01 Mo 09:45] :PROPERTIES: :LAST_REPEAT: [2010-03-01 Mo 09:45] :END: ---------->8---------->8---------->8---------->8---------->8---------- 4. ---------->8---------->8---------->8---------->8---------->8---------- * TODO test SCHEDULED: <2010-03-08 Mo +1w> - State "WAIT" from "TODO" [2010-03-01 Mo 09:45] :PROPERTIES:… ---------->8---------->8---------->8---------->8---------->8---------- 5. (My current setup emitted for brevity.) 1. and 3. may have to be extended to a set of files if more complex features should be tested like querying several files or export to PDF / …. 3. and 4. may need some wild-card representation for the timestamps in order to make the equality comparison portable. I imagine that bug reports could be accompanied by such test cases. An example, the expected output, and a sequence of actions should belong there anyways :-). What do you guys think? Cheers, Martin Pohlack