From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martin Pohlack Subject: Re: A testing framework Date: Sun, 21 Mar 2010 20:48:19 +0100 Message-ID: <4BA67803.8040701@os.inf.tu-dresden.de> References: <4B8B80DB.8040804@os.inf.tu-dresden.de> <4BA41471.6080902@os.inf.tu-dresden.de> <45E96F29-7015-47AD-9271-9B9914C968E8@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NtR6g-0000pu-N4 for emacs-orgmode@gnu.org; Sun, 21 Mar 2010 15:46:42 -0400 Received: from [140.186.70.92] (port=40185 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NtR6f-0000mI-1z for emacs-orgmode@gnu.org; Sun, 21 Mar 2010 15:46:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1NtR6d-0001TF-6m for emacs-orgmode@gnu.org; Sun, 21 Mar 2010 15:46:40 -0400 Received: from os.inf.tu-dresden.de ([141.76.48.99]:44506) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1NtR6d-0001T9-0O for emacs-orgmode@gnu.org; Sun, 21 Mar 2010 15:46:39 -0400 In-Reply-To: <45E96F29-7015-47AD-9271-9B9914C968E8@gmail.com> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Carsten Dominik Cc: org-mode Hi Carsten, all, On 21.03.2010 08:26, Carsten Dominik wrote: > we had an earlier threads about testing frameworks: > > http://orgmode.org/worg/org-tests/index.php > > http://thread.gmane.org/gmane.emacs.orgmode/8759/focus=8775 > > http://thread.gmane.org/gmane.emacs.orgmode/8743/focus=8743 Very interesting, I just went through all this. >From above material, two general approaches have been considered for testing org-mode: * Lisp based unit testing. * Example-file based feature testing. I am targeting the second approach here for now as I think it may provide more benefit per time spent and would allow more people to participate easily. I hope that every emacs user will be able to write tests with my framework. Basically only an example test case and the expected result should be need for tests, together with a minimal description of the setup. If only an example file triggering a bug is available, a corresponding test can be crafted with minimal effort. For now I have a minimal framework running with two tests demonstrating expected behavior for folding and a bug in the version I'm using. I would like to enquire comments regarding the following open issues, where I have no satisfying solution yet: * I would like to start the test in something like a temporally stable setup, such that generated timestamps would be somewhat predictable (basically with the system clock set to 1.1.2005 20:00, or something like that). There are several levels where this could be achieved, ranging from: 1. Experimental patches to the Linux kernel, 2. Library wrappers using LD_PRELOAD around emacs to virtualize the time and gettimeofday system calls, 3. Advised lisp functions for obtaining time stamps, and 4. Post processing the test output to generalized time stamps. Currently, I'm going with 4, 3 may make sense if someone could identify the hopefully 2-3 functions to be advised. 2 and 1 are probably too Linux specific and I'm not sure if 2 works with emacs. Comments? * Currently I use a list of lisp functions for specifying the behavior to be executed in a test, for example: (org-todo) or (org-end-of-line) (org-cycle) This is not very intuitive for users. I would rather like something like a recorded macro to execute. For example: "end, tab" Does anyone here has a tip how to specify a list of user-input actions in a file and play that back? * For visual comparison I currently simply save the visible part of buffers. One may get a step further here and also capture properties, faces and the cursor position. I remember vaguely that there is such a facility somewhat related to rich text in emacs, but wasn't able to find it again. Do you guys know of such a thing? > I think it would be great to have a testing framework and a > lot of tests, but I do not have time to make this happen. > Maybe you can try to get the people who were active in those > threads and energize them? I assumed everyone still interested in contributing to org-mode is still lurking in this list. So speak up if you are interested in contributing to the tests :-). Cheers, Martin