From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Tom Breton (Tehom)" Subject: Re: How you can help Date: Fri, 24 Oct 2008 17:09:23 -0400 (EDT) Message-ID: <1069.65.96.62.210.1224882563.squirrel@mail.panix.com> References: <967CE7ED-05E9-4031-9F3B-CFB826511554@alexanderonline.org> <87ljwfbdga.fsf@kassiopeya.MSHEIMNETZ> <752DA813-1B66-4FD1-B28E-3C23176BA13D@alexanderonline.org> <87r6679t1t.fsf@kassiopeya.MSHEIMNETZ> <877i7z9px8.fsf@kassiopeya.MSHEIMNETZ> Mime-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KtTuP-0000J2-WA for emacs-orgmode@gnu.org; Fri, 24 Oct 2008 17:09:26 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KtTuO-0000Gy-6V for emacs-orgmode@gnu.org; Fri, 24 Oct 2008 17:09:24 -0400 Received: from [199.232.76.173] (port=47912 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KtTuN-0000Go-UZ for emacs-orgmode@gnu.org; Fri, 24 Oct 2008 17:09:23 -0400 Received: from mail1.panix.com ([166.84.1.72]:56959) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1KtTuN-0000hk-Nk for emacs-orgmode@gnu.org; Fri, 24 Oct 2008 17:09:23 -0400 Received: from mailbackend.panix.com (mailbackend.panix.com [166.84.1.89]) by mail1.panix.com (Postfix) with ESMTP id 20A9D29407 for ; Fri, 24 Oct 2008 17:09:23 -0400 (EDT) Received: from mail.panix.com (localhost [127.0.0.1]) by mailbackend.panix.com (Postfix) with ESMTP id 14971DE84 for ; Fri, 24 Oct 2008 17:09:23 -0400 (EDT) In-Reply-To: <877i7z9px8.fsf@kassiopeya.MSHEIMNETZ> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Cc: emacs-orgmode Org-Mode > "Avdi Grimm" writes: > > A few points, from someone with a decent amount of testing background= : > > Jippie! Please stay with us for a few days :-D [...] As someone with also a decent amount of testing background, I'd like to answer also. > Did you work with unit-testing frameworks for elisp already? > Which one? I co-wrote regress.el (Wayne Mesard wrote the first version) and I wrote rtest. I've written some testing for allout.el before I switched over to org-mode (which is very nice, BTW, Carsten, and provides pretty much everything I wish-listed for allout mode). Of particular interest for testing org-mode, I wrote mockbuf.el, which mocks files and buffers so that tests can deal with files without changing anything permanently. > Recommendations? Hard for me to say objectively. I prefer my rtest but it's in flux right now. I wouldn't recommend the current public version, rtest 3.0 I've designed and half written a better version but I can't really ask you to wait until I get around to making a better version of rtest. There is a development version that works and does nice things; it's messy as hell right now and I'm embarrassed about that. It isn't public but I could put it up if you'll forgive its messy state, It has mockbuf, which I found indispensable in writing tests for allout.el. In fact, I mostly developed it for that, though all the allout-specific code is elsewhere. Also useful for writing tests: safe-equal.el, caselet.el, and select.cl, all in the ancillary directory. I would steer clear of frameworks that provide many of synonyms for equality tests, inequality tests, etc. You already know how to write (not (equal ...)) in elisp. Why learn the name and syntax of something like (shouldnt-be-equal A B)? Having a lot of that makes a package LOOK heftier but doesn't do anything more for you. IMO analyzing test failures should be done by inspecting the test form, not by making you write things in non-standard elisp that could be written in standard elisp. Using ert would mean requiring ewoc, which means tests wouldn't work on emacs 21. Too bad, because it has some appeal. > Could say something about the effort to get started with such a > framework? For rtest, right now you just need (a) the development version, (b) unzip it in your emacs-site-lisp or set the paths and autoloads manually if you prefer, (c) include the following in a file to be tested: (when (not (fboundp 'rtest:deftest)) (defmacro rtest:deftest (&rest dummy)) (defmacro rtest:if-avail (&rest dummy))) Then write tests as: (rtest:deftest function-name ("Docstring" (test-form) ;;Return a boolean ) ;;Repeat as wanted ) That will get you started, but really you'd want to get familiar with the stuff in mockbuf.el - which is so messy right now that I'm embarrassed about it, because it contains both old and new versions of many things in flux. Other advice: Make a directory to hold short examples of files that illustrate "before" and "after" states for the various transformations. Each unit (eg org-agenda.el, org-archive.el etc) should have its own subtree of examples. One thing I'd do early is build the following test helper functions: A macro that runs a test body inside a "let" of all the configuration options set to expected values. A function to compare vector-style strings. `equal' doesn't suffice. rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and safe-equal-compare-vectors, which IIRC suffice for this. The ability to write only the visible parts to a separate buffer was indispensable in testing allout's folding. You already have it in org-export-visible with the type that keeps the buffer, "?\ ". Then you can just compare contents. You will likely have a lot of what I call "output tests", tests of code that has the property that if the output looks right, it is right. IMO with output tests, you need to make it easy on yourself. TDD is all well and fine, but not if it makes you repeat yourself. If there is available output that looks right, capture it and use it as a base for comparison. That's not "pure" but it's far easier and it works right. If the available output isn't yet right, capture it anyways and tweak it to make it right - but this rarely happens. I automated that sort of thing for a Prolog test harness I wrote, works fine. You'd just write a test of a special type that produced and captured output. It would originally be ungraded, but you would then grade it, and it would remember the correct output and compare against it. I was planning to add the same to rtest and I partly did, but never finished it. Still on my todo list. > Can we add it without changing Org's code? If understand 2. correctly - yes? > rtest's tests like to be in the same file as the code. If you use the form above, the file still compiles and runs even when rtest is not available. It's possible to write the tests in another file, but then either tests aren't grouped together (must be run one by one) or you have to do extra work to group them. If they're in file, you can run every test in a package with `rtest:library'. The other really convenient way to run a test is rtest:defun-at-point. > If we can, would we loose quality/speed of tests? Not at all. Tom Breton (Tehom) PS: One more piece of advice: Make an auto-insert or at least a skeleton that inserts test headers, like the "(when (not (fboundp ..." stuff above. You'll be using it a fair bit, and it's real nice to have it just automatically be there when you start a new file. This advice is geared toward my way of working, though, with the test code in the same file as the functional code.