From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martyn Jago Subject: Re: [RFC] testing the org exporters Date: Mon, 4 Jul 2011 09:39:44 +0000 (UTC) Message-ID: References: <81iprmxk5x.fsf@gmail.com> <87k4c1lpx9.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Return-path: Received: from eggs.gnu.org ([140.186.70.92]:55894) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QdfdI-0004lG-Ud for emacs-orgmode@gnu.org; Mon, 04 Jul 2011 05:40:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QdfdG-0002r4-L0 for emacs-orgmode@gnu.org; Mon, 04 Jul 2011 05:40:00 -0400 Received: from lo.gmane.org ([80.91.229.12]:45789) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QdfdG-0002qt-2M for emacs-orgmode@gnu.org; Mon, 04 Jul 2011 05:39:58 -0400 Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1QdfdD-0005xC-Bm for emacs-orgmode@gnu.org; Mon, 04 Jul 2011 11:39:55 +0200 Received: from 88-96-171-142.dsl.zen.co.uk ([88.96.171.142]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 04 Jul 2011 11:39:55 +0200 Received: from martyn.jago by 88-96-171-142.dsl.zen.co.uk with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 04 Jul 2011 11:39:55 +0200 List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: emacs-orgmode@gnu.org Hi Jambunathan >> I have been using the attached test.org file as part of my org-odt >> related work. >> >> The usual steps I follow for testing are quite simple: >> 1. Visit the file >> 2. Export it to xhtml or odt >> 3. Visually make sure that everything is OK. >> >> I would like to add this file to the testing dir of the trunk. This file >> would be a good starting point for not only validating but also >> documenting (the quirks?) of various exporters. >> >> At the moment, I am not very inclined to have it integrated with >> ert[1]. Based on my experience, the standalone test.org file has proven >> it's worth multiple times over during the development phase. >> >> What do you think? I am happy to hear feedback on >> >> 1. what to put in that test.org >> 2. where to put the test.org >> 3. how to have it integrated with test framework. >> >> Footnotes: >> [1] This is mostly because I don't know nothing about ert. > >Hi Jambunathan, > >I would very much like to see this test file [1] integrated into the testing >repository of org-mode in (testing/examples) _and_ integrated into >Org-mode's ert test framework. > >ERT is easy to use, and will more than repay any time spent applying it >to your test file through the ability to quickly programmatically and >reproducibly evaluate export results. There are many tests [2] which >already implement a >1. visit example file >2. export to backend >3. confirm properties of backend (e.g., does or does not >contain some string) > >Please try running the Org-mode test suite and look at the examples in >testing/lisp. I see no reason not to take this final step and begin >writing ERT tests which can serve as both documentation of issues and >insurance against regression. I also think that this is a great idea. In fact I have recently discovered that the LilyPond (Music Score Engravement) community (gnu) do something like this to ensure their high-quality rendering is not regressively broken or degraded. AFAICR they generate visual snippets by continuous integration, and have a team of (non-programmer but heavy user) contributers analyse the visual results in order to report any regressions to the development team. These tests are performed weekly, and are intended to take no more than 20 minutes per week of each quality-assurance members time. Sign-offs happen once per week. I also agree with you that this could aid in providing highly maintained visual snippets (with simple source) for the provision of examples for documentation purposes. And even where ert can be configured to do the actual testing, it may be nice to generate the visual result for documentation purposes. Perhaps this visual output could live in worg? Being a user of ert I would be happy helping to set this up if the community thought it would be of use. ( just my 10 pence ;) Regards Martyn