From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Eric Schulte" Subject: Re: Re: Test framework needed Date: Wed, 30 Mar 2011 21:40:59 -0600 Message-ID: <87wrjgq7n8.fsf@gmail.com> References: <4D9329BC.6000106@gmail.com> <87r59og1pt.fsf@ericabrahamsen.net> <4D93369C.7010608@gmail.com> <87vcz03d4p.fsf@sbs.ch> <4D933E9E.7030907@gmail.com> <87hbakp9hw.fsf@gmail.com> <20110330171938.16343937@kuru.homelinux.net> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from [140.186.70.92] (port=32846 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Q58ku-0007kY-1f for emacs-orgmode@gnu.org; Wed, 30 Mar 2011 23:41:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q58kr-0004JS-OF for emacs-orgmode@gnu.org; Wed, 30 Mar 2011 23:41:07 -0400 Received: from mail-gy0-f169.google.com ([209.85.160.169]:44142) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q58kr-0004J8-Gp for emacs-orgmode@gnu.org; Wed, 30 Mar 2011 23:41:05 -0400 Received: by gyd8 with SMTP id 8so967517gyd.0 for ; Wed, 30 Mar 2011 20:41:04 -0700 (PDT) In-Reply-To: <20110330171938.16343937@kuru.homelinux.net> (Suvayu Ali's message of "Wed, 30 Mar 2011 17:19:38 -0700") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Suvayu Ali Cc: emacs-orgmode@gnu.org Suvayu Ali writes: > On Wed, 30 Mar 2011 15:42:19 -0600 > "Eric Schulte" wrote: > >> > >> > This suite should actually be updated with effectively each patch >> > which introduces new features and run after each patch. >> > >> >> Agreed, in a perfect world... >> >> > >> > So is it only necessary to add meat to this framework? >> > >> >> Yes, I believe the best way forward would be to add tests to the >> existing framework. > > I have a possibly completely useless idea regarding "automatically" > checking regressions. As far I understand the problem now is its not > very feasible to do automated tests with what ever test suite we have > (or will have after the improvements) and see the exported results for > each patch, as at some step it involves human intervention (as in, was > the export good). > I would disagree that we need user interaction in the test suite. There are already fully automated tests which (e.g., export to some backend like html or tex and then programatically check for properties of the exported results. It is certainly likely that I am missing something, but I can't think of a situation or a feature of Org-mode which could not be tested under the current setup (mainly due to the fact that *every* user action in Emacs reduces to a series of function calls which could be programatically recreated). > > So maybe we can have a directory on the Worg website (not part of the > Worg git repo) where every week or so the test suites will publish with > what ever the org-mode head is at the time for all the supported > formats. Then us "puny lisp illiterate" users can check up on it over > the course of the week and report back to the list if there is a > problem. > > Since this way people can look at the export formats they are > interested in, none of the formats get treated like a step child > either. Would that be feasible? Or did I completely misunderstand the > problem at hand? I'd think that a better way for contributing to the test suite in a non-lisp manner would be to submit test cases, e.g. "this block of Org-mode text should export to this but sometimes instead exports to this", or "when I press this key sequence in this place in this org-mode text I expect x to happen to the text". We could even potentially leverage the existing Emacs macro system to build a *record* method so that users could semi-automatically record their actions allowing an interactive method of recording tests (or submitting a re-creatable bug report). Or at least recording enough information so that someone with a little bit more elisp-fu could wrap the recorded actions into a unit test. Hope this is helpful -- Eric