From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Eric Schulte" Subject: Re: Re: Testing --- again... Date: Sat, 02 Oct 2010 12:12:37 -0600 Message-ID: <87fwwoihuy.fsf@gmail.com> References: <87vd5li75s.fsf@gmx.de> <0CDCC1C3-3D93-465E-827C-A722978F4D13@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from [140.186.70.92] (port=44448 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P26Zh-000353-TY for emacs-orgmode@gnu.org; Sat, 02 Oct 2010 14:12:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P26Ze-00078Y-8k for emacs-orgmode@gnu.org; Sat, 02 Oct 2010 14:12:43 -0400 Received: from mail-px0-f169.google.com ([209.85.212.169]:60480) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P26Ze-00078Q-1F for emacs-orgmode@gnu.org; Sat, 02 Oct 2010 14:12:42 -0400 Received: by pxi5 with SMTP id 5so2455865pxi.0 for ; Sat, 02 Oct 2010 11:12:41 -0700 (PDT) In-Reply-To: <0CDCC1C3-3D93-465E-827C-A722978F4D13@gmail.com> (Carsten Dominik's message of "Sat, 2 Oct 2010 10:09:03 +0200") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Carsten Dominik Cc: Emacs-orgmode mailing list Hi, This is exciting. Rather than impose a complete directory/layout schema before-hand I'd lean towards starting with a little more chaos and then letting the structure of the test directory develop /naturally/. From the discussion below it sounds like an initial structure of testing/lisp/ testing/contrib/lisp/ may make sense, reserving the top level for "meta" testing stuff, like functions for running tests, common fixtures, example files, etc... I have two questions. 1) while waiting for ert to be included into Emacs, should we include an ert distribution as part of the Org-mode repository (maybe using git sub-modules) or should we just agree that users should have a certain version of ert installed locally? I'm honestly not sure which of these options sounds preferable. 2) should the initial population of the testing/ directory take place in a separate branch of the repository or in the master branch? Again I don't know which I would prefer, branches add complication but could result in cleaner commit histories. Best -- Eric Carsten Dominik writes: > Hi Sebastian, > > the lack of a testing suite for Org-mode is really frustrating, > and even more frustrating is that we have had like seven attempts > to start one, and each of these lead to nothing. So I would > be perfectly happy to give a free hand, write access to the repo > and a full directory in the distribution to implement one. > Once there is a framework, I am sure many people would be > willing to contribute tests. > > More comments below. > > On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote: > >> Hi, >> >> >> I thought about testing again recently. This is something, that never >> really got started. For a reason: there's no framework for testing. >> >> I therefore wrote a very rough proposal, found on >> http://github.com/SebastianRose/org-test >> >> The idea is, to provide two simple commands: >> >> >> * org-test-test-current-defun >> will search for tests for the defun point is in or behind >> (`beginning-of-defun') and execute them surrounded by >> >> (let ((select (or selector "^org")) >> (deactivate-mark nil)) >> (save-excursion >> (save-match-data >> >> >> * org-test-test-buffer-file >> will search for tests for the entire file and execute them the >> same >> way. > > FIrst: I have *no* clue about testing. > > Second, I am surprised that you want to structure it by function. I > would have > thought that it could be structure by file at the most. And then > there will > be tests that involve code from many files. > > But I guess > >> >> If you use one of these commands, all currently registered ERT tests >> are >> deleted, and files are reloaded (since you're likely to work on the >> tests, too). To repeat the tests without reloading, you will use the >> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the >> ERT >> results buffer. >> >> >> >> I choose ERT (git clone http://github.com/ohler/ert.git) because >> that's >> likely to go into Emacs core (or elpa.gnu.org). >> >> >> >> >> The idea is to search the directory structure from the current source >> file upwards for a directory named "tests/" if it exists. Else ask >> the >> user. Similar to what `add-change-log-entry' does. >> >> Below that directory, a tree like the source tree exists: >> >> project >> +-- lisp/ >> | +-- a.el >> | `-- b/ >> | +-- b.el >> | >> `-- tests/ >> +-- a.el/ >> | +-- tests.el >> | `-- a-defun.el >> `-- b/ >> +-- b.el/ >> +-- tests.el >> `-- b-defun.el >> >> If this setup exists, when editing defun-x in lisp/a.el, >> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el >> (fallback: tests.el there) and execute all tests with selector >> "^a-defun". > > Well, OK, this is fine. But under a.el and b.el there should also be > general tests that are not function dependent, and there should be a > place > to put tests that you do not want to assign to a specific file. > > We do have a "testing" directory already, you can use that. > I would prefer the tests to be in testing, not in lisp/testing > if possible. I would like to have the lisp directory contain > only code. If possible. > > It would be OK to have a lisp subdirectory in testing, > just as it would be OK to have contrib/lisp in testing > for the contributed packages. > > PLEASE, go ahead. I do not think you have write access > yet on repo - give me your user name and I'll activate you. > > - Carsten > >> `M-x org-test-test-buffer-file' in that same source file will load all >> *.el files in tests/a.el/ and execute all ERT tests for selector "^a". >> >> >> Thus tests for >> org-mode/lisp/org-protocol.el >> will be searched in the directory >> org-mode/tests/lisp/org-protocol.el/*.el >> >> >> Once the basic route of testing is clear, I'd like to "translate" the >> existing tests for org-html.el to work with ERT, which will involve >> writing more tools (create output buffers, compare output with control >> files using ediff etc.). I know Lennart Borgman has wrote that stuff >> for nXhtml already. I hope we can use his stuff and help here. >> >> The directory org-mode/lisp/tests/ would not need to be part of the >> "official" Org mode package. It could as well be checked out >> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.). >> >> >> Any thoughts? >> >> >> >> >> Sebastian > > > _______________________________________________ > Emacs-orgmode mailing list > Please use `Reply All' to send replies to the list. > Emacs-orgmode@gnu.org > http://lists.gnu.org/mailman/listinfo/emacs-orgmode