From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Alexander Subject: Re: Re: How you can help Date: Thu, 23 Oct 2008 17:22:22 +0100 Message-ID: <024FF120-2C9B-449D-A2FA-279D53A47892@alexanderonline.org> References: <967CE7ED-05E9-4031-9F3B-CFB826511554@alexanderonline.org> <878wsfpgtp.fsf@gollum.intra.norang.ca> <877i7zbbe4.fsf@kassiopeya.MSHEIMNETZ> Mime-Version: 1.0 (Apple Message framework v929.2) Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Kt2xT-0003uR-44 for emacs-orgmode@gnu.org; Thu, 23 Oct 2008 12:22:47 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Kt2xR-0003tU-6m for emacs-orgmode@gnu.org; Thu, 23 Oct 2008 12:22:46 -0400 Received: from [199.232.76.173] (port=53465 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Kt2xR-0003tM-0W for emacs-orgmode@gnu.org; Thu, 23 Oct 2008 12:22:45 -0400 Received: from harley.gnservers.com ([72.52.242.16]:43361) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Kt2xQ-0000ab-Lq for emacs-orgmode@gnu.org; Thu, 23 Oct 2008 12:22:45 -0400 In-Reply-To: List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Richard Riley Cc: emacs-orgmode Org-Mode On 2008-Oct-23, at 16:49, Richard Riley wrote: > Sebastian Rose writes: > >> Bernt Hansen writes: >>> Running a minimal emacs should suppress custom config files: >>> emacs -q -l yourtest.el >> >> Added this one to the Clippboard section on new org-tests/index.org >> in >> Worg.git. (this section will be temporary...) > > Something like the above should only be a link (at most) to the emacs > manual. Reproducing standard info is bad in the long run in case > things > in the base product (emacs) change for example. > >> >>> Some kind of regression testing framework would be awesome. Org- >>> mode is >>> large enough that this is almost a necessity to keep things stable >>> and >>> bug-free. >> >> It's big and feature-RICH. > > The nature of OSS means that the community using the product keep it > stable and bug free. I dont think the efforts to produce meaningful > regression tests would be beneficial in an ever morphing product like > org-mode. Clearly my humble opinion on that one :-; > I'm a bit clue-free on what the words mean. I think of regression tests as ways of defining the way "Things Should Work(TM)" so if someone commits as fix to org-mode that works fine on Debian, but breaks on Windows, there's a way for someone on Windows to easily (brainlessly) run the test and report back to the developer. I don't think of org-mode as 'ever morphing'. There are some features that are stable, and should remain so. A regression test would be valuable, presumably, for developers working on a new feature (don't screw up important existing functionality!) > > I must say I am dubious about this. It means, for the tests to be > meaningful, that the output must be a fixed format in base org. I > doubt > this will ever be the case. The presentation will fluctuate while the > core information (dates, schedules periods etc) will remain pretty > much > constant. > I agree that the presentation is fluid and it was hard for me to imagine how to test it in an automated way. But I disagree (I think) in that having a test on the presentation is extremely powerful. I'm thinking that one example of how to write such a test could be copied when someone is trying to explain what they want to happen, and how it is not happening right now. Perhaps such a test would have a very limited useful life. I'd like to think that it could live and be useful for a very long time (that's why thinking about configuration/user customization issues in a test is important). But even if the test is only useful until the bug is fixed (or the feature implemented) , so be it. > The majority of bugs that I see are often down to people misusing or > using things in the base which are not fully explored. No amount of > regression testing can cover things like that unless the regression > tests include everyones customisations. > Not everyone's customizations, just every customization org-mode has. I mean, if the feature is there, it should work. If the feature is important, it should work in the same way on every platform. The trick -- and I hope it's either solved or easy -- is exactly how to show that "the feature works the same way as it should, right now, on this platform" If I understand you properly, you're just saying that the user customization creates so much variability in the output, there isn't anyway you could capture all the possible presentations in a single test. Well, yeah, so user customizations need to be left out of the test case. Except that in the scenario I'm thinking of where someone (a mere mortal using org-mode, with limited lisp knowledge) is trying to report a bug, user customization will be critical. My first thought was asking org-mode to dump all the customization variables it has into a buffer (as buffer-local variables maybe?) would help someone else recreate the bug. (By starting org-mode with NO customizations, except what's found in the test case) Or are you suggesting that the source of presentation variation is due mostly to OS, Emacs version, or something else beyond the scope of our tests, so they couldn't be replicated on another machine? > Do I think regression testing is important? Yes - in certain > environments. But every time Carsten, you, myself or anyone else fires > up org-mode we are already doing just that. > > > -- > Inventor: A person who makes an ingenious arrangement of wheels, > levers and springs, and believes it civilization. ~Ambrose Bierce, > The Devil's Dictionary