From: "Eric Schulte" <schulte.eric@gmail.com>
To: Suvayu Ali <fatkasuvayu+linux@gmail.com>
Cc: emacs-orgmode@gnu.org
Subject: Re: Re: Test framework needed
Date: Wed, 30 Mar 2011 21:40:59 -0600 [thread overview]
Message-ID: <87wrjgq7n8.fsf@gmail.com> (raw)
In-Reply-To: <20110330171938.16343937@kuru.homelinux.net> (Suvayu Ali's message of "Wed, 30 Mar 2011 17:19:38 -0700")
Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:
> On Wed, 30 Mar 2011 15:42:19 -0600
> "Eric Schulte" <schulte.eric@gmail.com> wrote:
>
>> >
>> > This suite should actually be updated with effectively each patch
>> > which introduces new features and run after each patch.
>> >
>>
>> Agreed, in a perfect world...
>>
>> >
>> > So is it only necessary to add meat to this framework?
>> >
>>
>> Yes, I believe the best way forward would be to add tests to the
>> existing framework.
>
> I have a possibly completely useless idea regarding "automatically"
> checking regressions. As far I understand the problem now is its not
> very feasible to do automated tests with what ever test suite we have
> (or will have after the improvements) and see the exported results for
> each patch, as at some step it involves human intervention (as in, was
> the export good).
>
I would disagree that we need user interaction in the test suite. There
are already fully automated tests which (e.g., export to some backend
like html or tex and then programatically check for properties of the
exported results.
It is certainly likely that I am missing something, but I can't think of
a situation or a feature of Org-mode which could not be tested under the
current setup (mainly due to the fact that *every* user action in Emacs
reduces to a series of function calls which could be programatically
recreated).
>
> So maybe we can have a directory on the Worg website (not part of the
> Worg git repo) where every week or so the test suites will publish with
> what ever the org-mode head is at the time for all the supported
> formats. Then us "puny lisp illiterate" users can check up on it over
> the course of the week and report back to the list if there is a
> problem.
>
> Since this way people can look at the export formats they are
> interested in, none of the formats get treated like a step child
> either. Would that be feasible? Or did I completely misunderstand the
> problem at hand?
I'd think that a better way for contributing to the test suite in a
non-lisp manner would be to submit test cases, e.g. "this block of
Org-mode text should export to this but sometimes instead exports to
this", or "when I press this key sequence in this place in this org-mode
text I expect x to happen to the text".
We could even potentially leverage the existing Emacs macro system to
build a *record* method so that users could semi-automatically record
their actions allowing an interactive method of recording tests (or
submitting a re-creatable bug report). Or at least recording enough
information so that someone with a little bit more elisp-fu could wrap
the recorded actions into a unit test.
Hope this is helpful -- Eric
next prev parent reply other threads:[~2011-03-31 3:41 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-30 13:01 Test framework needed Rainer M Krug
2011-03-30 13:46 ` Eric Abrahamsen
2011-03-30 13:56 ` Rainer M Krug
2011-03-30 14:11 ` Eric Abrahamsen
2011-03-30 14:22 ` Rainer M Krug
2011-03-30 14:26 ` MidLifeXis at PerlMonks
2011-03-30 14:18 ` Christian Egli
2011-03-30 14:30 ` Rainer M Krug
2011-03-30 15:13 ` Manuel Giraud
2011-03-30 20:14 ` Aankhen
2011-03-30 21:39 ` Eric Schulte
2011-03-30 21:42 ` Eric Schulte
2011-03-31 0:19 ` Suvayu Ali
2011-03-31 3:40 ` Eric Schulte [this message]
2011-03-31 7:15 ` Rainer M Krug
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87wrjgq7n8.fsf@gmail.com \
--to=schulte.eric@gmail.com \
--cc=emacs-orgmode@gnu.org \
--cc=fatkasuvayu+linux@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).