From: Ben Alexander <bva@alexanderonline.org>
To: Sebastian Rose <sebastian_rose@gmx.de>
Cc: emacs-orgmode Org-Mode <emacs-orgmode@gnu.org>
Subject: Re: How you can help
Date: Thu, 23 Oct 2008 15:55:41 +0100 [thread overview]
Message-ID: <752DA813-1B66-4FD1-B28E-3C23176BA13D@alexanderonline.org> (raw)
In-Reply-To: <87ljwfbdga.fsf@kassiopeya.MSHEIMNETZ>
Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting
Unfortunately for me, I can't tell if Emacs comes with any builtin
framework already, so I downloaded one of the many options listed on
that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el
The personal issue I have is that I'm on a Mac, using Aquamacs, and
the command line version of emacs is a different binary, so there
might be trouble when a test passes at the command line, but fails
where it matters to me. I don't even make sure that org-mode is up to
date at the command line. I thought it wasn't, but I just checked and
now it is. Plus, I don't really understand internals of emacs (like
basic internals: I understand point and mark, buffer and file, but not
transient mark, indirect buffer, symbols vs strings, window vs tab vs
frame)
The tutorial I'd need to write a test is one which lays out code I
could copy and paste to do the following
* setup the test environment
- create a test directory
- create a sample test.org file
- put the cursor in a particular place
* run the command we need to test
- hit the 'TAB' key, or C-c C-c (some folks might need to be
reminded how to find out exactly what command is actually being run
when you hit a keystroke. And some of me might need to be told what
lisp-code to use when the keystroke runs different commands at
different places in a file)
- reformat a table
- clock in/out
- create the agenda
- export .html .ics .dvi file
* How do we specify the correct result???
- check that the headline folded properly. What's the lisp code
for getting the folded string as displayed?
- check that the cursor is where it should be? especially when the
cursor is near elipses...
- check that the agenda is built properly. What's the lisp code
for getting the agenda as a string?
- check that the exported files are correct. Maybe the right
suggestion is to run the export on two different files, so the test
can focus on the 'diff' between them. That way different people who
run the same test on different hosts can get the same result.
More experienced folks must suggest whether it's better to have the
expectation of a test be specified as a string in the test code or as
a separate file in the test directory. It might depend on context, so
feel free to lay out three or four cases.
If even one kind of test for each of these 'cases' exists, hopefully
we can encourage people to report bugs with the test. I mean, if
people are already going through the effort to create the small sample
org file to demonstrate their bug, and it's only a short (obvious)
step to translating their english description of what should happen
into equivalent lisp code.... that seems like a big temptation for
someone to do it.
Plus it might help them convince themselves that it really is a bug,
and not that they are going crazy.
I gotta move on for the day here, but I'll keep thinking a bit about
this.
On 2008-Oct-23, at 15:20, Sebastian Rose wrote:
> Hi Ben,
>
> I cc'ed the list.
>
>
> The tests I described in my email to the list are not automated. The
> reason for that is my lack of (e)lisp knowledge.
>
> BUT, they where easy to handle for non programmers. I think the little
> test will make it to the worg site this week, when all private data is
> removed. You could take a look at it by then. And: improve, improve,
> improve... :-)
>
> If you know of someone who knows how to do automated tests in elisp,
> or
> some technique, package, whatever, please post it to the list, so we
> all
> can take a look at it and comment->decide something. This is _highly_
> _appreciated_.
>
>
> Ben Alexander <bva@alexanderonline.org> writes:
>> Sebastian Rose wrote:
>>> 5. I also think of little packages for testing parts of org.
>>
>> I'm curious if you or someone else has any ideas for writing
>> automated tests for
>> org-mode. I haven't the foggiest idea how someone would write a
>> test for the
>> parts of org that control what is displayed on the screen. I
>> mean, when the
>> bug is 'it doesn't look right' how can you tell?
>
> Believe me, my idea is the foggiest of all possible ideas ;-) (so
> where
> my ideas of JavaScript some mounth ago), so I won't be of much help, I
> fear.
>
>
>> Perhaps the git repository should have a small collection of small
>> org-
>> mode files that reproduce certain bugs? If there were some
>> examples of how to
>> create such a test, then perhaps bug reporters would find it much
>> easier to
>> create them.
>
> YES! Exactly. Every corner case an Org-file, every bug an Org-file.
> _DATA_ for testing is something, everyone _can_ provide.
>
> But git later, yes, maybe.
>
> Since this would need Carsten to think and act on this, I feel Worg
> is a
> nice place to start the first expieriments. We need Carstens power for
> other things (when will Org-mode finaly wash my car? It's repeater-
> TODO,
> but nothing happens!!!)
>
> Basically, I'd try to keep the the testing as simple as possible, and
> try to get elisp programmers to help with the code. We should try to
> keep the hurdles for testers as humble (?) as possible, and put all
> that's needed to be helpfull on one page in worg.
>
> I recently discovered the very unautomated `(print object)' in the
> elisp
> reference. Not everything can be done automated, maybe, but if I would
> have known this stupid `print' function, I would know more about elisp
> and some files in org already. And it would have been faster and
> easier
> to create the two minor patches I wrote. This is, where the 'links to
> elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy.
>
> Willingness to help is no problem, as the Org-community reveals. As
> for
> me, it's often a lack of time and/or knowledge, that prevents it.
>
> And the aim of all this is to help helping, in means of good and
> detailed bug reports in the first place.
>
>
>
>
>> I do see some confusing issues due to different configuration
>> files. So
>> creating a test file might involve making sure org-mode doesn't
>> read any
>> configuration (how do you do that?) and possible asking org-
>> mode to extract all the configuration variables it has right now
>> and dump them
>> into a test file (...and how do you do that?)
>
> True.
>
> Hm - to test without configuration, we already have `emacs -q'.
> Dumping the variables is just a list of (print var) statements. A
> (quite) complete list of variables could be extracted with grep?
>
> e.g.:
>
> grep -r defvar lisp/
>
> Once we have such a list, we could set/reset some/all variables.
>
> This will not be perfect, but could work reasonable well.
>
> And yes, it would indeed be nice, to have elisp to reliably reset
> emacs/org configuration, so one could do several different test in
> sequence. Preferably they would _always_ run and dump all errors to
> some
> file or buffer, even if one or more tests fail.
>
> I think elsip provides funtionalities to handle those errors and
> skip to
> the next test. This would be a huge step.
>
>
>
>
>
>
> But still, we have to test with different _DATA_ too! The test I
> described is just a quick hack I did to do the testing, while Carsten
> was bug-hunting on the other end of my internet connection.
>
> The XHTML-export test was much easier to do, then some other tests we
> might need. In the end, it was a test, automated or not. And I had
> to do
> it, because I'm one out of a few who use org to maintain a web-site
> (locally in my case) and export an entire directory tree with lots of
> _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This
> became obvious in this case. I was the one who even noticed the bug,
> which means, no one else was using recursive XHTML export for a while,
> or their setup didn't reveal that bug.
>
> So clearly I was the one to provide some support for this very part of
> the system. I can't rely on the assumption, that some maintainer has
> all
> possible setups at hand all the time (maybe this was possible years
> ago).
>
> While the testing of the HTML-export was quite simple, it requires a
> lot
> of _DATA_. Namly files and directories + different setups, per file
> setups, #+SETUPFILE, images, with/without org-info.js, extra styles
> set
> or not... to have a realistic testing environment (the test didn't
> have
> all of these...).
>
>
>
>
> A summary could be, that it's nice to provide different setups for
> maintainers and testers.
>
> - *Easy to install* (unpack and done),
> - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set
> everything up)
> - and *easy to download* once they are removed again (a central
> place on
> Worg).
> - *Corner cases*, like the 'empty line before first heading bug' in
> the
> LaTeX exporter recently.
>
>
>
>
> Regards,
>
>
> --
> Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449
> Hannover
>
> Tel.: +49 (0)511 - 36 58 472
> Fax: +49 (0)1805 - 233633 - 11044
> mobil: +49 (0)173 - 83 93 417
> Email: s.roseATemma-stil.de, sebastian_roseATgmx.de
> Http: www.emma-stil.de
next prev parent reply other threads:[~2008-10-23 14:56 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-10-23 12:04 How you can help Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
2008-10-23 15:04 ` Sebastian Rose
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
2008-10-23 17:02 ` Sebastian Rose
2008-10-24 12:13 ` Richard Riley
2008-10-24 15:39 ` Sebastian Rose
2008-10-24 16:27 ` Manish
2008-10-24 18:41 ` Avdi Grimm
2008-10-23 19:13 ` Avdi Grimm
2008-10-24 12:19 ` Richard Riley
2008-10-23 16:19 ` Bernt Hansen
2008-10-24 5:05 ` Carsten Dominik
2008-10-23 17:01 ` Jason F. McBrayer
2008-10-23 23:46 ` Eric Schulte
2008-10-23 14:20 ` Sebastian Rose
2008-10-23 14:50 ` Manish
2008-10-23 15:46 ` Eric Schulte
2008-10-23 16:18 ` Avdi Grimm
2008-10-23 14:55 ` Ben Alexander [this message]
2008-10-23 16:26 ` Sebastian Rose
2008-10-23 16:42 ` Avdi Grimm
2008-10-23 17:33 ` Sebastian Rose
2008-10-23 19:10 ` Avdi Grimm
2008-10-24 21:09 ` Tom Breton (Tehom)
2008-10-24 18:33 ` Ben Alexander
2008-10-24 18:44 ` Avdi Grimm
2008-10-24 19:02 ` Jeff Mickey
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
2008-10-26 21:31 ` Cameron Horsburgh
2008-10-27 8:47 ` Carsten Dominik
2008-10-27 8:47 ` Carsten Dominik
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
2008-10-23 17:12 ` How you can help Sebastian Rose
2008-10-23 15:08 ` Sebastian Rose
[not found] <E1Kt27M-0008Tx-0J@box188.bluehost.com>
2008-10-23 16:11 ` Robert Goldman
-- strict thread matches above, loose matches on Subject: below --
2008-10-23 7:35 Carsten Dominik
2008-10-23 8:12 ` Manish
2008-10-23 9:24 ` Sebastian Rose
2008-10-23 10:28 ` Sebastian Rose
2008-10-23 15:23 ` Russell Adams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=752DA813-1B66-4FD1-B28E-3C23176BA13D@alexanderonline.org \
--to=bva@alexanderonline.org \
--cc=emacs-orgmode@gnu.org \
--cc=sebastian_rose@gmx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).