* Re: How you can help
@ 2008-10-23 12:04 Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
` (2 more replies)
0 siblings, 3 replies; 35+ messages in thread
From: Ben Alexander @ 2008-10-23 12:04 UTC (permalink / raw)
To: sebastian_rose; +Cc: emacs-orgmode
Sebastian Rose wrote:
> 5. I also think of little packages for testing parts of org.
I'm curious if you or someone else has any ideas for writing automated
tests for org-mode. I haven't the foggiest idea how someone would
write a test for the parts of org that control what is displayed on
the screen. I mean, when the bug is 'it doesn't look right' how can
you tell?
Perhaps the git repository should have a small collection of small org-
mode files that reproduce certain bugs? If there were some examples
of how to create such a test, then perhaps bug reporters would find it
much easier to create them.
I do see some confusing issues due to different configuration files.
So creating a test file might involve making sure org-mode doesn't
read any configuration (how do you do that?) and possible asking org-
mode to extract all the configuration variables it has right now and
dump them into a test file (...and how do you do that?)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 12:04 How you can help Ben Alexander
@ 2008-10-23 13:43 ` Bernt Hansen
2008-10-23 15:04 ` Sebastian Rose
2008-10-23 17:01 ` Jason F. McBrayer
2008-10-23 14:20 ` Sebastian Rose
2008-10-23 15:08 ` Sebastian Rose
2 siblings, 2 replies; 35+ messages in thread
From: Bernt Hansen @ 2008-10-23 13:43 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode
Ben Alexander <bva@alexanderonline.org> writes:
> Sebastian Rose wrote:
>> 5. I also think of little packages for testing parts of org.
>
> I'm curious if you or someone else has any ideas for writing automated
> tests for org-mode. I haven't the foggiest idea how someone would
> write a test for the parts of org that control what is displayed on
> the screen. I mean, when the bug is 'it doesn't look right' how can
> you tell?
>
> Perhaps the git repository should have a small collection of small
> org-
> mode files that reproduce certain bugs? If there were some examples
> of how to create such a test, then perhaps bug reporters would find it
> much easier to create them.
>
> I do see some confusing issues due to different configuration files.
> So creating a test file might involve making sure org-mode doesn't
> read any configuration (how do you do that?) and possible asking org-
> mode to extract all the configuration variables it has right now and
> dump them into a test file (...and how do you do that?)
Running a minimal emacs should suppress custom config files:
emacs -q -l yourtest.el
Some kind of regression testing framework would be awesome. Org-mode is
large enough that this is almost a necessity to keep things stable and
bug-free.
Maybe something can be put together from the git testing framework and
use of emacs -batch to process test org files and verify the output is
as expected (with diff or some other tool).
-Bernt
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 12:04 How you can help Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
@ 2008-10-23 14:20 ` Sebastian Rose
2008-10-23 14:50 ` Manish
2008-10-23 14:55 ` Ben Alexander
2008-10-23 15:08 ` Sebastian Rose
2 siblings, 2 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 14:20 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
Hi Ben,
I cc'ed the list.
The tests I described in my email to the list are not automated. The
reason for that is my lack of (e)lisp knowledge.
BUT, they where easy to handle for non programmers. I think the little
test will make it to the worg site this week, when all private data is
removed. You could take a look at it by then. And: improve, improve,
improve... :-)
If you know of someone who knows how to do automated tests in elisp, or
some technique, package, whatever, please post it to the list, so we all
can take a look at it and comment->decide something. This is _highly_
_appreciated_.
Ben Alexander <bva@alexanderonline.org> writes:
> Sebastian Rose wrote:
>> 5. I also think of little packages for testing parts of org.
>
> I'm curious if you or someone else has any ideas for writing automated tests for
> org-mode. I haven't the foggiest idea how someone would write a test for the
> parts of org that control what is displayed on the screen. I mean, when the
> bug is 'it doesn't look right' how can you tell?
Believe me, my idea is the foggiest of all possible ideas ;-) (so where
my ideas of JavaScript some mounth ago), so I won't be of much help, I
fear.
> Perhaps the git repository should have a small collection of small org-
> mode files that reproduce certain bugs? If there were some examples of how to
> create such a test, then perhaps bug reporters would find it much easier to
> create them.
YES! Exactly. Every corner case an Org-file, every bug an Org-file.
_DATA_ for testing is something, everyone _can_ provide.
But git later, yes, maybe.
Since this would need Carsten to think and act on this, I feel Worg is a
nice place to start the first expieriments. We need Carstens power for
other things (when will Org-mode finaly wash my car? It's repeater-TODO,
but nothing happens!!!)
Basically, I'd try to keep the the testing as simple as possible, and
try to get elisp programmers to help with the code. We should try to
keep the hurdles for testers as humble (?) as possible, and put all
that's needed to be helpfull on one page in worg.
I recently discovered the very unautomated `(print object)' in the elisp
reference. Not everything can be done automated, maybe, but if I would
have known this stupid `print' function, I would know more about elisp
and some files in org already. And it would have been faster and easier
to create the two minor patches I wrote. This is, where the 'links to
elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy.
Willingness to help is no problem, as the Org-community reveals. As for
me, it's often a lack of time and/or knowledge, that prevents it.
And the aim of all this is to help helping, in means of good and
detailed bug reports in the first place.
> I do see some confusing issues due to different configuration files. So
> creating a test file might involve making sure org-mode doesn't read any
> configuration (how do you do that?) and possible asking org-
> mode to extract all the configuration variables it has right now and dump them
> into a test file (...and how do you do that?)
True.
Hm - to test without configuration, we already have `emacs -q'.
Dumping the variables is just a list of (print var) statements. A
(quite) complete list of variables could be extracted with grep?
e.g.:
grep -r defvar lisp/
Once we have such a list, we could set/reset some/all variables.
This will not be perfect, but could work reasonable well.
And yes, it would indeed be nice, to have elisp to reliably reset
emacs/org configuration, so one could do several different test in
sequence. Preferably they would _always_ run and dump all errors to some
file or buffer, even if one or more tests fail.
I think elsip provides funtionalities to handle those errors and skip to
the next test. This would be a huge step.
But still, we have to test with different _DATA_ too! The test I
described is just a quick hack I did to do the testing, while Carsten
was bug-hunting on the other end of my internet connection.
The XHTML-export test was much easier to do, then some other tests we
might need. In the end, it was a test, automated or not. And I had to do
it, because I'm one out of a few who use org to maintain a web-site
(locally in my case) and export an entire directory tree with lots of
_DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This
became obvious in this case. I was the one who even noticed the bug,
which means, no one else was using recursive XHTML export for a while,
or their setup didn't reveal that bug.
So clearly I was the one to provide some support for this very part of
the system. I can't rely on the assumption, that some maintainer has all
possible setups at hand all the time (maybe this was possible years
ago).
While the testing of the HTML-export was quite simple, it requires a lot
of _DATA_. Namly files and directories + different setups, per file
setups, #+SETUPFILE, images, with/without org-info.js, extra styles set
or not... to have a realistic testing environment (the test didn't have
all of these...).
A summary could be, that it's nice to provide different setups for
maintainers and testers.
- *Easy to install* (unpack and done),
- *easy to use* ('emacs -q' and click a few [[elisp:]] links to set
everything up)
- and *easy to download* once they are removed again (a central place on
Worg).
- *Corner cases*, like the 'empty line before first heading bug' in the
LaTeX exporter recently.
Regards,
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.roseATemma-stil.de, sebastian_roseATgmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 14:20 ` Sebastian Rose
@ 2008-10-23 14:50 ` Manish
2008-10-23 15:46 ` Eric Schulte
2008-10-23 14:55 ` Ben Alexander
1 sibling, 1 reply; 35+ messages in thread
From: Manish @ 2008-10-23 14:50 UTC (permalink / raw)
To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode Org-Mode
On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote:
[snip]
>
> If you know of someone who knows how to do automated tests in
> elisp, or some technique, package, whatever, please post it to the
> list, so we all can take a look at it and comment->decide
> something. This is _highly_ _appreciated_.
We could start here:
http://www.emacswiki.org/emacs-fr/UnitTesting#toc2
-- Manish
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 14:20 ` Sebastian Rose
2008-10-23 14:50 ` Manish
@ 2008-10-23 14:55 ` Ben Alexander
2008-10-23 16:26 ` Sebastian Rose
1 sibling, 1 reply; 35+ messages in thread
From: Ben Alexander @ 2008-10-23 14:55 UTC (permalink / raw)
To: Sebastian Rose; +Cc: emacs-orgmode Org-Mode
Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting
Unfortunately for me, I can't tell if Emacs comes with any builtin
framework already, so I downloaded one of the many options listed on
that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el
The personal issue I have is that I'm on a Mac, using Aquamacs, and
the command line version of emacs is a different binary, so there
might be trouble when a test passes at the command line, but fails
where it matters to me. I don't even make sure that org-mode is up to
date at the command line. I thought it wasn't, but I just checked and
now it is. Plus, I don't really understand internals of emacs (like
basic internals: I understand point and mark, buffer and file, but not
transient mark, indirect buffer, symbols vs strings, window vs tab vs
frame)
The tutorial I'd need to write a test is one which lays out code I
could copy and paste to do the following
* setup the test environment
- create a test directory
- create a sample test.org file
- put the cursor in a particular place
* run the command we need to test
- hit the 'TAB' key, or C-c C-c (some folks might need to be
reminded how to find out exactly what command is actually being run
when you hit a keystroke. And some of me might need to be told what
lisp-code to use when the keystroke runs different commands at
different places in a file)
- reformat a table
- clock in/out
- create the agenda
- export .html .ics .dvi file
* How do we specify the correct result???
- check that the headline folded properly. What's the lisp code
for getting the folded string as displayed?
- check that the cursor is where it should be? especially when the
cursor is near elipses...
- check that the agenda is built properly. What's the lisp code
for getting the agenda as a string?
- check that the exported files are correct. Maybe the right
suggestion is to run the export on two different files, so the test
can focus on the 'diff' between them. That way different people who
run the same test on different hosts can get the same result.
More experienced folks must suggest whether it's better to have the
expectation of a test be specified as a string in the test code or as
a separate file in the test directory. It might depend on context, so
feel free to lay out three or four cases.
If even one kind of test for each of these 'cases' exists, hopefully
we can encourage people to report bugs with the test. I mean, if
people are already going through the effort to create the small sample
org file to demonstrate their bug, and it's only a short (obvious)
step to translating their english description of what should happen
into equivalent lisp code.... that seems like a big temptation for
someone to do it.
Plus it might help them convince themselves that it really is a bug,
and not that they are going crazy.
I gotta move on for the day here, but I'll keep thinking a bit about
this.
On 2008-Oct-23, at 15:20, Sebastian Rose wrote:
> Hi Ben,
>
> I cc'ed the list.
>
>
> The tests I described in my email to the list are not automated. The
> reason for that is my lack of (e)lisp knowledge.
>
> BUT, they where easy to handle for non programmers. I think the little
> test will make it to the worg site this week, when all private data is
> removed. You could take a look at it by then. And: improve, improve,
> improve... :-)
>
> If you know of someone who knows how to do automated tests in elisp,
> or
> some technique, package, whatever, please post it to the list, so we
> all
> can take a look at it and comment->decide something. This is _highly_
> _appreciated_.
>
>
> Ben Alexander <bva@alexanderonline.org> writes:
>> Sebastian Rose wrote:
>>> 5. I also think of little packages for testing parts of org.
>>
>> I'm curious if you or someone else has any ideas for writing
>> automated tests for
>> org-mode. I haven't the foggiest idea how someone would write a
>> test for the
>> parts of org that control what is displayed on the screen. I
>> mean, when the
>> bug is 'it doesn't look right' how can you tell?
>
> Believe me, my idea is the foggiest of all possible ideas ;-) (so
> where
> my ideas of JavaScript some mounth ago), so I won't be of much help, I
> fear.
>
>
>> Perhaps the git repository should have a small collection of small
>> org-
>> mode files that reproduce certain bugs? If there were some
>> examples of how to
>> create such a test, then perhaps bug reporters would find it much
>> easier to
>> create them.
>
> YES! Exactly. Every corner case an Org-file, every bug an Org-file.
> _DATA_ for testing is something, everyone _can_ provide.
>
> But git later, yes, maybe.
>
> Since this would need Carsten to think and act on this, I feel Worg
> is a
> nice place to start the first expieriments. We need Carstens power for
> other things (when will Org-mode finaly wash my car? It's repeater-
> TODO,
> but nothing happens!!!)
>
> Basically, I'd try to keep the the testing as simple as possible, and
> try to get elisp programmers to help with the code. We should try to
> keep the hurdles for testers as humble (?) as possible, and put all
> that's needed to be helpfull on one page in worg.
>
> I recently discovered the very unautomated `(print object)' in the
> elisp
> reference. Not everything can be done automated, maybe, but if I would
> have known this stupid `print' function, I would know more about elisp
> and some files in org already. And it would have been faster and
> easier
> to create the two minor patches I wrote. This is, where the 'links to
> elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy.
>
> Willingness to help is no problem, as the Org-community reveals. As
> for
> me, it's often a lack of time and/or knowledge, that prevents it.
>
> And the aim of all this is to help helping, in means of good and
> detailed bug reports in the first place.
>
>
>
>
>> I do see some confusing issues due to different configuration
>> files. So
>> creating a test file might involve making sure org-mode doesn't
>> read any
>> configuration (how do you do that?) and possible asking org-
>> mode to extract all the configuration variables it has right now
>> and dump them
>> into a test file (...and how do you do that?)
>
> True.
>
> Hm - to test without configuration, we already have `emacs -q'.
> Dumping the variables is just a list of (print var) statements. A
> (quite) complete list of variables could be extracted with grep?
>
> e.g.:
>
> grep -r defvar lisp/
>
> Once we have such a list, we could set/reset some/all variables.
>
> This will not be perfect, but could work reasonable well.
>
> And yes, it would indeed be nice, to have elisp to reliably reset
> emacs/org configuration, so one could do several different test in
> sequence. Preferably they would _always_ run and dump all errors to
> some
> file or buffer, even if one or more tests fail.
>
> I think elsip provides funtionalities to handle those errors and
> skip to
> the next test. This would be a huge step.
>
>
>
>
>
>
> But still, we have to test with different _DATA_ too! The test I
> described is just a quick hack I did to do the testing, while Carsten
> was bug-hunting on the other end of my internet connection.
>
> The XHTML-export test was much easier to do, then some other tests we
> might need. In the end, it was a test, automated or not. And I had
> to do
> it, because I'm one out of a few who use org to maintain a web-site
> (locally in my case) and export an entire directory tree with lots of
> _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This
> became obvious in this case. I was the one who even noticed the bug,
> which means, no one else was using recursive XHTML export for a while,
> or their setup didn't reveal that bug.
>
> So clearly I was the one to provide some support for this very part of
> the system. I can't rely on the assumption, that some maintainer has
> all
> possible setups at hand all the time (maybe this was possible years
> ago).
>
> While the testing of the HTML-export was quite simple, it requires a
> lot
> of _DATA_. Namly files and directories + different setups, per file
> setups, #+SETUPFILE, images, with/without org-info.js, extra styles
> set
> or not... to have a realistic testing environment (the test didn't
> have
> all of these...).
>
>
>
>
> A summary could be, that it's nice to provide different setups for
> maintainers and testers.
>
> - *Easy to install* (unpack and done),
> - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set
> everything up)
> - and *easy to download* once they are removed again (a central
> place on
> Worg).
> - *Corner cases*, like the 'empty line before first heading bug' in
> the
> LaTeX exporter recently.
>
>
>
>
> Regards,
>
>
> --
> Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449
> Hannover
>
> Tel.: +49 (0)511 - 36 58 472
> Fax: +49 (0)1805 - 233633 - 11044
> mobil: +49 (0)173 - 83 93 417
> Email: s.roseATemma-stil.de, sebastian_roseATgmx.de
> Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 13:43 ` Bernt Hansen
@ 2008-10-23 15:04 ` Sebastian Rose
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:19 ` Bernt Hansen
2008-10-23 17:01 ` Jason F. McBrayer
1 sibling, 2 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 15:04 UTC (permalink / raw)
To: Bernt Hansen; +Cc: Ben Alexander, emacs-orgmode
Bernt Hansen <bernt@norang.ca> writes:
> Running a minimal emacs should suppress custom config files:
> emacs -q -l yourtest.el
Added this one to the Clippboard section on new org-tests/index.org in
Worg.git. (this section will be temporary...)
> Some kind of regression testing framework would be awesome. Org-mode is
> large enough that this is almost a necessity to keep things stable and
> bug-free.
It's big and feature-RICH.
> Maybe something can be put together from the git testing framework and
> use of emacs -batch to process test org files and verify the output is
> as expected (with diff or some other tool).
Hey, diff is a good idea!!
I didn't take the verification of the output into account yet :-)
I just pushed a change of Worgs start page, and added a directory
'org-tests'. I've placed an index.org there, which now is just a
collection of ideas (I'm on my day job, so I can't really work on it
now).
Don't know how often the git repo is published.
Bernt and Ben, are you 'worgers' allready?
Do you think it makes sense to add snippets and ideas to the new page in
Worg? I think while the list great to exchange ideas, it's good to have
a place, where all those ideas are destilled to one-liners.
Cheers,
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 12:04 How you can help Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
2008-10-23 14:20 ` Sebastian Rose
@ 2008-10-23 15:08 ` Sebastian Rose
2 siblings, 0 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 15:08 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
Ben Alexander <bva@alexanderonline.org> writes:
> Perhaps the git repository should have a small collection of small org-
> mode files that reproduce certain bugs?
Ah, stupid me, Worg IS a git repo :-D
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 14:50 ` Manish
@ 2008-10-23 15:46 ` Eric Schulte
2008-10-23 16:18 ` Avdi Grimm
0 siblings, 1 reply; 35+ messages in thread
From: Eric Schulte @ 2008-10-23 15:46 UTC (permalink / raw)
To: Manish; +Cc: Ben Alexander, emacs-orgmode Org-Mode
Hi,
I'd recommend ert.el, it is actively maintained, and I have it on good
authority that this the tool of choice for elisp unit testing.
http://github.com/ohler/ert/tree/master
,----[from ert.el]
| ;;; Commentary:
|
| ;; ERT is a tool for automated testing in Emacs Lisp. Its main
| ;; features are facilities for defining and running test cases and
| ;; reporting the results as well as for debugging test failures
| ;; interactively.
| ;;
| ;; The main entry points are `ert-deftest', which is similar to
| ;; `defun' but defines a test, and `ert-run-tests-interactively',
| ;; which runs tests and offers an interactive interface for inspecting
| ;; results and debugging. There is also `ert-run-tests-batch' for
| ;; non-interactive use.
| ;;
| ;; The body of `ert-deftest' forms resembles a function body, but the
| ;; additional operators `should', `should-not' and `should-error' are
| ;; available. `should' is similar to cl's `assert', but signals a
| ;; different error when its condition is violated that is caught and
| ;; processed by ERT. In addition, it analyzes its argument form and
| ;; records information that helps debugging (`assert' tries to do
| ;; something similar when its second argument SHOW-ARGS is true, but
| ;; `should' is more sophisticated). For information on `should-not'
| ;; and `should-error', see their docstrings.
`----
Manish <mailtomanish.sharma@gmail.com> writes:
> On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote:
> [snip]
> >
> > If you know of someone who knows how to do automated tests in
> > elisp, or some technique, package, whatever, please post it to the
> > list, so we all can take a look at it and comment->decide
> > something. This is _highly_ _appreciated_.
>
> We could start here:
> http://www.emacswiki.org/emacs-fr/UnitTesting#toc2
>
> -- Manish
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 15:04 ` Sebastian Rose
@ 2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
` (2 more replies)
2008-10-23 16:19 ` Bernt Hansen
1 sibling, 3 replies; 35+ messages in thread
From: Richard Riley @ 2008-10-23 15:49 UTC (permalink / raw)
To: Sebastian Rose; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander
Sebastian Rose <sebastian_rose@gmx.de> writes:
> Bernt Hansen <bernt@norang.ca> writes:
>> Running a minimal emacs should suppress custom config files:
>> emacs -q -l yourtest.el
>
> Added this one to the Clippboard section on new org-tests/index.org in
> Worg.git. (this section will be temporary...)
Something like the above should only be a link (at most) to the emacs
manual. Reproducing standard info is bad in the long run in case things
in the base product (emacs) change for example.
>
>> Some kind of regression testing framework would be awesome. Org-mode is
>> large enough that this is almost a necessity to keep things stable and
>> bug-free.
>
> It's big and feature-RICH.
The nature of OSS means that the community using the product keep it
stable and bug free. I dont think the efforts to produce meaningful
regression tests would be beneficial in an ever morphing product like
org-mode. Clearly my humble opinion on that one :-;
>
>> Maybe something can be put together from the git testing framework and
>> use of emacs -batch to process test org files and verify the output is
>> as expected (with diff or some other tool).
>
>
> Hey, diff is a good idea!!
>
> I didn't take the verification of the output into account yet :-)
>
> I just pushed a change of Worgs start page, and added a directory
> 'org-tests'. I've placed an index.org there, which now is just a
> collection of ideas (I'm on my day job, so I can't really work on it
> now).
>
> Don't know how often the git repo is published.
> Bernt and Ben, are you 'worgers' allready?
>
> Do you think it makes sense to add snippets and ideas to the new page in
> Worg? I think while the list great to exchange ideas, it's good to have
> a place, where all those ideas are destilled to one-liners.
I must say I am dubious about this. It means, for the tests to be
meaningful, that the output must be a fixed format in base org. I doubt
this will ever be the case. The presentation will fluctuate while the
core information (dates, schedules periods etc) will remain pretty much
constant.
The majority of bugs that I see are often down to people misusing or
using things in the base which are not fully explored. No amount of
regression testing can cover things like that unless the regression
tests include everyones customisations.
Do I think regression testing is important? Yes - in certain
environments. But every time Carsten, you, myself or anyone else fires
up org-mode we are already doing just that.
--
Inventor: A person who makes an ingenious arrangement of wheels, levers and springs, and believes it civilization. ~Ambrose Bierce, The Devil's Dictionary
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 15:46 ` Eric Schulte
@ 2008-10-23 16:18 ` Avdi Grimm
0 siblings, 0 replies; 35+ messages in thread
From: Avdi Grimm @ 2008-10-23 16:18 UTC (permalink / raw)
To: emacs-orgmode Org-Mode
I've been considering buckling down and doing some more ELisp coding,
which would mean org-mode coding because I've become ridiculously
dependent on org-mode. If I do, I would only do it in a test-first
way, because that's the only way I'll write code anymore; so it would
definitely be good to know that unit tests will be be accepted into
the codebase. Personally, if I did this I think I would like to use
Emacs Lisp Expectations:
http://www.emacswiki.org/cgi-bin/emacs/EmacsLispExpectations
I'm a big fan of Jay Field's work, I like the way the original
Expectations for Ruby is structured, and I like the looks of the EL
Expectations API. I'm open to other libraries though - so long as
SOME unit-testing standard it accepted.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 15:04 ` Sebastian Rose
2008-10-23 15:49 ` Richard Riley
@ 2008-10-23 16:19 ` Bernt Hansen
2008-10-24 5:05 ` Carsten Dominik
1 sibling, 1 reply; 35+ messages in thread
From: Bernt Hansen @ 2008-10-23 16:19 UTC (permalink / raw)
To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode
Sebastian Rose <sebastian_rose@gmx.de> writes:
> Don't know how often the git repo is published.
> Bernt and Ben, are you 'worgers' allready?
I think it's published hourly (maybe) and yes I have access to Worg.
-Bernt
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 15:49 ` Richard Riley
@ 2008-10-23 16:22 ` Ben Alexander
2008-10-23 17:02 ` Sebastian Rose
2008-10-23 19:13 ` Avdi Grimm
2 siblings, 0 replies; 35+ messages in thread
From: Ben Alexander @ 2008-10-23 16:22 UTC (permalink / raw)
To: Richard Riley; +Cc: emacs-orgmode Org-Mode
On 2008-Oct-23, at 16:49, Richard Riley wrote:
> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> Bernt Hansen <bernt@norang.ca> writes:
>>> Running a minimal emacs should suppress custom config files:
>>> emacs -q -l yourtest.el
>>
>> Added this one to the Clippboard section on new org-tests/index.org
>> in
>> Worg.git. (this section will be temporary...)
>
> Something like the above should only be a link (at most) to the emacs
> manual. Reproducing standard info is bad in the long run in case
> things
> in the base product (emacs) change for example.
>
>>
>>> Some kind of regression testing framework would be awesome. Org-
>>> mode is
>>> large enough that this is almost a necessity to keep things stable
>>> and
>>> bug-free.
>>
>> It's big and feature-RICH.
>
> The nature of OSS means that the community using the product keep it
> stable and bug free. I dont think the efforts to produce meaningful
> regression tests would be beneficial in an ever morphing product like
> org-mode. Clearly my humble opinion on that one :-;
>
I'm a bit clue-free on what the words mean. I think of regression
tests as ways of defining the way "Things Should Work(TM)" so if
someone commits as fix to org-mode that works fine on Debian, but
breaks on Windows, there's a way for someone on Windows to easily
(brainlessly) run the test and report back to the developer.
I don't think of org-mode as 'ever morphing'. There are some features
that are stable, and should remain so. A regression test would be
valuable, presumably, for developers working on a new feature (don't
screw up important existing functionality!)
>
> I must say I am dubious about this. It means, for the tests to be
> meaningful, that the output must be a fixed format in base org. I
> doubt
> this will ever be the case. The presentation will fluctuate while the
> core information (dates, schedules periods etc) will remain pretty
> much
> constant.
>
I agree that the presentation is fluid and it was hard for me to
imagine how to test it in an automated way. But I disagree (I think)
in that having a test on the presentation is extremely powerful.
I'm thinking that one example of how to write such a test could be
copied when someone is trying to explain what they want to happen, and
how it is not happening right now.
Perhaps such a test would have a very limited useful life. I'd like
to think that it could live and be useful for a very long time (that's
why thinking about configuration/user customization issues in a test
is important). But even if the test is only useful until the bug is
fixed (or the feature implemented) , so be it.
> The majority of bugs that I see are often down to people misusing or
> using things in the base which are not fully explored. No amount of
> regression testing can cover things like that unless the regression
> tests include everyones customisations.
>
Not everyone's customizations, just every customization org-mode has.
I mean, if the feature is there, it should work. If the feature is
important, it should work in the same way on every platform.
The trick -- and I hope it's either solved or easy -- is exactly how
to show that "the feature works the same way as it should, right now,
on this platform"
If I understand you properly, you're just saying that the user
customization creates so much variability in the output, there isn't
anyway you could capture all the possible presentations in a single
test. Well, yeah, so user customizations need to be left out of the
test case.
Except that in the scenario I'm thinking of where someone (a mere
mortal using org-mode, with limited lisp knowledge) is trying to
report a bug, user customization will be critical. My first thought
was asking org-mode to dump all the customization variables it has
into a buffer (as buffer-local variables maybe?) would help someone
else recreate the bug. (By starting org-mode with NO customizations,
except what's found in the test case)
Or are you suggesting that the source of presentation variation is due
mostly to OS, Emacs version, or something else beyond the scope of our
tests, so they couldn't be replicated on another machine?
> Do I think regression testing is important? Yes - in certain
> environments. But every time Carsten, you, myself or anyone else fires
> up org-mode we are already doing just that.
>
>
> --
> Inventor: A person who makes an ingenious arrangement of wheels,
> levers and springs, and believes it civilization. ~Ambrose Bierce,
> The Devil's Dictionary
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 14:55 ` Ben Alexander
@ 2008-10-23 16:26 ` Sebastian Rose
2008-10-23 16:42 ` Avdi Grimm
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
0 siblings, 2 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 16:26 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
Ben,
I added this to the new page on worg. Please tell me: is it OK for you,
if I simply add your ideas there??
What do the others think?
Ben Alexander <bva@alexanderonline.org> writes:
> The tutorial I'd need to write a test is one which lays out code I could copy
> and paste to do the following
>
> * setup the test environment
> - create a test directory
> - create a sample test.org file
> - put the cursor in a particular place
>
> * run the command we need to test
> - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to
> find out exactly what command is actually being run when you hit a keystroke.
> And some of me might need to be told what lisp-code to use when the keystroke
> runs different commands at different places in a file)
> - reformat a table
> - clock in/out
> - create the agenda
> - export .html .ics .dvi file
> * How do we specify the correct result???
> - check that the headline folded properly. What's the lisp code for getting
> the folded string as displayed?
> - check that the cursor is where it should be? especially when the cursor is
> near elipses...
> - check that the agenda is built properly. What's the lisp code for getting
> the agenda as a string?
> - check that the exported files are correct. Maybe the right suggestion is to
> run the export on two different files, so the test can focus on the 'diff'
> between them. That way different people who run the same test on different
> hosts can get the same result.
Please, if you find time, add the rest somewhere there?
If not, I'll take a closer look tonight and check in what's interesting
(which seems to be all of it, actually).
http://www.emacswiki.org/emacs-fr/UnitTesting:
Does this mean to add code to Org's *.el files? This would mean a lot of
work for Carsten :-( OK, once done...
But:
sh> grep -r defun lisp/ | wc -l
1290
Checking in/out of every function makes 2580 places to check. I can't
estimate the effort, actually, because I never did Unit testing. I guess
this number much higher than needed?
Could anyone with testing expierience estimate?
While I think automated tests are a great thing to do and I'm happy to
see this reaction on the list :-), I still think we should also provide
the 'simple' tests without additional dependencies (when will package
management make it into emacs...). And I'd like to keep the effort and
impact for Carsten as small as possible. He will have to check each and
every patch because of feeling, and actually being, responsible for the
code.
Another question: will the code compile in emacs without the testing
framework installed? Probably yes.
Should we try to think of those 'simple test packages' as the more
_DATA_ part of other tests? This would mean one could code an automated
test without the need to create data too, since we have the data
seperated. Again, I'd like to keep the hurdles low for everyone
involved. When automated testing turns out to be a foolish easy and rock
solid, this could change.
* Three basic approaches:
1 Simple data and setup packages.
2 Code, that executes org commands and test for correct output,
no changes in Org's code needed.
3 Test framework with changes to Org's code.
I'd promote a combination from 1 and 2, and see, how far it goes.
DATA
+--- d1
+--- d2
+--- d3
TESTS
+---t-a
+---t-b
+---t-c
Now one would run test t-a over d1, d2, d3.
Still a programmer could use d1 to fix a bug in d1 and finally push the
fix for automated testing against d2 and d3.
bug X
tester verifies (is ist bug? how to reproduce...)
tester adds X to dX (message -> list or programmer)
programmer installs dX
programmer fixes bugX
programmer push (message -> list or tester)
tester tests against d1, d2, d3
where tester = [human|machine] preferable human (simplicity)
Should we track the dependencies between code and tests?
Suppose one has worked on a.el.
Shure more than one test has to run.
Example: org-publish.el contains code common to LaTeX-export,
HTML-export...
Regards,
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 16:26 ` Sebastian Rose
@ 2008-10-23 16:42 ` Avdi Grimm
2008-10-23 17:33 ` Sebastian Rose
2008-10-24 18:33 ` Ben Alexander
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
1 sibling, 2 replies; 35+ messages in thread
From: Avdi Grimm @ 2008-10-23 16:42 UTC (permalink / raw)
To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode Org-Mode
A few points, from someone with a decent amount of testing background:
* As someone who has contributed to OS projects, the lack of a
pre-existing set of regression tests in org-mode is actually *the*
most significant blocker to my getting involved in org-mode
development. I do all of my coding - both professional and personal -
in the context of tests, and not having the existing framework means
that before I can start working on new features I first need to spend
time yak-shaving on testing infrastructure.
* I think there's a lot of over-thinking going on here. Here's the
test-first coding discipline in a nutshell:
1. Identify a problem/missing feature.
2. Write a test (possibly using a unit-testing framework to help)
which will pass when the bug has been fixed or feature added. This can
be as simple as calling a function and validating its return value.
3. Run the test. Verify it FAILS.
4. Write code to make the test PASS.
5. Refactor, if you introduced any code duplication in step 4.
6. Run all the tests, to make sure you didn't break anything else.
7. Commit.
If someone would be so kind as to identify a small bug or feature, I
would be happy to demonstrate this workflow in the form of code, time
permitting.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 13:43 ` Bernt Hansen
2008-10-23 15:04 ` Sebastian Rose
@ 2008-10-23 17:01 ` Jason F. McBrayer
2008-10-23 23:46 ` Eric Schulte
1 sibling, 1 reply; 35+ messages in thread
From: Jason F. McBrayer @ 2008-10-23 17:01 UTC (permalink / raw)
To: emacs-orgmode
Bernt Hansen <bernt@norang.ca> writes:
> Some kind of regression testing framework would be awesome. Org-mode is
> large enough that this is almost a necessity to keep things stable and
> bug-free.
>
Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ?
--
+-----------------------------------------------------------+
| Jason F. McBrayer jmcbray@carcosa.net |
| If someone conquers a thousand times a thousand others in |
| battle, and someone else conquers himself, the latter one |
| is the greatest of all conquerors. --- The Dhammapada |
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
@ 2008-10-23 17:02 ` Sebastian Rose
2008-10-24 12:13 ` Richard Riley
2008-10-23 19:13 ` Avdi Grimm
2 siblings, 1 reply; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 17:02 UTC (permalink / raw)
To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander
Hi Richard,
Richard Riley <rileyrgdev@googlemail.com> writes:
>> Added this one to the Clippboard section on new org-tests/index.org in
>> Worg.git. (this section will be temporary...)
>
> Something like the above should only be a link (at most) to the emacs
> manual. Reproducing standard info is bad in the long run in case things
> in the base product (emacs) change for example.
On the long run, yes. I just added this section, to start the page. And
since it's just STARTing... this was in the first or second reaction
that came in. Feel free to edit ('go wild' ;-))!
I wouldn't bother to have several pages like this one will be (it's
supposed to be an index on the long run), each covering another way of
testing.
I just meant to take _ACT_. Scepticism is a good thing, as long as it
doesn't stop you from doing.
>>
>>> Some kind of regression testing framework would be awesome. Org-mode is
>>> large enough that this is almost a necessity to keep things stable and
>>> bug-free.
>>
>> It's big and feature-RICH.
>
> The nature of OSS means that the community using the product keep it
> stable and bug free. I dont think the efforts to produce meaningful
> regression tests would be beneficial in an ever morphing product like
> org-mode. Clearly my humble opinion on that one :-;
We do test our software by using it. But the bug in the HTML-exporter
that Carsten has fixed two days ago, was introduced in early September
and would be in 6.10, which is supposed to be in the emacs 23 release.
A very simple test plan would have revealed it.
>>> Maybe something can be put together from the git testing framework and
>>> use of emacs -batch to process test org files and verify the output is
>>> as expected (with diff or some other tool).
>>
>>
>> Hey, diff is a good idea!!
>>
>> I didn't take the verification of the output into account yet :-)
>>
>> I just pushed a change of Worgs start page, and added a directory
>> 'org-tests'. I've placed an index.org there, which now is just a
>> collection of ideas (I'm on my day job, so I can't really work on it
>> now).
>>
>> Don't know how often the git repo is published.
>> Bernt and Ben, are you 'worgers' allready?
>>
>> Do you think it makes sense to add snippets and ideas to the new page in
>> Worg? I think while the list great to exchange ideas, it's good to have
>> a place, where all those ideas are destilled to one-liners.
>
> I must say I am dubious about this. It means, for the tests to be
> meaningful, that the output must be a fixed format in base org.
Why? If the test bails out 'ERROR', I will have to look for the
reason. If the format changed in a legal way => adjust the test.
> I doubt
> this will ever be the case. The presentation will fluctuate while the
> core information (dates, schedules periods etc) will remain pretty much
> constant.
>
> The majority of bugs that I see are often down to people misusing or
> using things in the base which are not fully explored. No amount of
> regression testing can cover things like that unless the regression
> tests include everyones customisations.
Yes, because Carsten add features en masse :-)
I see the testing differently. In the first place we need THINK of
testing.
New Org-revision out?
Ahh, OK, I have to the HTML-exports, I want to be working.
To do this, I need several different setups, several different data
directories (Org-files) and an easy way to test, that doesn't eat my
time and gives a result. The quality may vary, but ERRORs will be
detected for shure.
Not one or mounths later - immediately.
If someone installs emacs 23, tries to export to HTML and gets an
error....
> Do I think regression testing is important? Yes - in certain
> environments. But every time Carsten, you, myself or anyone else fires
> up org-mode we are already doing just that.
Yes, but we can do better, easier and more complete.
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
@ 2008-10-23 17:12 ` Sebastian Rose
0 siblings, 0 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 17:12 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
(cc'ing the list again)
Ben Alexander <bva@alexanderonline.org> writes:
> On 2008-Oct-23, at 17:26, Sebastian Rose wrote:
>
>> Ben,
>>
>> I added this to the new page on worg. Please tell me: is it OK for you,
>> if I simply add your ideas there??
>>
> Sure put what you want there.
Great.
> FWIW, I'm brand new (today!) to git and I've cloned Worg today too. But I not a
> worger' -- I'm not sure I've read much of it online.
So I'll look what else should be added? Hm - this thread is so busy,
that I'd say I take a few hours on weekend to sum it up.
>> What do the others think?
>>
>>
>> Ben Alexander <bva@alexanderonline.org> writes:
>>> way too much
>>>
>>
>>
>> Please, if you find time, add the rest somewhere there?
>> If not, I'll take a closer look tonight and check in what's interesting
>> (which seems to be all of it, actually).
>>
>>
>> http://www.emacswiki.org/emacs-fr/UnitTesting:
>>
>> Does this mean to add code to Org's *.el files? This would mean a lot of
>> work for Carsten :-( OK, once done...
>>
>
> Uh, I hope that's not necessary. I'd hope that the tests could be written
> completely independently of any org-code. If the basic idea is to launch emacs
> in batch mode, then the test could redefine any function locally, in the test
> code, instead of screwing around with what Carsten has.
>
> If every test is written this way, it'll be very slow to run all the tests.
> Maybe that doesn't matter for bug-report tests? But here's where I should shut
> up and listen instead of talk.
Great! I hope so too!!
If slow, it's still no problem to run all those tests before a release
or once a day, whatever.
Glad to here your interested in keeping impact on Carsten low!
Cheers,
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 16:42 ` Avdi Grimm
@ 2008-10-23 17:33 ` Sebastian Rose
2008-10-23 19:10 ` Avdi Grimm
2008-10-24 21:09 ` Tom Breton (Tehom)
2008-10-24 18:33 ` Ben Alexander
1 sibling, 2 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-23 17:33 UTC (permalink / raw)
To: emacs-orgmode Org-Mode
"Avdi Grimm" <avdi@avdi.org> writes:
> A few points, from someone with a decent amount of testing background:
Jippie! Please stay with us for a few days :-D
> * As someone who has contributed to OS projects, the lack of a
> pre-existing set of regression tests in org-mode is actually *the*
> most significant blocker to my getting involved in org-mode
> development. I do all of my coding - both professional and personal -
> in the context of tests, and not having the existing framework means
> that before I can start working on new features I first need to spend
> time yak-shaving on testing infrastructure.
>
> * I think there's a lot of over-thinking going on here. Here's the
> test-first coding discipline in a nutshell:
> 1. Identify a problem/missing feature.
> 2. Write a test (possibly using a unit-testing framework to help)
> which will pass when the bug has been fixed or feature added. This can
> be as simple as calling a function and validating its return value.
> 3. Run the test. Verify it FAILS.
> 4. Write code to make the test PASS.
> 5. Refactor, if you introduced any code duplication in step 4.
> 6. Run all the tests, to make sure you didn't break anything else.
> 7. Commit.
>
> If someone would be so kind as to identify a small bug or feature, I
> would be happy to demonstrate this workflow in the form of code, time
> permitting.
Sounds great! Not shure if we can provide a bug though :-D
Bug anyone?
Hm - how about:
1. A not yet existent elisp file test-worg.el, that defines a function
hello-worg, and a variable lang, and simply puts "Hello Worg"into the
minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang
"de")) ?
Did you work with unit-testing frameworks for elisp already?
Which one?
Recommendations?
Could say something about the effort to get started with such a
framework?
Can we add it without changing Org's code? If understand 2. correctly - yes?
If we can, would we loose quality/speed of tests?
How much?
Tell us all :-)
Oha - I have an appointment (concert) - back tonight!
All the best,
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http: www.emma-stil.de
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 17:33 ` Sebastian Rose
@ 2008-10-23 19:10 ` Avdi Grimm
2008-10-24 21:09 ` Tom Breton (Tehom)
1 sibling, 0 replies; 35+ messages in thread
From: Avdi Grimm @ 2008-10-23 19:10 UTC (permalink / raw)
To: Sebastian Rose; +Cc: emacs-orgmode Org-Mode
On Thu, Oct 23, 2008 at 1:33 PM, Sebastian Rose <sebastian_rose@gmx.de> wrote:
> Hm - how about:
>
> 1. A not yet existent elisp file test-worg.el, that defines a function
> hello-worg, and a variable lang, and simply puts "Hello Worg"into the
> minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang
> "de")) ?
I'd prefer something that would be of concrete value to someone in
using org-mode - made-up scenarios tend to lead to unrealistic tests.
> Did you work with unit-testing frameworks for elisp already?
> Which one?
I have not. I've done test-driven development in Java, C#, Ruby, and
C++(!) though, and if I can do it in C++ I think I can get up to speed
in any language pretty quickly ;-)
> Recommendations?
As I said in another comment, I'm liking the look of Emacs Lisp
Expectations a *lot*. Partly because I think Jay Fields is bang on
the money for how maintain good habits in your tests, and partly
because it has a mocking framework to go with it. Testing without a
mocking framework is less fun than testing with. However, I haven't
actually used it yet, so I can't responsibly recommend it. Let's just
say that the discovery of it recently was the thing that got me
excited about doing some serious ELisp development.
> Could say something about the effort to get started with such a
> framework?
I think if you have some *running* examples to crib from it shouldn't
take someone who is already ELisp-savvy more than a few hours to get
the hang of it.
>
> Can we add it without changing Org's code? If understand 2. correctly - yes?
In a dynamic language like ELisp, any testing framework that requires
you to change the code just to get it under test isn't worth using.
No, you should not have to change org-mode's code to get started with
tests.
That said, developing in concert with a test suite will definitely
tend to influence how you write the code. Generally speaking,
developing with tests encourages smaller functions which do one thing
and one thing alone and don't have a lot of non-obvious inputs and
outputs. Since this is usually the direction you *want* your code to
go in anyway, it's a virtuous cycle.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
2008-10-23 17:02 ` Sebastian Rose
@ 2008-10-23 19:13 ` Avdi Grimm
2008-10-24 12:19 ` Richard Riley
2 siblings, 1 reply; 35+ messages in thread
From: Avdi Grimm @ 2008-10-23 19:13 UTC (permalink / raw)
To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander
On Thu, Oct 23, 2008 at 11:49 AM, Richard Riley
<rileyrgdev@googlemail.com> wrote:
> The nature of OSS means that the community using the product keep it
> stable and bug free. I dont think the efforts to produce meaningful
> regression tests would be beneficial in an ever morphing product like
> org-mode. Clearly my humble opinion on that one :-;
I don't know of a single open-source project with more than one or two
developers which doesn't have a suite of regression tests. In fact, I
know of a number of OSS projects which *will not* accept a patch
unless tests accompany it. Once a codebase grows beyond a one-person
pet project, tests become increasingly necessary for maintainability.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 17:01 ` Jason F. McBrayer
@ 2008-10-23 23:46 ` Eric Schulte
0 siblings, 0 replies; 35+ messages in thread
From: Eric Schulte @ 2008-10-23 23:46 UTC (permalink / raw)
To: Jason F. McBrayer; +Cc: emacs-orgmode
jmcbray@carcosa.net (Jason F. McBrayer) writes:
> Bernt Hansen <bernt@norang.ca> writes:
>
>> Some kind of regression testing framework would be awesome. Org-mode is
>> large enough that this is almost a necessity to keep things stable and
>> bug-free.
>>
>
> Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ?
The author of elunit (Phil) has abandoned it in favor of ert.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 16:19 ` Bernt Hansen
@ 2008-10-24 5:05 ` Carsten Dominik
0 siblings, 0 replies; 35+ messages in thread
From: Carsten Dominik @ 2008-10-24 5:05 UTC (permalink / raw)
To: Bernt Hansen; +Cc: Ben Alexander, emacs-orgmode
On Oct 23, 2008, at 6:19 PM, Bernt Hansen wrote:
> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> Don't know how often the git repo is published.
>> Bernt and Ben, are you 'worgers' allready?
>
> I think it's published hourly (maybe) and yes I have access to Worg.
The webserver pulls changes from the Worg git repo on the full hour
and publishes these changes on the half hour. So changes will appear
on the web after 30-90 minutes.
If necessary, this procedure can be upgraded.
- Carsten
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 17:02 ` Sebastian Rose
@ 2008-10-24 12:13 ` Richard Riley
2008-10-24 15:39 ` Sebastian Rose
2008-10-24 16:27 ` Manish
0 siblings, 2 replies; 35+ messages in thread
From: Richard Riley @ 2008-10-24 12:13 UTC (permalink / raw)
To: Sebastian Rose; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander
Sebastian Rose <sebastian_rose@gmx.de> writes:
> Hi Richard,
>
>
> Richard Riley <rileyrgdev@googlemail.com> writes:
>>> Added this one to the Clippboard section on new org-tests/index.org in
>>> Worg.git. (this section will be temporary...)
>>
>> Something like the above should only be a link (at most) to the emacs
>> manual. Reproducing standard info is bad in the long run in case things
>> in the base product (emacs) change for example.
>
>
> On the long run, yes. I just added this section, to start the page. And
> since it's just STARTing... this was in the first or second reaction
> that came in. Feel free to edit ('go wild' ;-))!
>
> I wouldn't bother to have several pages like this one will be (it's
> supposed to be an index on the long run), each covering another way of
> testing.
>
> I just meant to take _ACT_. Scepticism is a good thing, as long as it
> doesn't stop you from doing.
Of course not.
>
>
>>>
>>>> Some kind of regression testing framework would be awesome. Org-mode is
>>>> large enough that this is almost a necessity to keep things stable and
>>>> bug-free.
>>>
>>> It's big and feature-RICH.
>>
>> The nature of OSS means that the community using the product keep it
>> stable and bug free. I dont think the efforts to produce meaningful
>> regression tests would be beneficial in an ever morphing product like
>> org-mode. Clearly my humble opinion on that one :-;
>
>
> We do test our software by using it. But the bug in the HTML-exporter
> that Carsten has fixed two days ago, was introduced in early September
> and would be in 6.10, which is supposed to be in the emacs 23 release.
>
> A very simple test plan would have revealed it.
The question is : define simple. I am not against test plans, but the
nature of such means a process in place where such test plans are
carried out. Carsten knows his SW inside out and would find it hard to
apply test cases every time a bug is fixed for time alone. Bugs happen
in SW and the nature (excuse the repetition) of OSS is that we, the
users, are kind of the testers and users.
>>>> Maybe something can be put together from the git testing framework and
>>>> use of emacs -batch to process test org files and verify the output is
>>>> as expected (with diff or some other tool).
>>>
>>>
>>> Hey, diff is a good idea!!
>>>
>>> I didn't take the verification of the output into account yet :-)
>>>
>>> I just pushed a change of Worgs start page, and added a directory
>>> 'org-tests'. I've placed an index.org there, which now is just a
>>> collection of ideas (I'm on my day job, so I can't really work on it
>>> now).
>>>
>>> Don't know how often the git repo is published.
>>> Bernt and Ben, are you 'worgers' allready?
>>>
>>> Do you think it makes sense to add snippets and ideas to the new page in
>>> Worg? I think while the list great to exchange ideas, it's good to have
>>> a place, where all those ideas are destilled to one-liners.
>>
>> I must say I am dubious about this. It means, for the tests to be
>> meaningful, that the output must be a fixed format in base org.
>
>
> Why? If the test bails out 'ERROR', I will have to look for the
> reason. If the format changed in a legal way => adjust the test.
What is an error? And for whom? That the categories are one more tab to
the right? I know I am probably sounding a bit of a killjoy here but
having been involved in closed source SW development with dedicated QA
and testing teams it was hard enough then. Maybe I am just being a bit
of an old curmudgeon here :-(
>
>
>> I doubt
>> this will ever be the case. The presentation will fluctuate while the
>> core information (dates, schedules periods etc) will remain pretty much
>> constant.
>>
>> The majority of bugs that I see are often down to people misusing or
>> using things in the base which are not fully explored. No amount of
>> regression testing can cover things like that unless the regression
>> tests include everyones customisations.
>
> Yes, because Carsten add features en masse :-)
> I see the testing differently. In the first place we need THINK of
> testing.
I think we all do. And the main contributors do too. But it is
unrealistic to expect a full regression test for each and every release
unless there is someone willing to step up to the mark and take the time
to perform them. I personally feel that enough take the "cutting edge"
and report back before the "official" cuts. We are all beat testers. Its
the nature of being an emacs and OSS user in the first place :-;
>
> New Org-revision out?
> Ahh, OK, I have to the HTML-exports, I want to be working.
>
> To do this, I need several different setups, several different data
> directories (Org-files) and an easy way to test, that doesn't eat my
> time and gives a result. The quality may vary, but ERRORs will be
> detected for shure.
>
> Not one or mounths later - immediately.
If you don't find it immediately then how often do you use the feature?
Now I know I probably sound negative. I assure I am not. Maybe the
earlier word "sceptic" is more applicable ..
>
> If someone installs emacs 23, tries to export to HTML and gets an
> error....
I have been there too. In fact you found a bug that I left hanging for
ages because I found a work around ( dont publish from anywhere other
than the root of the project) - so thanks for your perseverance on that.
I think the best thing people can do is keep a stable version and also
test the CVS (oops) GIT version too very now and again.
>
>
>> Do I think regression testing is important? Yes - in certain
>> environments. But every time Carsten, you, myself or anyone else fires
>> up org-mode we are already doing just that.
>
> Yes, but we can do better, easier and more complete.
Possibly. I look forward to seeing proposed solutions and would gladly
lend some time to applying them.
--
God never made his work for man to mend.
~John Dryden
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-23 19:13 ` Avdi Grimm
@ 2008-10-24 12:19 ` Richard Riley
0 siblings, 0 replies; 35+ messages in thread
From: Richard Riley @ 2008-10-24 12:19 UTC (permalink / raw)
To: Avdi Grimm; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander
"Avdi Grimm" <avdi@avdi.org> writes:
> On Thu, Oct 23, 2008 at 11:49 AM, Richard Riley
> <rileyrgdev@googlemail.com> wrote:
>> The nature of OSS means that the community using the product keep it
>> stable and bug free. I dont think the efforts to produce meaningful
>> regression tests would be beneficial in an ever morphing product like
>> org-mode. Clearly my humble opinion on that one :-;
>
>
> I don't know of a single open-source project with more than one or two
> developers which doesn't have a suite of regression tests. In fact, I
> know of a number of OSS projects which *will not* accept a patch
> unless tests accompany it. Once a codebase grows beyond a one-person
> pet project, tests become increasingly necessary for maintainability.
Then I have to disagree. There are many OSS projects with no regression
tests. Is this a good thing? Not necessarily. But the nature of emacs,
elisp and the customization features make it, for me, a slightly
different kettle of fish. In fact the nature of this thread and the
nature of most emacs customizations suggests to me that the regression
tests are merely using the SW and seeing if it works as expected - not
an automated process. This is the important piece : automated regression
tests. Regression testing as in "run it and see if it works" is fine -
automated ones are, well, another issue. Lets see how this pans out. Be
sure I am not against testing, regression testing etc, I am merely
suggesting that automated tests are possibly less effective than a
proper beta release to users which is currently provided.
--
Do you realize if it weren't for Edison we'd be watching TV by candlelight? ~Al Boliska
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-24 12:13 ` Richard Riley
@ 2008-10-24 15:39 ` Sebastian Rose
2008-10-24 16:27 ` Manish
1 sibling, 0 replies; 35+ messages in thread
From: Sebastian Rose @ 2008-10-24 15:39 UTC (permalink / raw)
To: emacs-orgmode
Hi Richard,
Richard Riley wrote:
> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> Hi Richard,
>>
>>
>> Richard Riley <rileyrgdev@googlemail.com> writes:
>>>> Added this one to the Clippboard section on new org-tests/index.org in
>>>> Worg.git. (this section will be temporary...)
>>> Something like the above should only be a link (at most) to the emacs
>>> manual. Reproducing standard info is bad in the long run in case things
>>> in the base product (emacs) change for example.
>>
>> On the long run, yes. I just added this section, to start the page. And
>> since it's just STARTing... this was in the first or second reaction
>> that came in. Feel free to edit ('go wild' ;-))!
>>
>> I wouldn't bother to have several pages like this one will be (it's
>> supposed to be an index on the long run), each covering another way of
>> testing.
>>
>> I just meant to take _ACT_. Scepticism is a good thing, as long as it
>> doesn't stop you from doing.
>
> Of course not.
Glad you replied. While I speek a little English, I'm not alwas shure
how it _feels_ for native speakers. For example this one about
scepticism.
>>
>>>>> Some kind of regression testing framework would be awesome. Org-mode is
>>>>> large enough that this is almost a necessity to keep things stable and
>>>>> bug-free.
>>>> It's big and feature-RICH.
>>> The nature of OSS means that the community using the product keep it
>>> stable and bug free. I dont think the efforts to produce meaningful
>>> regression tests would be beneficial in an ever morphing product like
>>> org-mode. Clearly my humble opinion on that one :-;
>>
>> We do test our software by using it. But the bug in the HTML-exporter
>> that Carsten has fixed two days ago, was introduced in early September
>> and would be in 6.10, which is supposed to be in the emacs 23 release.
>>
>> A very simple test plan would have revealed it.
>
> The question is : define simple. I am not against test plans, but the
> nature of such means a process in place where such test plans are
> carried out. Carsten knows his SW inside out and would find it hard to
> apply test cases every time a bug is fixed for time alone. Bugs happen
> in SW and the nature (excuse the repetition) of OSS is that we, the
> users, are kind of the testers and users.
OK, simplicity.
That's an important point I'd like to make clear. I _want_ simplicity
in two ways:
a) This here is _not_ meant to make Carsten work _more_, but to _help_
Carsten and all contributors. We must keep that in mind, while
searching for the right way to test.
Not Carsten, but _we_ collect the bugs, _we_ write the
tests (whithout interfering with Carstens code), _we_ setup the
testenvironment, and _we_ finally _do_ the testing.
b) I _want_ tests, that everyone can do. Nobody knows, who of us is
still here in a year or two. I like the idea of automated testing,
but I'd also like to get every org-user to test. Imagine to
have org-test.el in contrib (split into modules maybe).
(setq org/test-html-export-base-dir "~/org-test/data/")
(setq org/test-html-export-publising-dir "~/public_html/org-test/")
M-x org/test-test-html-export
The whole thing automatically executed once a week/day/mounth,
after every git-pull, <SCHEDULED +1d>, sending a backtrace on
failure.
Once tests exist, we will find out how the interaction with Carsten
works best / where neccessary. I know it will be hard to adjust all
those tests to 'movin target' called Org.
>>>>> Maybe something can be put together from the git testing framework and
>>>>> use of emacs -batch to process test org files and verify the output is
>>>>> as expected (with diff or some other tool).
>>>>
>>>> Hey, diff is a good idea!!
>>>>
>>>> I didn't take the verification of the output into account yet :-)
>>>>
>>>> I just pushed a change of Worgs start page, and added a directory
>>>> 'org-tests'. I've placed an index.org there, which now is just a
>>>> collection of ideas (I'm on my day job, so I can't really work on it
>>>> now).
>>>>
>>>> Don't know how often the git repo is published.
>>>> Bernt and Ben, are you 'worgers' allready?
>>>>
>>>> Do you think it makes sense to add snippets and ideas to the new page in
>>>> Worg? I think while the list great to exchange ideas, it's good to have
>>>> a place, where all those ideas are destilled to one-liners.
>>> I must say I am dubious about this. It means, for the tests to be
>>> meaningful, that the output must be a fixed format in base org.
>>
>> Why? If the test bails out 'ERROR', I will have to look for the
>> reason. If the format changed in a legal way => adjust the test.
>
>
> What is an error? And for whom? That the categories are one more tab to
> the right?
In this case:
if I have a project defined in org-publish-projects-alist,
and :base-dir contains a file /foo/bar.org,
and, when publishing,
this file goes to /bar.html (instead of /foo/bar.html),
it is definitively an ERROR. At least in my eyes :-)
> I know I am probably sounding a bit of a killjoy here but
> having been involved in closed source SW development with dedicated QA
> and testing teams it was hard enough then. Maybe I am just being a bit
> of an old curmudgeon here :-(
It will _never_ be that hard, since we are who we are :-D
>>
>>> I doubt
>>> this will ever be the case. The presentation will fluctuate while the
>>> core information (dates, schedules periods etc) will remain pretty much
>>> constant.
>>>
>>> The majority of bugs that I see are often down to people misusing or
>>> using things in the base which are not fully explored. No amount of
>>> regression testing can cover things like that unless the regression
>>> tests include everyones customisations.
Yes, the customizations:
M-x org/test-test-html-export
That's why I wrote (in another mai), we shouldn't urge users to use a
bug tracker. We would just fill the bug tracker with lots of useless
information (from the bug trackers point of view). I love the bug
tracking on the list the way it is.
>> Yes, because Carsten add features en masse :-)
>> I see the testing differently. In the first place we need THINK of
>> testing.
>
> I think we all do. And the main contributors do too.
I didn't.
I didn't even recognize the bug for a while, until I found, that one of
the published org files had the wrong contents. A simple test would have
revealed the bug.
> But it is
> unrealistic to expect a full regression test for each and every release
> unless there is someone willing to step up to the mark and take the time
> to perform them. I personally feel that enough take the "cutting edge"
> and report back before the "official" cuts. We are all beat testers. Its
> the nature of being an emacs and OSS user in the first place :-;
There will always remain parts of the system untested. It'd be
funny to calculate all possible combinations of emacs configuration,
emacs versions and OS version Org-mode is supposed to run on, org mode
configurations and invariants. Seems like a big big number :-D
But the number of invariants itself is not endless! Differences
between OSs are, from an elisp view, minor. Not all the possible
combinations of configuration values are sensible. Those configs
could be added as needed (or simply left up to the users).
Shure, tests are just one piece of the puzzle.
But I'm so happy to see this thread grow, see new names here, see
people editing Worg, learn something myself - that makes me hope, we can
make the best out of it.
>> New Org-revision out?
>> Ahh, OK, I have to the HTML-exports, I want to be working.
>>
>> To do this, I need several different setups, several different data
>> directories (Org-files) and an easy way to test, that doesn't eat my
>> time and gives a result. The quality may vary, but ERRORs will be
>> detected for shure.
>>
>> Not one or mounths later - immediately.
>
> If you don't find it immediately then how often do you use the feature?
> Now I know I probably sound negative. I assure I am not. Maybe the
> earlier word "sceptic" is more applicable ..
I use that feature nearly every day. But the project I export is quite
big and I happend to not visit one of the affected files - hm, maybe I
did, but the contents where not updated (the exporter doesn't delete
files in :publishing-dir - which is OK btw - a test should do that).
When I finally found something was wrong, I did an
rm -r ~/public_html/or-notes
rm ~/.org-timestamps/*
and exported again, I found it was a desaster. Many files missing or in
wrong directories (I think about 20%).
HTML export worked like a charme 6 mounth for me. Suddenly it doesn't
and none of us even noticed. A test for this should be easy to write.
>> If someone installs emacs 23, tries to export to HTML and gets an
>> error....
>
> I have been there too. In fact you found a bug that I left hanging for
> ages because I found a work around ( dont publish from anywhere other
> than the root of the project) - so thanks for your perseverance on that.
>
> I think the best thing people can do is keep a stable version and also
> test the CVS (oops) GIT version too very now and again.
>
>>
>>> Do I think regression testing is important? Yes - in certain
>>> environments. But every time Carsten, you, myself or anyone else fires
>>> up org-mode we are already doing just that.
>> Yes, but we can do better, easier and more complete.
>
> Possibly. I look forward to seeing proposed solutions and would gladly
> lend some time to applying them.
Ahh, Great! Thanks!
Sebastian
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-24 12:13 ` Richard Riley
2008-10-24 15:39 ` Sebastian Rose
@ 2008-10-24 16:27 ` Manish
2008-10-24 18:41 ` Avdi Grimm
1 sibling, 1 reply; 35+ messages in thread
From: Manish @ 2008-10-24 16:27 UTC (permalink / raw)
To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander
On Fri, Oct 24, 2008 at 5:43 PM, Richard Riley wrote:
[snip]
> I think the best thing people can do is keep a stable version and also
> test the CVS (oops) GIT version too very now and again.
+1
That's exactly what I try to do. I update almost every other day, go
through (using tig) the diffs of code, changelog, documentation,
comments.. and see if I can try out and learn something. I also try
and report if something doesn't work right for me with whatever
details I can provide. It really doesn't take too long. I definitely
need to learn more about generating backtraces and debuggers so I
could do a better job at providing right kind and level of details.
With the kind of people we have on the list, we already have a high
quallity Volunteer Powered Massively Parallel Testing System - VPMPTS
(tm) in place. :)
Let me take a step back and think aloud why we need a bug-tracking and
testing system (if only to clarify my understanding) and who/when does
it help. Following scenarios come to mind (and how they can be
handled best (again only my limited understanding)):
1. Someone sees something odd, assumes it's a bug and wants/needs to
report it.
Report it to list, people will help out if it's a configuration or
understanding issue or confirm if it's a repeatble issue.
2. A bug report needs to be confirmed.
VPMPTS goes to work. :)
3. A bug needs to be communicated to the developers.
Developers look at the bug confirmation reports on the list and
decide whether to accept it as a bug or improve our collective
understanding.
4. New functionality needs to be tested for regression.
VPMPTS to rescue!
5. Bug reporter needs to know when the bug is fixed.
Read the list! (Possibly track git repo as well.)
6. Developers needs to sync up about who is working on a specific bug.
Seems like bug tracker will be useful. This is for developers to
comment upon but bug acknowledging developer can be safely assumed
to know the reasons behind the bug and assumed to own it.
I confess I do not have programming or testing background so I may be
completely out of whack here.
IMHO, we should go in the direction of taking on the overhead of
maintaining additional infrastructure only when it's inevitable or
really valuable.
Let me take another step back and recall Carsten't first email on this
thread; he asked for help in 5 specific areas:
1. Providing sufficient details when reporting bugs.
2. Reproducing reported bugs.
3. Sub-system adoption.
4. Answering emails on list.
5. Adding to Worg in form of FAQ, tutorials, and hacks.
I think #1 needs a tutorial (at least for me.) We already do #2 and
#4. #5 needs more work to cover as many features as possible (mea
culpa.) #3 is up to elisp gurus here.
I believe Carsten thought long and hard before he came up with this
list of very specific areas which, IMHO, covers most (if not all) of
the scenarios I conjured up above.
>>
>>> Do I think regression testing is important? Yes - in certain
>>> environments. But every time Carsten, you, myself or anyone else fires
>>> up org-mode we are already doing just that.
>>
>> Yes, but we can do better, easier and more complete.
>
> Possibly. I look forward to seeing proposed solutions and would gladly
> lend some time to applying them.
Ditto.
Please do not assume that I am against any change in our systems and
processes. I am just trying to see the system working in my mind's
eye.
-- Manish
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 16:42 ` Avdi Grimm
2008-10-23 17:33 ` Sebastian Rose
@ 2008-10-24 18:33 ` Ben Alexander
2008-10-24 18:44 ` Avdi Grimm
1 sibling, 1 reply; 35+ messages in thread
From: Ben Alexander @ 2008-10-24 18:33 UTC (permalink / raw)
To: Avdi Grimm; +Cc: emacs-orgmode Org-Mode
On 2008-Oct-23, at 17:42, Avdi Grimm wrote:
>
> If someone would be so kind as to identify a small bug or feature, I
> would be happy to demonstrate this workflow in the form of code, time
> permitting.
>
>
Ok, here's your chance. This is something that has bothered me for
quite some time, but I've never been able to reliably reproduce the
problem. And it's such a small issue.
Sometimes, when it looks like my cursor is at the end of a headline
and I hit tab, the headline does not cycle. Normally it does.
It hit me that the point was not on the headline, it only looked like
it.
So here's the sample test.org file
* First Headline
Some body text
* Second Headline
(goto-char 17) ;that's the end of the first line
(org-cycle)
At this point the buffer looks like:
* First Headline...
* Second Headline
(goto-char 25)
(org-cycle)
I think the buffer should look like this
* First Headline
Some body text
* Second Headline
but it still looks like:
* First Headline...
* Second Headline
* ABOUT THIS BEHAVIOR
I should be honest here. This may not be a bug.
As I created this test data, it became a bit more clear that exactly
where the cursor is matters a lot here. I *promise* that I've seen
this behavior even when the cursor shows up before the ellipses, but
in the test case I set up here, I could not make that happen. I also
found that if I used the key-chord M-x goto-char <RETURN> 25,
everything worked fine. The cursor stayed in front of the ellipses
and the tab key worked. But when I used M-: (goto-char 25) the cursor
moved to the same line as the ellipses, but after them. and the <TAB>
key stopped 'working'.
I finally figured this out while playing with git. I switched
branches at the command line and needed to perform a 'revert-buffer'.
This left the cursor before the ellipses, but unable to properly (org-
cycle) using the <TAB> key.
More experimenting.... I *think* the revert-buffer command tries to
keep point close to the same place, and the org buffer had automatic
folding. The bad thing about this for me is that I'll hit the <TAB>
key four times trying to make sure it's not just an empty tree.
Meanwhile, (org-cycle) has indented the first line of the body, but
hasn't shown me the text I'm changing. Then I get confused about why
the buffer needs saving, probably 10 minutes later when I've forgotten
all about hitting the tab key.
But I'm way out of my depth to try to understand the relationship
between (revert-buffer) (org-cycle) the arrow keys, and all. I just
hit M-< or S-<TAB> and try again. Or C-a, which sometimes works.
Sometimes I grab the mouse as a last resort.
* ABOUT TESTING ISSUES
Just from this example, I'm eager to understand:
1. How can we differentiate between where the cursor appears and the
value of point
2. How can we tell what the users sees on the screen versus what the
buffer contains
So the value of a testing framework is that this: if I'm going to
announce to the world that I can't get something to work, (like the
tab key), I'm going to make darn sure it's a real problem. I'm going
to spend the time to create a small sample file, that reliable creates
the problem, and I'm going to try a few different scenarios. But if
I start to get lost or confused about which settings I've fiddled with
or what is supposed to happen, I'll wander away from the problem and I
won't submit the bug. We all lose in that scenario. But if the
testing framework exists, and it is lightweight enough for novice
emacs users (advanced enough to use M-x but not advanced enough to
read lisp) then maybe I would have used it for this example.
-Ben
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help
2008-10-24 16:27 ` Manish
@ 2008-10-24 18:41 ` Avdi Grimm
0 siblings, 0 replies; 35+ messages in thread
From: Avdi Grimm @ 2008-10-24 18:41 UTC (permalink / raw)
To: Manish; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander
On Fri, Oct 24, 2008 at 12:27 PM, Manish <mailtomanish.sharma@gmail.com> wrote:
> Let me take a step back and think aloud why we need a bug-tracking and
> testing system (if only to clarify my understanding) and who/when does
> it help. Following scenarios come to mind (and how they can be
> handled best (again only my limited understanding)):
First of all, bug tracking are completely separate beasts, and should
be considered separately. You can have either one without the other.
Second: I don't care as much about bug tracking, except for the
following case: if there is a bug tracker and I'm feeling bored and
want to contribute a little, I'll check it for something that I think
I can fix. If there is no such list, I won't do this, and I'll
contribute less. I do *not* follow the mailing list closely enough to
pick up bugs from there except by sheer chance, and this is not going
to change. I really don't care where or how the list is kept, so long
as there is a list.
Third, regarding testing. I'm a coder, I know some ELisp, I *love*
org-mode, and I would like to contribute. That said, I am completely
uninterested in doing manual QA for Org. I am not going to go
clicking around just to look for broken behavior. I have neither the
time nor the interest. However, if there is an automated test suite I
can promise you that a) I will run it before submitting any change I
make; and b) I will add new tests to cover any functionality I add.
However, you should take all of this with a grain of salt because I
haven't submitted any code at all yet.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-24 18:33 ` Ben Alexander
@ 2008-10-24 18:44 ` Avdi Grimm
2008-10-24 19:02 ` Jeff Mickey
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
0 siblings, 2 replies; 35+ messages in thread
From: Avdi Grimm @ 2008-10-24 18:44 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org> wrote:
> Ok, here's your chance. This is something that has bothered me for quite
> some time, but I've never been able to reliably reproduce the problem. And
> it's such a small issue.
Thanks! I'll try to take a look tonight or this weekend.
> So the value of a testing framework is that this: if I'm going to announce
> to the world that I can't get something to work, (like the tab key), I'm
> going to make darn sure it's a real problem. I'm going to spend the time to
> create a small sample file, that reliable creates the problem, and I'm going
> to try a few different scenarios. But if I start to get lost or confused
> about which settings I've fiddled with or what is supposed to happen, I'll
> wander away from the problem and I won't submit the bug. We all lose in
> that scenario. But if the testing framework exists, and it is lightweight
> enough for novice emacs users (advanced enough to use M-x but not advanced
> enough to read lisp) then maybe I would have used it for this example.
For the record, I normally would never expect the user to submit
tests. A description of the problem and some sample data is all I
expect. Writing the a test to reproduce the behavior is the
programmer's responsibility as far as I'm concerned.
--
Avdi
Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-24 18:44 ` Avdi Grimm
@ 2008-10-24 19:02 ` Jeff Mickey
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
1 sibling, 0 replies; 35+ messages in thread
From: Jeff Mickey @ 2008-10-24 19:02 UTC (permalink / raw)
To: Avdi Grimm; +Cc: Ben Alexander, emacs-orgmode Org-Mode
On Fri, Oct 24, 2008 at 14:44, Avdi Grimm <avdi@avdi.org> wrote:
> For the record, I normally would never expect the user to submit
> tests. A description of the problem and some sample data is all I
> expect. Writing the a test to reproduce the behavior is the
> programmer's responsibility as far as I'm concerned.
That's the best part of open source software! A surprising amount of
the time, the user IS a programmer, and thus is able to provide tests.
And it's not that we should require anyone to provide tests before
they report a bug, but they should be required to outline the steps to
reproduce the behavior, so any programmer could take 5 minutes and
quickly whip up a test.
ps: I almost have a baby ert framework for org-mode. It doesn't do
anything yet, but the design is you write a test, and add it to the
org-test-hook to be run by org-test-all.
--
. : [ + carpe diem totus tuus + ] : .
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help
2008-10-23 17:33 ` Sebastian Rose
2008-10-23 19:10 ` Avdi Grimm
@ 2008-10-24 21:09 ` Tom Breton (Tehom)
1 sibling, 0 replies; 35+ messages in thread
From: Tom Breton (Tehom) @ 2008-10-24 21:09 UTC (permalink / raw)
Cc: emacs-orgmode Org-Mode
> "Avdi Grimm" <avdi@avdi.org> writes:
> > A few points, from someone with a decent amount of testing background:
>
> Jippie! Please stay with us for a few days :-D
[...]
As someone with also a decent amount of testing background, I'd like
to answer also.
> Did you work with unit-testing frameworks for elisp already?
> Which one?
I co-wrote regress.el (Wayne Mesard wrote the first version) and I
wrote rtest. I've written some testing for allout.el before I
switched over to org-mode (which is very nice, BTW, Carsten, and
provides pretty much everything I wish-listed for allout mode). Of
particular interest for testing org-mode, I wrote mockbuf.el, which mocks
files and buffers so that tests can deal with files without changing
anything permanently.
> Recommendations?
Hard for me to say objectively.
I prefer my rtest but it's in flux right now. I wouldn't recommend
the current public version, rtest 3.0 I've designed and half written a
better version but I can't really ask you to wait until I get around
to making a better version of rtest. There is a development version
that works and does nice things; it's messy as hell right now and I'm
embarrassed about that. It isn't public but I could put it up if
you'll forgive its messy state,
It has mockbuf, which I found indispensable in writing tests for
allout.el. In fact, I mostly developed it for that, though all the
allout-specific code is elsewhere. Also useful for writing tests:
safe-equal.el, caselet.el, and select.cl, all in the ancillary
directory.
I would steer clear of frameworks that provide many of synonyms for
equality tests, inequality tests, etc. You already know how to write
(not (equal ...)) in elisp. Why learn the name and syntax of
something like (shouldnt-be-equal A B)? Having a lot of that makes a
package LOOK heftier but doesn't do anything more for you. IMO
analyzing test failures should be done by inspecting the test form,
not by making you write things in non-standard elisp that could be
written in standard elisp.
Using ert would mean requiring ewoc, which means tests wouldn't work
on emacs 21. Too bad, because it has some appeal.
> Could say something about the effort to get started with such a
> framework?
For rtest, right now you just need (a) the development version, (b)
unzip it in your emacs-site-lisp or set the paths and autoloads
manually if you prefer, (c) include the following in a file to be
tested:
(when (not (fboundp 'rtest:deftest))
(defmacro rtest:deftest (&rest dummy))
(defmacro rtest:if-avail (&rest dummy)))
Then write tests as:
(rtest:deftest function-name
("Docstring"
(test-form) ;;Return a boolean
)
;;Repeat as wanted
)
That will get you started, but really you'd want to get familiar with
the stuff in mockbuf.el - which is so messy right now that I'm
embarrassed about it, because it contains both old and new versions of
many things in flux.
Other advice: Make a directory to hold short examples of files that
illustrate "before" and "after" states for the various
transformations. Each unit (eg org-agenda.el, org-archive.el etc)
should have its own subtree of examples.
One thing I'd do early is build the following test helper functions:
A macro that runs a test body inside a "let" of all the configuration
options set to expected values.
A function to compare vector-style strings. `equal' doesn't suffice.
rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and
safe-equal-compare-vectors, which IIRC suffice for this.
The ability to write only the visible parts to a separate buffer was
indispensable in testing allout's folding. You already have it in
org-export-visible with the type that keeps the buffer, "?\ ". Then
you can just compare contents.
You will likely have a lot of what I call "output tests", tests of
code that has the property that if the output looks right, it is
right.
IMO with output tests, you need to make it easy on yourself. TDD is
all well and fine, but not if it makes you repeat yourself. If there
is available output that looks right, capture it and use it as a base
for comparison. That's not "pure" but it's far easier and it works
right. If the available output isn't yet right, capture it anyways
and tweak it to make it right - but this rarely happens.
I automated that sort of thing for a Prolog test harness I wrote,
works fine. You'd just write a test of a special type that produced
and captured output. It would originally be ungraded, but you would
then grade it, and it would remember the correct output and compare
against it.
I was planning to add the same to rtest and I partly did, but never
finished it. Still on my todo list.
> Can we add it without changing Org's code? If understand 2. correctly -
yes?
>
rtest's tests like to be in the same file as the code. If you use the
form above, the file still compiles and runs even when rtest is not
available.
It's possible to write the tests in another file, but then either
tests aren't grouped together (must be run one by one) or you have to
do extra work to group them. If they're in file, you can run every
test in a package with `rtest:library'. The other really convenient
way to run a test is rtest:defun-at-point.
> If we can, would we loose quality/speed of tests?
Not at all.
Tom Breton (Tehom)
PS: One more piece of advice: Make an auto-insert or at least a
skeleton that inserts test headers, like the "(when (not (fboundp ..."
stuff above. You'll be using it a fair bit, and it's real nice to
have it just automatically be there when you start a new file. This
advice is geared toward my way of working, though, with the test code
in the same file as the functional code.
^ permalink raw reply [flat|nested] 35+ messages in thread
* org-cycle broken when cursor is at ellipses
2008-10-24 18:44 ` Avdi Grimm
2008-10-24 19:02 ` Jeff Mickey
@ 2008-10-26 19:49 ` Ben Alexander
2008-10-26 21:31 ` Cameron Horsburgh
2008-10-27 8:47 ` Carsten Dominik
1 sibling, 2 replies; 35+ messages in thread
From: Ben Alexander @ 2008-10-26 19:49 UTC (permalink / raw)
To: Avdi Grimm; +Cc: emacs-orgmode Org-Mode
So about this bug...
I think I've figured out why I get this so often. I use Aquamacs on
MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs
Emacs/places.el) a list of files and values for point.
Something about the timing of this means that point is left somewhere
in the body (or maybe subheading?) of a headline, even though org-mode
presents the file in 'Overview'. Since the point is in an unexpected
location, even though the cursor looks fine, (org-cycle) doesn't work
as it looks like it should.
I think I'd like to turn off the 'places.el' thing.
And I reiterate: this is really a tiny issue. Just hitting C-a fixes
the state of point.
On 2008-Oct-24, at 19:44, Avdi Grimm wrote:
> On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org
> > wrote:
>> Ok, here's your chance. This is something that has bothered me for
>> quite
>> some time, but I've never been able to reliably reproduce the
>> problem. And
>> it's such a small issue.
>
> Thanks! I'll try to take a look tonight
...
-Ben
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
@ 2008-10-26 21:31 ` Cameron Horsburgh
2008-10-27 8:47 ` Carsten Dominik
2008-10-27 8:47 ` Carsten Dominik
1 sibling, 1 reply; 35+ messages in thread
From: Cameron Horsburgh @ 2008-10-26 21:31 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
On Sun, Oct 26, 2008 at 07:49:46PM +0000, Ben Alexander wrote:
> So about this bug...
>
> I think I've figured out why I get this so often. I use Aquamacs on
> MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs
> Emacs/places.el) a list of files and values for point.
>
For the record, I'm using emacs-snapshot on Debian sid and I have the
same (or a very similar) issue.
--
Cameron Horsburgh
Blog: http://spiritcry.wordpress.com/
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
2008-10-26 21:31 ` Cameron Horsburgh
@ 2008-10-27 8:47 ` Carsten Dominik
1 sibling, 0 replies; 35+ messages in thread
From: Carsten Dominik @ 2008-10-27 8:47 UTC (permalink / raw)
To: Ben Alexander; +Cc: emacs-orgmode Org-Mode
[-- Attachment #1.1: Type: text/plain, Size: 2304 bytes --]
Hi Ben
this is what I do not like about AquaEmacs, that it installs all kinds
of defaults which you might not want.
You can turn off saveplace with
(setq-default save-place nil)
You might still have to remove places.el.
Or you can make sure that the cursor always starts out at the
beginning of a visible line with
(add-hook 'org-mode-hook
(lambda () (let (org-special-ctrl-a/e) (org-beginning-of-line))))
Then saveplace will get its will by moving the cursor to the recent
location, but the cursor would be moved to the beginning of a visible
line near that location.
Or you can make sure that the point visited by saveplace will be
visible, with
(eval-after-load "saveplace"
'(defadvice save-place-find-file-hook (after org-make-visible
activate)
"Make the position visible."
(org-bookmark-jump-unhide)))
I am actually putting this last piece of code into org.el.
HTH
- Carsten
On Oct 26, 2008, at 8:49 PM, Ben Alexander wrote:
> So about this bug...
>
> I think I've figured out why I get this so often. I use Aquamacs on
> MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs
> Emacs/places.el) a list of files and values for point.
>
> Something about the timing of this means that point is left
> somewhere in the body (or maybe subheading?) of a headline, even
> though org-mode presents the file in 'Overview'. Since the point is
> in an unexpected location, even though the cursor looks fine, (org-
> cycle) doesn't work as it looks like it should.
>
> I think I'd like to turn off the 'places.el' thing.
>
> And I reiterate: this is really a tiny issue. Just hitting C-a
> fixes the state of point.
>
> On 2008-Oct-24, at 19:44, Avdi Grimm wrote:
>
>> On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org
>> > wrote:
>>> Ok, here's your chance. This is something that has bothered me
>>> for quite
>>> some time, but I've never been able to reliably reproduce the
>>> problem. And
>>> it's such a small issue.
>>
>> Thanks! I'll try to take a look tonight
> ...
>
> -Ben
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[-- Attachment #1.2: Type: text/html, Size: 3428 bytes --]
[-- Attachment #2: Type: text/plain, Size: 204 bytes --]
_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses
2008-10-26 21:31 ` Cameron Horsburgh
@ 2008-10-27 8:47 ` Carsten Dominik
0 siblings, 0 replies; 35+ messages in thread
From: Carsten Dominik @ 2008-10-27 8:47 UTC (permalink / raw)
To: Cameron Horsburgh; +Cc: Ben Alexander, emacs-orgmode Org-Mode
[-- Attachment #1.1: Type: text/plain, Size: 780 bytes --]
On Oct 26, 2008, at 10:31 PM, Cameron Horsburgh wrote:
> On Sun, Oct 26, 2008 at 07:49:46PM +0000, Ben Alexander wrote:
>> So about this bug...
>>
>> I think I've figured out why I get this so often. I use Aquamacs on
>> MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs
>> Emacs/places.el) a list of files and values for point.
>>
>
> For the record, I'm using emacs-snapshot on Debian sid and I have the
> same (or a very similar) issue.
Is that also related to saveplace? In general, in outlines there is
the danger that the cursor can be located at an invisible point, but
in practice that should not happen too often.
I myself never press TAB with the cursor at the ellipsis dots, because
you cannot be sure where the cursor truly is.
- Carsten
[-- Attachment #1.2: Type: text/html, Size: 1330 bytes --]
[-- Attachment #2: Type: text/plain, Size: 204 bytes --]
_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2008-10-27 8:47 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-10-23 12:04 How you can help Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
2008-10-23 15:04 ` Sebastian Rose
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
2008-10-23 17:02 ` Sebastian Rose
2008-10-24 12:13 ` Richard Riley
2008-10-24 15:39 ` Sebastian Rose
2008-10-24 16:27 ` Manish
2008-10-24 18:41 ` Avdi Grimm
2008-10-23 19:13 ` Avdi Grimm
2008-10-24 12:19 ` Richard Riley
2008-10-23 16:19 ` Bernt Hansen
2008-10-24 5:05 ` Carsten Dominik
2008-10-23 17:01 ` Jason F. McBrayer
2008-10-23 23:46 ` Eric Schulte
2008-10-23 14:20 ` Sebastian Rose
2008-10-23 14:50 ` Manish
2008-10-23 15:46 ` Eric Schulte
2008-10-23 16:18 ` Avdi Grimm
2008-10-23 14:55 ` Ben Alexander
2008-10-23 16:26 ` Sebastian Rose
2008-10-23 16:42 ` Avdi Grimm
2008-10-23 17:33 ` Sebastian Rose
2008-10-23 19:10 ` Avdi Grimm
2008-10-24 21:09 ` Tom Breton (Tehom)
2008-10-24 18:33 ` Ben Alexander
2008-10-24 18:44 ` Avdi Grimm
2008-10-24 19:02 ` Jeff Mickey
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
2008-10-26 21:31 ` Cameron Horsburgh
2008-10-27 8:47 ` Carsten Dominik
2008-10-27 8:47 ` Carsten Dominik
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
2008-10-23 17:12 ` How you can help Sebastian Rose
2008-10-23 15:08 ` Sebastian Rose
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).