How you can help

How you can help

[Carsten Dominik]

Hi Org users,

I need to get control over the time I spent on developing
Org-mode.  Recently, I have again worked too hard on it, spending
more time than I should, in order to get 6.9 and 6.10 out and to
seize the chance to get the best possible version into Emacs
23.1.

However, this is getting out of control, and I am now putting
myself a hard limit of 1 hour per day, clocked by Org-mode, which
will clearly cut in on my development speed and posting rate.

Here is how you can help me to make the most of that one hour.

1. If you report a bug, please try to do as much work as you can
    yourself.  Before you send it, think if you have collected all
    the information you can.  There are great examples of good bug
    reports on the list already.  The best bug reports are, of
    course, those that are accompanied by a patch.

2. If you are reading the list and see bug reports, consider
    putting in 10 minutes, trying to reproduce this problem
    yourself, and maybe adding information that might be useful
    for me to track down that bug.  Again, this is already
    happening, I am only trying to encourage this type of
    behavior.

3. I have recently spent much time on fixing bugs in parts of Org
    that I did not write myself, so this is taking much more time
    than usually.  If you know Emacs Lisp, and I know there are a
    number of excellent Lisp programmers on this group, consider
    "adopting" one of the following subsystems.  By "adoption", I
    mean that you make it your mission to deeply understand this
    part of Org, so that *you* will be in the position to fix
    bugs, taking that off my shoulders.

    - org-publish.el.  I think the biggest bugs are out now, but I
      am sure this system can be improved quite a bit.  If you are
      that Lisp programmer trying to take up this task, consider
      teaming up with Sebastian Rose who is a great guy, has quite
      some understanding of that system and good ideas about it,
      but just is not enough of a Lisp programmer to really take
      that on.

    - org-export-latex.el.  This is a tough one because you need
      to know both LaTeX and Lisp.  And the code is complex, in
      part because Org-mode allows to write LaTeX is a relatively
      lazy way.  Bastien has done a great great job capturing this
      into an exporter.  However, there are still problems and
      bugs, some came up recently on the list.  And Bastien
      currently does not have the time to contribute consistently.
      As a result, I have started to fix some bugs, but this is
      really eating too much of my time.  This subsystem can also
      use feature additions, for example better handling of image
      insertion, maybe with captions.  I have ideas about this, so
      talk to me if you want to help out.

4. Try to answer as many messages on the mailing list as you can.
    I have been trying hard to make sure that each and every
    reasonable question on the list (and this is really the only
    kind we get in this amazing community) will be answered.
    Doing this still takes a significant fraction of my Org-mode
    development time.  I will clearly spend less time on this in
    the future, in this way also giving others more time to
    answer.  Again, there are already quite a few people who
    regularly *answer* questions, and all I am trying to do here
    is encouraging this activity.

5. Help developing the Org-mode FAQ by adding useful information
    to it yourself.  All you need to do is to get acces to Worg,
    which will help getting to know git in the process.  Worg is
    meant to be user edited, so please go wild and add information
    and links at will.

Thanks to you all, for your understanding and help.

- Carsten

Re: How you can help

[Manish]

On Thu, Oct 23, 2008 at 1:05 PM, Carsten Dominik wrote:
> Hi Org users,
>
> I need to get control over the time I spent on developing
> Org-mode.  Recently, I have again worked too hard on it, spending
> more time than I should, in order to get 6.9 and 6.10 out and to
> seize the chance to get the best possible version into Emacs
> 23.1.
>
> However, this is getting out of control, and I am now putting
> myself a hard limit of 1 hour per day, clocked by Org-mode, which
> will clearly cut in on my development speed and posting rate.
>
> Here is how you can help me to make the most of that one hour.
>
> 1. If you report a bug, please try to do as much work as you can
>   yourself.  Before you send it, think if you have collected all
>   the information you can.  There are great examples of good bug
>   reports on the list already.  The best bug reports are, of
>   course, those that are accompanied by a patch.
>
> 2. If you are reading the list and see bug reports, consider
>   putting in 10 minutes, trying to reproduce this problem
>   yourself, and maybe adding information that might be useful
>   for me to track down that bug.  Again, this is already
>   happening, I am only trying to encourage this type of
>   behavior.
>
> 3. I have recently spent much time on fixing bugs in parts of Org
>   that I did not write myself, so this is taking much more time
>   than usually.  If you know Emacs Lisp, and I know there are a
>   number of excellent Lisp programmers on this group, consider
>   "adopting" one of the following subsystems.  By "adoption", I
>   mean that you make it your mission to deeply understand this
>   part of Org, so that *you* will be in the position to fix
>   bugs, taking that off my shoulders.
>
>   - org-publish.el.  I think the biggest bugs are out now, but I
>     am sure this system can be improved quite a bit.  If you are
>     that Lisp programmer trying to take up this task, consider
>     teaming up with Sebastian Rose who is a great guy, has quite
>     some understanding of that system and good ideas about it,
>     but just is not enough of a Lisp programmer to really take
>     that on.
>
>   - org-export-latex.el.  This is a tough one because you need
>     to know both LaTeX and Lisp.  And the code is complex, in
>     part because Org-mode allows to write LaTeX is a relatively
>     lazy way.  Bastien has done a great great job capturing this
>     into an exporter.  However, there are still problems and
>     bugs, some came up recently on the list.  And Bastien
>     currently does not have the time to contribute consistently.
>     As a result, I have started to fix some bugs, but this is
>     really eating too much of my time.  This subsystem can also
>     use feature additions, for example better handling of image
>     insertion, maybe with captions.  I have ideas about this, so
>     talk to me if you want to help out.
>
> 4. Try to answer as many messages on the mailing list as you can.
>   I have been trying hard to make sure that each and every
>   reasonable question on the list (and this is really the only
>   kind we get in this amazing community) will be answered.
>   Doing this still takes a significant fraction of my Org-mode
>   development time.  I will clearly spend less time on this in
>   the future, in this way also giving others more time to
>   answer.  Again, there are already quite a few people who
>   regularly *answer* questions, and all I am trying to do here
>   is encouraging this activity.
>
> 5. Help developing the Org-mode FAQ by adding useful information
>   to it yourself.  All you need to do is to get acces to Worg,
>   which will help getting to know git in the process.  Worg is
>   meant to be user edited, so please go wild and add information
>   and links at will.
>
> Thanks to you all, for your understanding and help.

It is quite a challenge to try out and understand numerous features
and facilities already in Org and the pace at which the new changes
were coming was unreal and super-human.  The backlog of things I
wanted to try, understand and absorb in my workflow was consistently
increasing.  This is not a complaint; I am having so much fun.  *BUT*
I do not want my favourite developer of my favourite package in my
favourite editor to burn out.  Please do take it easy.

Unfortunately, I can not help with #3 (adopting a subsystem) since I
am Lisp-challenged but with rest I will.

Sincere thanks for all your time, effort and sacrifices.

-- 
Manish

Re: How you can help

[Sebastian Rose]

Should we put a page on worg by this name (see subject of this
thread)? We could show, how to turn on debugging, write a good bug
report, link to good elisp tutorials and describe how to use the elisp
debugger in emacs for simple debugging.

If no one stops me, I'll do it these days. As or me, a page like that
would be helpfull. I'll be back for report - maybe one of the lisp
freaks could add some details later on?


Regards,

   Sebastian



Carsten Dominik wrote:
> Hi Org users,
> 
> I need to get control over the time I spent on developing
> Org-mode.  Recently, I have again worked too hard on it, spending
> more time than I should, in order to get 6.9 and 6.10 out and to
> seize the chance to get the best possible version into Emacs
> 23.1.
> 
> However, this is getting out of control, and I am now putting
> myself a hard limit of 1 hour per day, clocked by Org-mode, which
> will clearly cut in on my development speed and posting rate.
> 
> Here is how you can help me to make the most of that one hour.
> 
> 1. If you report a bug, please try to do as much work as you can
>    yourself.  Before you send it, think if you have collected all
>    the information you can.  There are great examples of good bug
>    reports on the list already.  The best bug reports are, of
>    course, those that are accompanied by a patch.
> 
> 2. If you are reading the list and see bug reports, consider
>    putting in 10 minutes, trying to reproduce this problem
>    yourself, and maybe adding information that might be useful
>    for me to track down that bug.  Again, this is already
>    happening, I am only trying to encourage this type of
>    behavior.
> 
> 3. I have recently spent much time on fixing bugs in parts of Org
>    that I did not write myself, so this is taking much more time
>    than usually.  If you know Emacs Lisp, and I know there are a
>    number of excellent Lisp programmers on this group, consider
>    "adopting" one of the following subsystems.  By "adoption", I
>    mean that you make it your mission to deeply understand this
>    part of Org, so that *you* will be in the position to fix
>    bugs, taking that off my shoulders.
> 
>    - org-publish.el.  I think the biggest bugs are out now, but I
>      am sure this system can be improved quite a bit.  If you are
>      that Lisp programmer trying to take up this task, consider
>      teaming up with Sebastian Rose who is a great guy, has quite
>      some understanding of that system and good ideas about it,
>      but just is not enough of a Lisp programmer to really take
>      that on.
> 
>    - org-export-latex.el.  This is a tough one because you need
>      to know both LaTeX and Lisp.  And the code is complex, in
>      part because Org-mode allows to write LaTeX is a relatively
>      lazy way.  Bastien has done a great great job capturing this
>      into an exporter.  However, there are still problems and
>      bugs, some came up recently on the list.  And Bastien
>      currently does not have the time to contribute consistently.
>      As a result, I have started to fix some bugs, but this is
>      really eating too much of my time.  This subsystem can also
>      use feature additions, for example better handling of image
>      insertion, maybe with captions.  I have ideas about this, so
>      talk to me if you want to help out.
> 
> 4. Try to answer as many messages on the mailing list as you can.
>    I have been trying hard to make sure that each and every
>    reasonable question on the list (and this is really the only
>    kind we get in this amazing community) will be answered.
>    Doing this still takes a significant fraction of my Org-mode
>    development time.  I will clearly spend less time on this in
>    the future, in this way also giving others more time to
>    answer.  Again, there are already quite a few people who
>    regularly *answer* questions, and all I am trying to do here
>    is encouraging this activity.
> 
> 5. Help developing the Org-mode FAQ by adding useful information
>    to it yourself.  All you need to do is to get acces to Worg,
>    which will help getting to know git in the process.  Worg is
>    meant to be user edited, so please go wild and add information
>    and links at will.
> 
> Thanks to you all, for your understanding and help.
> 
> - Carsten
> 
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> 

Re: How you can help

[Sebastian Rose]

Sebastian Rose wrote:
> Should we put a page on worg by this name (see subject of this
> thread)? We could show, how to turn on debugging, write a good bug
> report, link to good elisp tutorials and describe how to use the elisp
> debugger in emacs for simple debugging.
> 
> If no one stops me, I'll do it these days. As or me, a page like that
> would be helpfull. I'll be back for report - maybe one of the lisp
> freaks could add some details later on?



Hm - there is a link to the 'Feedback' section of the manual
(http://orgmode.org/org.html#Feedback) on
http://orgmode.org/worg/org-contribute.php but

  1. It's hard to find. One has find worg (http://orgmode.org/worg/),
     and then click 'How to contribute to Org?' to get to
     http://orgmode.org/worg/org-contribute.php where the link is.
     I think I wouldn't follow the link 'How to contribute to Org', if
     I was to report a bug (maybe it's my bad english, but I lots of
     people whos english is even worse).

  2. While the 'Feedback' section in the manual is as good as the rest
     of it, it's short and does, for good reasons, not provide more
     information, than absolutely needed.


Suggestions:

   1. on Worgs startpage (http://orgmode.org/worg/): add an extra section
      (as the FIRST one) 'How you can help'.

   2. Under that headline, put the link to the 'Feedback' section of the
      manual.

   3. Add more links to one or more pages for tips on debugging etc.

   4. Add subdirectory for those debugging tips. We could also add things
      like coding conventions, notes about the APIs in org etc. to that
      directory.



....Please comment...




   5. I also think of little packages for testing parts of org.

      Example XHTML-export:

      When the HTML-exporter didn't work lately, I set up a little
      directory or testing. On top of that tree I had an Org-file with
      [[elisp:]] links to click on, like
      [[elisp:(setq debug-on-error t)]]
      or [[elisp:(debug-on-entry 'org-export-org-to)]] and one for
      loading a setup.el in the same directory (primaryly to adjust
      org-publish-project-alist - in principle, everything could be
      setup in that file, but needed two different setups for these
      tests).

      Now I can just open that Org-file, click on two or three links
      and get a backtrace, if something is wrong. It's so simple to
      test the exporter this way, that it takes 30 seconds or even less
      (on failure :-)

      With simple trees like that (eventually downloadable as *.tar.bz2)
      one may test a module with all the little corner cases, which
      might not be in the Org-files the tester uses for his daily work
      and without risk.

      If we encounter a new corner case, we could just add it to the
      test packages Org-files and document it in the packages index.org
      (or README.org) to prevent a regression.

      When working on a module, one could programm against those tests.



Creating and maintaining those test packages is a simple thing to do.
All it requires is knowledge of USING org and just a tiny little extra.
Hence I suppose this work should be done by the community/worgers.

It will be much more fun to add tested patches (and more secure to
create one)!






Regards,

   Sebastian


> Carsten Dominik wrote:
>> Hi Org users,
>>
>> I need to get control over the time I spent on developing
>> Org-mode.  Recently, I have again worked too hard on it, spending
>> more time than I should, in order to get 6.9 and 6.10 out and to
>> seize the chance to get the best possible version into Emacs
>> 23.1.
>>
>> However, this is getting out of control, and I am now putting
>> myself a hard limit of 1 hour per day, clocked by Org-mode, which
>> will clearly cut in on my development speed and posting rate.
>>
>> Here is how you can help me to make the most of that one hour.
>>
>> 1. If you report a bug, please try to do as much work as you can
>>    yourself.  Before you send it, think if you have collected all
>>    the information you can.  There are great examples of good bug
>>    reports on the list already.  The best bug reports are, of
>>    course, those that are accompanied by a patch.
>>
>> 2. If you are reading the list and see bug reports, consider
>>    putting in 10 minutes, trying to reproduce this problem
>>    yourself, and maybe adding information that might be useful
>>    for me to track down that bug.  Again, this is already
>>    happening, I am only trying to encourage this type of
>>    behavior.
>>
>> 3. I have recently spent much time on fixing bugs in parts of Org
>>    that I did not write myself, so this is taking much more time
>>    than usually.  If you know Emacs Lisp, and I know there are a
>>    number of excellent Lisp programmers on this group, consider
>>    "adopting" one of the following subsystems.  By "adoption", I
>>    mean that you make it your mission to deeply understand this
>>    part of Org, so that *you* will be in the position to fix
>>    bugs, taking that off my shoulders.
>>
>>    - org-publish.el.  I think the biggest bugs are out now, but I
>>      am sure this system can be improved quite a bit.  If you are
>>      that Lisp programmer trying to take up this task, consider
>>      teaming up with Sebastian Rose who is a great guy, has quite
>>      some understanding of that system and good ideas about it,
>>      but just is not enough of a Lisp programmer to really take
>>      that on.
>>
>>    - org-export-latex.el.  This is a tough one because you need
>>      to know both LaTeX and Lisp.  And the code is complex, in
>>      part because Org-mode allows to write LaTeX is a relatively
>>      lazy way.  Bastien has done a great great job capturing this
>>      into an exporter.  However, there are still problems and
>>      bugs, some came up recently on the list.  And Bastien
>>      currently does not have the time to contribute consistently.
>>      As a result, I have started to fix some bugs, but this is
>>      really eating too much of my time.  This subsystem can also
>>      use feature additions, for example better handling of image
>>      insertion, maybe with captions.  I have ideas about this, so
>>      talk to me if you want to help out.
>>
>> 4. Try to answer as many messages on the mailing list as you can.
>>    I have been trying hard to make sure that each and every
>>    reasonable question on the list (and this is really the only
>>    kind we get in this amazing community) will be answered.
>>    Doing this still takes a significant fraction of my Org-mode
>>    development time.  I will clearly spend less time on this in
>>    the future, in this way also giving others more time to
>>    answer.  Again, there are already quite a few people who
>>    regularly *answer* questions, and all I am trying to do here
>>    is encouraging this activity.
>>
>> 5. Help developing the Org-mode FAQ by adding useful information
>>    to it yourself.  All you need to do is to get acces to Worg,
>>    which will help getting to know git in the process.  Worg is
>>    meant to be user edited, so please go wild and add information
>>    and links at will.
>>
>> Thanks to you all, for your understanding and help.
>>
>> - Carsten
>>
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
> 
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> 

Re: How you can help

[Ben Alexander]

Sebastian Rose wrote:
 >  5. I also think of little packages for testing parts of org.

I'm curious if you or someone else has any ideas for writing automated  
tests for org-mode.  I haven't the foggiest idea how someone would  
write a test for the parts of org that control what is displayed on  
the screen.  I mean, when the bug is 'it doesn't look right' how can  
you tell?

Perhaps the git repository should have a small collection of small org- 
mode files that reproduce certain bugs?  If there were some examples  
of how to create such a test, then perhaps bug reporters would find it  
much easier to create them.

I do see some confusing issues due to different configuration files.   
So creating a test file might involve making sure org-mode doesn't  
read any configuration (how do you do that?) and possible asking org- 
mode to extract all the configuration variables it has right now and  
dump them into a test file (...and how do you do that?)

Re: How you can help

[Bernt Hansen]

Ben Alexander <bva@alexanderonline.org> writes:

> Sebastian Rose wrote:
>>  5. I also think of little packages for testing parts of org.
>
> I'm curious if you or someone else has any ideas for writing automated
> tests for org-mode.  I haven't the foggiest idea how someone would
> write a test for the parts of org that control what is displayed on
> the screen.  I mean, when the bug is 'it doesn't look right' how can
> you tell?
>
> Perhaps the git repository should have a small collection of small
> org- 
> mode files that reproduce certain bugs?  If there were some examples
> of how to create such a test, then perhaps bug reporters would find it
> much easier to create them.
>
> I do see some confusing issues due to different configuration files.
> So creating a test file might involve making sure org-mode doesn't
> read any configuration (how do you do that?) and possible asking org- 
> mode to extract all the configuration variables it has right now and
> dump them into a test file (...and how do you do that?)

Running a minimal emacs should suppress custom config files:
         emacs -q -l yourtest.el

Some kind of regression testing framework would be awesome.  Org-mode is
large enough that this is almost a necessity to keep things stable and
bug-free.

Maybe something can be put together from the git testing framework and
use of emacs -batch to process test org files and verify the output is
as expected (with diff or some other tool).

-Bernt

Re: How you can help

[Sebastian Rose]

Hi Ben,

I cc'ed the list.


The tests I described in my email to the list are not automated. The
reason for that is my lack of (e)lisp knowledge.

BUT, they where easy to handle for non programmers. I think the little
test will make it to the worg site this week, when all private data is
removed. You could take a look at it by then. And: improve, improve,
improve... :-)

If you know of someone who knows how to do automated tests in elisp, or
some technique, package, whatever, please post it to the list, so we all
can take a look at it and comment->decide something. This is _highly_
_appreciated_.


Ben Alexander <bva@alexanderonline.org> writes:
> Sebastian Rose wrote:
>>  5. I also think of little packages for testing parts of org.
>
> I'm curious if you or someone else has any ideas for writing automated tests for
> org-mode.  I haven't the foggiest idea how someone would  write a test for the
> parts of org that control what is displayed on  the screen.  I mean, when the
> bug is 'it doesn't look right' how can  you tell?

Believe me, my idea is the foggiest of all possible ideas ;-) (so where
my ideas of JavaScript some mounth ago), so I won't be of much help, I
fear.


> Perhaps the git repository should have a small collection of small org-
> mode files that reproduce certain bugs?  If there were some examples of how to
> create such a test, then perhaps bug reporters would find it  much easier to
> create them.

YES! Exactly. Every corner case an Org-file, every bug an Org-file.
_DATA_ for testing is something, everyone _can_ provide.

But git later, yes, maybe.

Since this would need Carsten to think and act on this, I feel Worg is a
nice place to start the first expieriments. We need Carstens power for
other things (when will Org-mode finaly wash my car? It's repeater-TODO,
but nothing happens!!!)

Basically, I'd try to keep the the testing as simple as possible, and
try to get elisp programmers to help with the code. We should try to
keep the hurdles for testers as humble (?) as possible, and put all
that's needed to be helpfull on one page in worg.

I recently discovered the very unautomated `(print object)' in the elisp
reference. Not everything can be done automated, maybe, but if I would
have known this stupid `print' function, I would know more about elisp
and some files in org already. And it would have been faster and easier
to create the two minor patches I wrote. This is, where the 'links to
elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy.

Willingness to help is no problem, as the Org-community reveals. As for
me, it's often a lack of time and/or knowledge, that prevents it.

And the aim of all this is to help helping, in means of good and
detailed bug reports in the first place.




> I do see some confusing issues due to different configuration files.  So
> creating a test file might involve making sure org-mode doesn't  read any
> configuration (how do you do that?) and possible asking org-
> mode to extract all the configuration variables it has right now and dump them
> into a test file (...and how do you do that?)

True.

Hm - to test without configuration, we already have `emacs -q'.
Dumping the variables is just a list of (print var) statements. A
(quite) complete list of variables could be extracted with grep?

e.g.:

grep -r defvar lisp/

Once we have such a list, we could set/reset some/all variables.

This will not be perfect, but could work reasonable well.

And yes, it would indeed be nice, to have elisp to reliably reset
emacs/org configuration, so one could do several different test in
sequence. Preferably they would _always_ run and dump all errors to some
file or buffer, even if one or more tests fail.

I think elsip provides funtionalities to handle those errors and skip to
the next test. This would be a huge step.






But still, we have to test with different _DATA_ too! The test I
described is just a quick hack I did to do the testing, while Carsten
was bug-hunting on the other end of my internet connection.

The XHTML-export test was much easier to do, then some other tests we
might need. In the end, it was a test, automated or not. And I had to do
it, because I'm one out of a few who use org to maintain a web-site
(locally in my case) and export an entire directory tree with lots of
_DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This
became obvious in this case. I was the one who even noticed the bug,
which means, no one else was using recursive XHTML export for a while,
or their setup didn't reveal that bug.

So clearly I was the one to provide some support for this very part of
the system. I can't rely on the assumption, that some maintainer has all
possible setups at hand all the time (maybe this was possible years
ago).

While the testing of the HTML-export was quite simple, it requires a lot
of _DATA_. Namly files and directories + different setups, per file
setups, #+SETUPFILE, images, with/without org-info.js, extra styles set
or not... to have a realistic testing environment (the test didn't have
all of these...).




A summary could be, that it's nice to provide different setups for
maintainers and testers.

 - *Easy to install* (unpack and done),
 - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set
   everything up)
 - and *easy to download* once they are removed again (a central place on
   Worg).
 - *Corner cases*, like the 'empty line before first heading bug' in the
   LaTeX exporter recently. 




Regards,


--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.roseATemma-stil.de, sebastian_roseATgmx.de
Http:  www.emma-stil.de

Re: How you can help

[Manish]

  On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote:
  [snip]
  >
  > If you know of someone who knows how to do automated tests in
  > elisp, or some technique, package, whatever, please post it to the
  > list, so we all can take a look at it and comment->decide
  > something. This is _highly_ _appreciated_.

We could start here:
http://www.emacswiki.org/emacs-fr/UnitTesting#toc2

-- Manish

Re: How you can help

[Ben Alexander]

Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting

Unfortunately for me, I can't tell if Emacs comes with any builtin  
framework already, so I downloaded one of the many options listed on  
that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el

The personal issue I have is that I'm on a Mac, using Aquamacs, and  
the command line version of emacs is a different binary, so there  
might be trouble when a test passes at the command line, but fails  
where it matters to me. I don't even make sure that org-mode is up to  
date at the command line.  I thought it wasn't, but I just checked and  
now it is.  Plus, I don't really understand internals of emacs (like  
basic internals: I understand point and mark, buffer and file, but not  
transient mark, indirect buffer, symbols vs strings, window vs tab vs  
frame)

The tutorial I'd need to write a test is one which lays out code I  
could copy and paste to do the following

* setup the test environment
   - create a test directory
   - create a sample test.org file
   - put the cursor in a particular place

* run the command we need to test
   - hit the 'TAB' key, or C-c C-c (some folks might need to be  
reminded how to find out exactly what command is actually being run  
when you hit a keystroke.  And some of me might need to be told what  
lisp-code to use when the keystroke runs different commands at  
different places in a file)
   - reformat a table
   - clock in/out
   - create the agenda
   - export .html .ics .dvi file
* How do we specify the correct result???
   - check that the headline folded properly.  What's the lisp code  
for getting the folded string as displayed?
   - check that the cursor is where it should be? especially when the  
cursor is near elipses...
   - check that the agenda is built properly.  What's the lisp code  
for getting the agenda as a string?
   - check that the exported files are correct.  Maybe the right  
suggestion is to run the export on two different files, so the test  
can focus on the 'diff' between them.  That way different people who  
run the same test on different hosts can get the same result.

More experienced folks must suggest whether it's better to have the  
expectation of a test be specified as a string in the test code or as  
a separate file in the test directory.  It might depend on context, so  
feel free to lay out three or four cases.

If even one kind of test for each of these 'cases' exists, hopefully  
we can encourage people to report bugs with the test.  I mean, if  
people are already going through the effort to create the small sample  
org file to demonstrate their bug, and it's only a short (obvious)  
step to translating their english description of what should happen  
into equivalent lisp code.... that seems like a big temptation for  
someone to do it.

Plus it might help them convince themselves that it really is a bug,  
and not that they are going crazy.

I gotta move on for the day here, but I'll keep thinking a bit about  
this.

On 2008-Oct-23, at 15:20, Sebastian Rose wrote:

> Hi Ben,
>
> I cc'ed the list.
>
>
> The tests I described in my email to the list are not automated. The
> reason for that is my lack of (e)lisp knowledge.
>
> BUT, they where easy to handle for non programmers. I think the little
> test will make it to the worg site this week, when all private data is
> removed. You could take a look at it by then. And: improve, improve,
> improve... :-)
>
> If you know of someone who knows how to do automated tests in elisp,  
> or
> some technique, package, whatever, please post it to the list, so we  
> all
> can take a look at it and comment->decide something. This is _highly_
> _appreciated_.
>
>
> Ben Alexander <bva@alexanderonline.org> writes:
>> Sebastian Rose wrote:
>>> 5. I also think of little packages for testing parts of org.
>>
>> I'm curious if you or someone else has any ideas for writing  
>> automated tests for
>> org-mode.  I haven't the foggiest idea how someone would  write a  
>> test for the
>> parts of org that control what is displayed on  the screen.  I  
>> mean, when the
>> bug is 'it doesn't look right' how can  you tell?
>
> Believe me, my idea is the foggiest of all possible ideas ;-) (so  
> where
> my ideas of JavaScript some mounth ago), so I won't be of much help, I
> fear.
>
>
>> Perhaps the git repository should have a small collection of small  
>> org-
>> mode files that reproduce certain bugs?  If there were some  
>> examples of how to
>> create such a test, then perhaps bug reporters would find it  much  
>> easier to
>> create them.
>
> YES! Exactly. Every corner case an Org-file, every bug an Org-file.
> _DATA_ for testing is something, everyone _can_ provide.
>
> But git later, yes, maybe.
>
> Since this would need Carsten to think and act on this, I feel Worg  
> is a
> nice place to start the first expieriments. We need Carstens power for
> other things (when will Org-mode finaly wash my car? It's repeater- 
> TODO,
> but nothing happens!!!)
>
> Basically, I'd try to keep the the testing as simple as possible, and
> try to get elisp programmers to help with the code. We should try to
> keep the hurdles for testers as humble (?) as possible, and put all
> that's needed to be helpfull on one page in worg.
>
> I recently discovered the very unautomated `(print object)' in the  
> elisp
> reference. Not everything can be done automated, maybe, but if I would
> have known this stupid `print' function, I would know more about elisp
> and some files in org already. And it would have been faster and  
> easier
> to create the two minor patches I wrote. This is, where the 'links to
> elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy.
>
> Willingness to help is no problem, as the Org-community reveals. As  
> for
> me, it's often a lack of time and/or knowledge, that prevents it.
>
> And the aim of all this is to help helping, in means of good and
> detailed bug reports in the first place.
>
>
>
>
>> I do see some confusing issues due to different configuration  
>> files.  So
>> creating a test file might involve making sure org-mode doesn't   
>> read any
>> configuration (how do you do that?) and possible asking org-
>> mode to extract all the configuration variables it has right now  
>> and dump them
>> into a test file (...and how do you do that?)
>
> True.
>
> Hm - to test without configuration, we already have `emacs -q'.
> Dumping the variables is just a list of (print var) statements. A
> (quite) complete list of variables could be extracted with grep?
>
> e.g.:
>
> grep -r defvar lisp/
>
> Once we have such a list, we could set/reset some/all variables.
>
> This will not be perfect, but could work reasonable well.
>
> And yes, it would indeed be nice, to have elisp to reliably reset
> emacs/org configuration, so one could do several different test in
> sequence. Preferably they would _always_ run and dump all errors to  
> some
> file or buffer, even if one or more tests fail.
>
> I think elsip provides funtionalities to handle those errors and  
> skip to
> the next test. This would be a huge step.
>
>
>
>
>
>
> But still, we have to test with different _DATA_ too! The test I
> described is just a quick hack I did to do the testing, while Carsten
> was bug-hunting on the other end of my internet connection.
>
> The XHTML-export test was much easier to do, then some other tests we
> might need. In the end, it was a test, automated or not. And I had  
> to do
> it, because I'm one out of a few who use org to maintain a web-site
> (locally in my case) and export an entire directory tree with lots of
> _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This
> became obvious in this case. I was the one who even noticed the bug,
> which means, no one else was using recursive XHTML export for a while,
> or their setup didn't reveal that bug.
>
> So clearly I was the one to provide some support for this very part of
> the system. I can't rely on the assumption, that some maintainer has  
> all
> possible setups at hand all the time (maybe this was possible years
> ago).
>
> While the testing of the HTML-export was quite simple, it requires a  
> lot
> of _DATA_. Namly files and directories + different setups, per file
> setups, #+SETUPFILE, images, with/without org-info.js, extra styles  
> set
> or not... to have a realistic testing environment (the test didn't  
> have
> all of these...).
>
>
>
>
> A summary could be, that it's nice to provide different setups for
> maintainers and testers.
>
> - *Easy to install* (unpack and done),
> - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set
>   everything up)
> - and *easy to download* once they are removed again (a central  
> place on
>   Worg).
> - *Corner cases*, like the 'empty line before first heading bug' in  
> the
>   LaTeX exporter recently.
>
>
>
>
> Regards,
>
>
> --
> Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449  
> Hannover
>
> Tel.:  +49 (0)511 - 36 58 472
> Fax:   +49 (0)1805 - 233633 - 11044
> mobil: +49 (0)173 - 83 93 417
> Email: s.roseATemma-stil.de, sebastian_roseATgmx.de
> Http:  www.emma-stil.de

Re: How you can help

[Sebastian Rose]

Bernt Hansen <bernt@norang.ca> writes:
> Running a minimal emacs should suppress custom config files:
>          emacs -q -l yourtest.el

Added this one to the Clippboard section on new org-tests/index.org in
Worg.git. (this section will be temporary...)

> Some kind of regression testing framework would be awesome.  Org-mode is
> large enough that this is almost a necessity to keep things stable and
> bug-free.

It's big and feature-RICH.

> Maybe something can be put together from the git testing framework and
> use of emacs -batch to process test org files and verify the output is
> as expected (with diff or some other tool).


Hey, diff is a good idea!!

I didn't take the verification of the output into account yet :-)

I just pushed a change of Worgs start page, and added a directory
'org-tests'. I've placed an index.org there, which now is just a
collection of ideas (I'm on my day job, so I can't really work on it
now).

Don't know how often the git repo is published.
Bernt and Ben, are you 'worgers' allready?

Do you think it makes sense to add snippets and ideas to the new page in
Worg? I think while the list great to exchange ideas, it's good to have
a place, where all those ideas are destilled to one-liners.


Cheers,

-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http:  www.emma-stil.de

Re: How you can help

[Sebastian Rose]

Ben Alexander <bva@alexanderonline.org> writes:
> Perhaps the git repository should have a small collection of small org-
> mode files that reproduce certain bugs?

Ah, stupid me, Worg IS a git repo :-D



-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http:  www.emma-stil.de

Re: How you can help

[Russell Adams]

On Thu, Oct 23, 2008 at 09:35:46AM +0200, Carsten Dominik wrote:
> Hi Org users,
>
> I need to get control over the time I spent on developing
> Org-mode.  Recently, I have again worked too hard on it, spending
> more time than I should, in order to get 6.9 and 6.10 out and to
> seize the chance to get the best possible version into Emacs
> 23.1.
>
> However, this is getting out of control, and I am now putting
> myself a hard limit of 1 hour per day, clocked by Org-mode, which
> will clearly cut in on my development speed and posting rate.

Carsten, you need a break sometime too. ;]

>    - org-export-latex.el.  This is a tough one because you need
>      to know both LaTeX and Lisp.  And the code is complex, in
>      part because Org-mode allows to write LaTeX is a relatively
>      lazy way.  Bastien has done a great great job capturing this
>      into an exporter.  However, there are still problems and
>      bugs, some came up recently on the list.  And Bastien
>      currently does not have the time to contribute consistently.
>      As a result, I have started to fix some bugs, but this is
>      really eating too much of my time.  This subsystem can also
>      use feature additions, for example better handling of image
>      insertion, maybe with captions.  I have ideas about this, so
>      talk to me if you want to help out.

As you've discovered I'm willing to test latex issues, but I don't
know Lisp. Whoever picks this up can continue to use me for testing.

> Thanks to you all, for your understanding and help.
>
> - Carsten

You still rock! 

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: How you can help

[Eric Schulte]

Hi,

I'd recommend ert.el, it is actively maintained, and I have it on good
authority that this the tool of choice for elisp unit testing.

http://github.com/ohler/ert/tree/master

,----[from ert.el]
| ;;; Commentary:
|  
| ;; ERT is a tool for automated testing in Emacs Lisp. Its main
| ;; features are facilities for defining and running test cases and
| ;; reporting the results as well as for debugging test failures
| ;; interactively.
| ;;
| ;; The main entry points are `ert-deftest', which is similar to
| ;; `defun' but defines a test, and `ert-run-tests-interactively',
| ;; which runs tests and offers an interactive interface for inspecting
| ;; results and debugging. There is also `ert-run-tests-batch' for
| ;; non-interactive use.
| ;;
| ;; The body of `ert-deftest' forms resembles a function body, but the
| ;; additional operators `should', `should-not' and `should-error' are
| ;; available. `should' is similar to cl's `assert', but signals a
| ;; different error when its condition is violated that is caught and
| ;; processed by ERT. In addition, it analyzes its argument form and
| ;; records information that helps debugging (`assert' tries to do
| ;; something similar when its second argument SHOW-ARGS is true, but
| ;; `should' is more sophisticated). For information on `should-not'
| ;; and `should-error', see their docstrings.
`----


Manish <mailtomanish.sharma@gmail.com> writes:

>   On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote:
>   [snip]
>   >
>   > If you know of someone who knows how to do automated tests in
>   > elisp, or some technique, package, whatever, please post it to the
>   > list, so we all can take a look at it and comment->decide
>   > something. This is _highly_ _appreciated_.
>
> We could start here:
> http://www.emacswiki.org/emacs-fr/UnitTesting#toc2
>
> -- Manish
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: How you can help

[Robert Goldman]

> Date: Thu, 23 Oct 2008 15:55:41 +0100
> From: Ben Alexander <bva@alexanderonline.org>
> Subject: Re: [Orgmode] How you can help
> To: Sebastian Rose <sebastian_rose@gmx.de>
> Cc: emacs-orgmode Org-Mode <emacs-orgmode@gnu.org>
> Message-ID: <752DA813-1B66-4FD1-B28E-3C23176BA13D@alexanderonline.org>
> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
> 
> Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting
> 
> Unfortunately for me, I can't tell if Emacs comes with any builtin  
> framework already, so I downloaded one of the many options listed on  
> that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el
> 
> The personal issue I have is that I'm on a Mac, using Aquamacs, and  
> the command line version of emacs is a different binary, so there  
> might be trouble when a test passes at the command line, but fails  
> where it matters to me. I don't even make sure that org-mode is up to  
> date at the command line.  I thought it wasn't, but I just checked and  
> now it is.  Plus, I don't really understand internals of emacs (like  
> basic internals: I understand point and mark, buffer and file, but not  
> transient mark, indirect buffer, symbols vs strings, window vs tab vs  
> frame)

Actually, if you want, you *can* run Aquamacs from the command line, but
it can be a pain to do it.  I figured out how to do this when I was
trying to use the Makefile for org-mode.  I ended up with the following
emacs command-line:

EMACS=/Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\ Emacs

and this line for batchmode compiling.  Note that I had to augment the
standard emacs command-line -q option with Aquamacs' -Q:

BATCH=$(EMACS) -batch -Q -q -eval                             \
 "(progn (add-to-list (quote load-path) (expand-file-name \"./lisp/\")) \
        (add-to-list (quote load-path) \"$(lispdir)\"))"


HTH,
r

Re: How you can help

[Avdi Grimm]

I've been considering buckling down and doing some more ELisp coding,
which would mean org-mode coding because I've become ridiculously
dependent on org-mode.  If I do, I would only do it in a test-first
way, because that's the only way I'll write code anymore; so it would
definitely be good to know that unit tests will be be accepted into
the codebase.  Personally, if I did this I think I would like to use
Emacs Lisp Expectations:

http://www.emacswiki.org/cgi-bin/emacs/EmacsLispExpectations

I'm a big fan of Jay Field's work, I like the way the original
Expectations for Ruby is structured, and I like the looks of the EL
Expectations API.  I'm open to other libraries though - so long as
SOME unit-testing standard it accepted.

--
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: How you can help

[Bernt Hansen]

Sebastian Rose <sebastian_rose@gmx.de> writes:

> Don't know how often the git repo is published.
> Bernt and Ben, are you 'worgers' allready?

I think it's published hourly (maybe) and yes I have access to Worg.

-Bernt

Re: How you can help

[Sebastian Rose]

Ben,

I added this to the new page on worg. Please tell me: is it OK for you,
if I simply add your ideas there??

What do the others think?


Ben Alexander <bva@alexanderonline.org> writes:
> The tutorial I'd need to write a test is one which lays out code I could copy
> and paste to do the following
>
> * setup the test environment
>   - create a test directory
>   - create a sample test.org file
>   - put the cursor in a particular place
>
> * run the command we need to test
>   - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to
> find out exactly what command is actually being run  when you hit a keystroke.
> And some of me might need to be told what  lisp-code to use when the keystroke
> runs different commands at  different places in a file)
>   - reformat a table
>   - clock in/out
>   - create the agenda
>   - export .html .ics .dvi file
> * How do we specify the correct result???
>   - check that the headline folded properly.  What's the lisp code for getting
> the folded string as displayed?
>   - check that the cursor is where it should be? especially when the cursor is
> near elipses...
>   - check that the agenda is built properly.  What's the lisp code for getting
> the agenda as a string?
>   - check that the exported files are correct.  Maybe the right suggestion is to
> run the export on two different files, so the test  can focus on the 'diff'
> between them.  That way different people who  run the same test on different
> hosts can get the same result.


Please, if you find time, add the rest somewhere there?
If not, I'll take a closer look tonight and check in what's interesting
(which seems to be all of it, actually).


http://www.emacswiki.org/emacs-fr/UnitTesting:

Does this mean to add code to Org's *.el files? This would mean a lot of
work for Carsten :-( OK, once done... 

But:
sh> grep -r defun lisp/ | wc -l
1290

Checking in/out of every function makes 2580 places to check. I can't
estimate the effort, actually, because I never did Unit testing. I guess
this number much higher than needed?

Could anyone with testing expierience estimate?



While I think automated tests are a great thing to do and I'm happy to
see this reaction on the list :-), I still think we should also provide
the 'simple' tests without additional dependencies (when will package
management make it into emacs...). And I'd like to keep the effort and
impact for Carsten as small as possible. He will have to check each and
every patch because of feeling, and actually being, responsible for the
code.

Another question: will the code compile in emacs without the testing
framework installed? Probably yes.

Should we try to think of those 'simple test packages' as the more
_DATA_ part of other tests? This would mean one could code an automated
test without the need to create data too, since we have the data
seperated. Again, I'd like to keep the hurdles low for everyone
involved. When automated testing turns out to be a foolish easy and rock
solid, this could change.


* Three basic approaches:

  1 Simple data and setup packages.
  2 Code, that executes org commands and test for correct output,
    no changes in Org's code needed.
  3 Test framework with changes to Org's code.

I'd promote a combination from 1 and 2, and see, how far it goes.




DATA
  +--- d1
  +--- d2
  +--- d3

TESTS
  +---t-a
  +---t-b
  +---t-c

Now one would run test t-a over d1, d2, d3.
Still a programmer could use d1 to fix a bug in d1 and finally push the
fix for automated testing against d2 and d3.


bug X
tester verifies (is ist bug? how to reproduce...)
tester adds X to dX (message -> list or programmer)
programmer installs dX
programmer fixes bugX
programmer push   (message -> list or tester)
tester tests against d1, d2, d3

where tester = [human|machine] preferable human (simplicity)


Should we track the dependencies between code and tests?

Suppose one has worked on a.el.
Shure more than one test has to run.
Example: org-publish.el contains code common to LaTeX-export,
HTML-export... 



Regards,

-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http:  www.emma-stil.de

Re: How you can help

[Avdi Grimm]

A few points, from someone with a decent amount of testing background:

* As someone who has contributed to OS projects, the lack of a
pre-existing set of regression tests in org-mode is actually *the*
most significant blocker to my getting involved in org-mode
development.  I do all of my coding - both professional and personal -
in the context of tests, and not having the existing framework means
that before I can start working on new features I first need to spend
time yak-shaving on testing infrastructure.

*  I think there's a lot of over-thinking going on here.  Here's the
test-first coding discipline in a nutshell:
   1. Identify a problem/missing feature.
   2. Write a test (possibly using a unit-testing framework to help)
which will pass when the bug has been fixed or feature added. This can
be as simple as calling a function and validating its return value.
   3.  Run the test.  Verify it FAILS.
   4. Write code to make the test PASS.
   5. Refactor, if you introduced any code duplication in step 4.
   6. Run all the tests, to make sure you didn't break anything else.
   7. Commit.

If someone would be so kind as to identify a small bug or feature, I
would be happy to demonstrate this workflow in the form of code, time
permitting.

-- 
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: How you can help

[Sebastian Rose]

(cc'ing the list again)


Ben Alexander <bva@alexanderonline.org> writes:
> On 2008-Oct-23, at 17:26, Sebastian Rose wrote:
>
>> Ben,
>>
>> I added this to the new page on worg. Please tell me: is it OK for you,
>> if I simply add your ideas there??
>>
> Sure put what you want there.

Great. 

> FWIW, I'm brand new (today!) to git and I've cloned Worg today too.  But I not a
> worger' -- I'm not sure I've read much of it online.

So I'll look what else should be added? Hm - this thread is so busy,
that I'd say I take a few hours on weekend to sum it up.

>> What do the others think?
>>
>>
>> Ben Alexander <bva@alexanderonline.org> writes:
>>> way too much
>>>
>>
>>
>> Please, if you find time, add the rest somewhere there?
>> If not, I'll take a closer look tonight and check in what's interesting
>> (which seems to be all of it, actually).
>>
>>
>> http://www.emacswiki.org/emacs-fr/UnitTesting:
>>
>> Does this mean to add code to Org's *.el files? This would mean a lot of
>> work for Carsten :-( OK, once done...
>>
>
> Uh, I hope that's not necessary.  I'd hope that the tests could be written
> completely independently of any org-code.  If the basic idea  is to launch emacs
> in batch mode, then the test could redefine any  function locally, in the test
> code, instead of screwing around with  what Carsten has.
>
> If every test is written this way, it'll be very slow to run all the tests.
> Maybe that doesn't matter for bug-report tests?  But here's  where I should shut
> up and listen instead of talk.


Great! I hope so too!!

If slow, it's still no problem to run all those tests before a release
or once a day, whatever.

Glad to here your interested in keeping impact on Carsten low!


Cheers,


-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http:  www.emma-stil.de

Re: How you can help

[Sebastian Rose]

"Avdi Grimm" <avdi@avdi.org> writes:
> A few points, from someone with a decent amount of testing background:

Jippie! Please stay with us for a few days :-D

> * As someone who has contributed to OS projects, the lack of a
> pre-existing set of regression tests in org-mode is actually *the*
> most significant blocker to my getting involved in org-mode
> development.  I do all of my coding - both professional and personal -
> in the context of tests, and not having the existing framework means
> that before I can start working on new features I first need to spend
> time yak-shaving on testing infrastructure.
>
> *  I think there's a lot of over-thinking going on here.  Here's the
> test-first coding discipline in a nutshell:
>    1. Identify a problem/missing feature.
>    2. Write a test (possibly using a unit-testing framework to help)
> which will pass when the bug has been fixed or feature added. This can
> be as simple as calling a function and validating its return value.
>    3.  Run the test.  Verify it FAILS.
>    4. Write code to make the test PASS.
>    5. Refactor, if you introduced any code duplication in step 4.
>    6. Run all the tests, to make sure you didn't break anything else.
>    7. Commit.
>
> If someone would be so kind as to identify a small bug or feature, I
> would be happy to demonstrate this workflow in the form of code, time
> permitting.

Sounds great! Not shure if we can provide a bug though :-D


Bug anyone?



Hm - how about:

1. A not yet existent elisp file test-worg.el, that defines a function
hello-worg, and a variable lang, and simply puts "Hello Worg"into the
minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang
"de")) ?



Did you work with unit-testing frameworks for elisp already?
Which one?
Recommendations?
Could say something about the effort to get started with such a
framework?

Can we add it without changing Org's code? If understand 2. correctly - yes?

If we can, would we loose quality/speed of tests?
How much?


Tell us all :-)


Oha - I have an appointment (concert) - back tonight!


All the best,

-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: s.rose@emma-stil.de, sebastian_rose@gmx.de
Http:  www.emma-stil.de

Re: How you can help

[Avdi Grimm]

On Thu, Oct 23, 2008 at 1:33 PM, Sebastian Rose <sebastian_rose@gmx.de> wrote:
> Hm - how about:
>
> 1. A not yet existent elisp file test-worg.el, that defines a function
> hello-worg, and a variable lang, and simply puts "Hello Worg"into the
> minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang
> "de")) ?

I'd prefer something that would be of concrete value to someone in
using org-mode - made-up scenarios tend to lead to unrealistic tests.

> Did you work with unit-testing frameworks for elisp already?
> Which one?

I have not.  I've done test-driven development in Java, C#, Ruby, and
C++(!) though, and if I can do it in C++ I think I can get up to speed
in any language pretty quickly ;-)

> Recommendations?

As I said in another comment, I'm liking the look of Emacs Lisp
Expectations a *lot*.  Partly because I think Jay Fields is bang on
the money for how maintain good habits in your tests, and partly
because it has a mocking framework to go with it.  Testing without a
mocking framework is less fun than testing with.  However, I haven't
actually used it yet, so I can't responsibly recommend it.  Let's just
say that the discovery of it recently was the thing that got me
excited about doing some serious ELisp development.

> Could say something about the effort to get started with such a
> framework?

I think if you have some *running* examples to crib from it shouldn't
take someone who is already ELisp-savvy more than a few hours to get
the hang of it.

>
> Can we add it without changing Org's code? If understand 2. correctly - yes?

In a dynamic language like ELisp, any testing framework that requires
you to change the code just to get it under test isn't worth using.
No, you should not have to change org-mode's code to get started with
tests.

That said, developing in concert with a test suite will definitely
tend to influence how you write the code.  Generally speaking,
developing with tests encourages smaller functions which do one thing
and one thing alone and don't have a lot of non-obvious inputs and
outputs.  Since this is usually the direction you *want* your code to
go in anyway, it's a virtuous cycle.

-- 
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: How you can help

[Ben Alexander]

On 2008-Oct-23, at 17:42, Avdi Grimm wrote:

>
> If someone would be so kind as to identify a small bug or feature, I
> would be happy to demonstrate this workflow in the form of code, time
> permitting.
>
>

Ok, here's your chance.  This is something that has bothered me for  
quite some time, but I've never been able to reliably reproduce the  
problem.  And it's such a small issue.

Sometimes, when it looks like my cursor is at the end of a headline  
and I hit tab, the headline does not cycle.  Normally it does.

It hit me that the point was not on the headline, it only looked like  
it.

So here's the sample test.org file

* First Headline
    Some body text
* Second Headline

(goto-char 17)  ;that's the end of the first line
(org-cycle)

At this point the buffer looks like:
* First Headline...
* Second Headline

(goto-char 25)
(org-cycle)

I think the buffer should look like this

* First Headline
    Some body text
* Second Headline

but it still looks like:

* First Headline...
* Second Headline


* ABOUT THIS BEHAVIOR
I should be honest here.  This may not be a bug.

As I created this test data, it became a bit more clear that exactly  
where the cursor is matters a lot here.  I *promise* that I've seen  
this behavior even when the cursor shows up before the ellipses, but  
in the test case I set up here, I could not make that happen.  I also  
found that if I used the key-chord M-x goto-char <RETURN> 25,  
everything worked fine.  The cursor stayed in front of the ellipses  
and the tab key worked. But when I used M-: (goto-char 25) the cursor  
moved to the same line as the ellipses, but after them. and the <TAB>  
key stopped 'working'.

I finally figured this out while playing with git.  I switched  
branches at the command line and needed to perform a 'revert-buffer'.   
This left the cursor before the ellipses, but unable to properly (org- 
cycle) using the <TAB> key.

More experimenting.... I *think* the revert-buffer command tries to  
keep point close to the same place, and the org buffer had automatic  
folding.  The bad thing about this for me is that I'll hit the <TAB>  
key four times trying to make sure it's not just an empty tree.   
Meanwhile, (org-cycle) has indented the first line of the body, but  
hasn't shown me the text I'm changing.  Then I get confused about why  
the buffer needs saving, probably 10 minutes later when I've forgotten  
all about hitting the tab key.

But I'm way out of my depth to try to understand the relationship  
between (revert-buffer) (org-cycle) the arrow keys, and all.  I just  
hit M-< or S-<TAB> and try again. Or C-a, which sometimes works.   
Sometimes I grab the mouse as a last resort.

* ABOUT TESTING ISSUES
Just from this example, I'm eager to understand:

1. How can we differentiate between where the cursor appears and the  
value of point
2. How can we tell what the users sees on the screen versus what the  
buffer contains

So the value of a testing framework is that this: if I'm going to  
announce to the world that I can't get something to work, (like the  
tab key), I'm going to make darn sure it's a real problem.  I'm going  
to spend the time to create a small sample file, that reliable creates  
the problem, and I'm going to try a few different scenarios.   But if  
I start to get lost or confused about which settings I've fiddled with  
or what is supposed to happen, I'll wander away from the problem and I  
won't submit the bug.  We all lose in that scenario.  But if the  
testing framework exists, and it is lightweight enough for novice  
emacs users (advanced enough to use M-x but not advanced enough to  
read lisp) then maybe I would have used it for this example.


-Ben

Re: How you can help

[Avdi Grimm]

On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org> wrote:
> Ok, here's your chance.  This is something that has bothered me for quite
> some time, but I've never been able to reliably reproduce the problem.  And
> it's such a small issue.

Thanks!  I'll try to take a look tonight or this weekend.

> So the value of a testing framework is that this: if I'm going to announce
> to the world that I can't get something to work, (like the tab key), I'm
> going to make darn sure it's a real problem.  I'm going to spend the time to
> create a small sample file, that reliable creates the problem, and I'm going
> to try a few different scenarios.   But if I start to get lost or confused
> about which settings I've fiddled with or what is supposed to happen, I'll
> wander away from the problem and I won't submit the bug.  We all lose in
> that scenario.  But if the testing framework exists, and it is lightweight
> enough for novice emacs users (advanced enough to use M-x but not advanced
> enough to read lisp) then maybe I would have used it for this example.

For the record, I normally would never expect the user to submit
tests.  A description of the problem and some sample data is all I
expect.  Writing the a test to reproduce the behavior is the
programmer's responsibility as far as I'm concerned.

-- 
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: How you can help

[Jeff Mickey]

On Fri, Oct 24, 2008 at 14:44, Avdi Grimm <avdi@avdi.org> wrote:
> For the record, I normally would never expect the user to submit
> tests.  A description of the problem and some sample data is all I
> expect.  Writing the a test to reproduce the behavior is the
> programmer's responsibility as far as I'm concerned.

That's the best part of open source software!  A surprising amount of
the time, the user IS a programmer, and thus is able to provide tests.

And it's not that we should require anyone to provide tests before
they report a bug, but they should be required to outline the steps to
reproduce the behavior, so any programmer could take 5 minutes and
quickly whip up a test.

ps: I almost have a baby ert framework for org-mode.  It doesn't do
anything yet, but the design is you write a test, and add it to the
org-test-hook to be run by org-test-all.
-- 
. : [ + carpe diem totus tuus + ] : .

Re: How you can help

[Tom Breton (Tehom)]

> "Avdi Grimm" <avdi@avdi.org> writes:
> > A few points, from someone with a decent amount of testing background:
>
> Jippie! Please stay with us for a few days :-D

[...]

As someone with also a decent amount of testing background, I'd like
to answer also.

> Did you work with unit-testing frameworks for elisp already?
> Which one?

I co-wrote regress.el (Wayne Mesard wrote the first version) and I
wrote rtest.  I've written some testing for allout.el before I
switched over to org-mode (which is very nice, BTW, Carsten, and
provides pretty much everything I wish-listed for allout mode).  Of
particular interest for testing org-mode, I wrote mockbuf.el, which mocks
files and buffers so that tests can deal with files without changing
anything permanently.

> Recommendations?

Hard for me to say objectively.

I prefer my rtest but it's in flux right now.  I wouldn't recommend
the current public version, rtest 3.0  I've designed and half written a
better version but I can't really ask you to wait until I get around
to making a better version of rtest.  There is a development version
that works and does nice things; it's messy as hell right now and I'm
embarrassed about that.  It isn't public but I could put it up if
you'll forgive its messy state,

It has mockbuf, which I found indispensable in writing tests for
allout.el.  In fact, I mostly developed it for that, though all the
allout-specific code is elsewhere.  Also useful for writing tests:
safe-equal.el, caselet.el, and select.cl, all in the ancillary
directory.

I would steer clear of frameworks that provide many of synonyms for
equality tests, inequality tests, etc.  You already know how to write
(not (equal ...)) in elisp.  Why learn the name and syntax of
something like (shouldnt-be-equal A B)?  Having a lot of that makes a
package LOOK heftier but doesn't do anything more for you.  IMO
analyzing test failures should be done by inspecting the test form,
not by making you write things in non-standard elisp that could be
written in standard elisp.

Using ert would mean requiring ewoc, which means tests wouldn't work
on emacs 21.  Too bad, because it has some appeal.



> Could say something about the effort to get started with such a
> framework?

For rtest, right now you just need (a) the development version, (b)
unzip it in your emacs-site-lisp or set the paths and autoloads
manually if you prefer, (c) include the following in a file to be
tested:

	(when (not (fboundp 'rtest:deftest))
	    (defmacro rtest:deftest (&rest dummy))
	    (defmacro rtest:if-avail (&rest dummy)))

Then write tests as:

	(rtest:deftest function-name
	   ("Docstring"
	      (test-form) ;;Return a boolean
	      )
	   ;;Repeat as wanted
	   )

That will get you started, but really you'd want to get familiar with
the stuff in mockbuf.el - which is so messy right now that I'm
embarrassed about it, because it contains both old and new versions of
many things in flux.

Other advice: Make a directory to hold short examples of files that
illustrate "before" and "after" states for the various
transformations.  Each unit (eg org-agenda.el, org-archive.el etc)
should have its own subtree of examples.

One thing I'd do early is build the following test helper functions:

A macro that runs a test body inside a "let" of all the configuration
options set to expected values.

A function to compare vector-style strings.  `equal' doesn't suffice.
rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and
safe-equal-compare-vectors, which IIRC suffice for this.

The ability to write only the visible parts to a separate buffer was
indispensable in testing allout's folding.  You already have it in
org-export-visible with the type that keeps the buffer, "?\ ".  Then
you can just compare contents.

You will likely have a lot of what I call "output tests", tests of
code that has the property that if the output looks right, it is
right.

IMO with output tests, you need to make it easy on yourself.  TDD is
all well and fine, but not if it makes you repeat yourself.  If there
is available output that looks right, capture it and use it as a base
for comparison.  That's not "pure" but it's far easier and it works
right.  If the available output isn't yet right, capture it anyways
and tweak it to make it right - but this rarely happens.

I automated that sort of thing for a Prolog test harness I wrote,
works fine.  You'd just write a test of a special type that produced
and captured output.  It would originally be ungraded, but you would
then grade it, and it would remember the correct output and compare
against it.

I was planning to add the same to rtest and I partly did, but never
finished it.  Still on my todo list.

> Can we add it without changing Org's code? If understand 2. correctly -
yes?
>

rtest's tests like to be in the same file as the code.  If you use the
form above, the file still compiles and runs even when rtest is not
available.

It's possible to write the tests in another file, but then either
tests aren't grouped together (must be run one by one) or you have to
do extra work to group them.  If they're in file, you can run every
test in a package with `rtest:library'.  The other really convenient
way to run a test is rtest:defun-at-point.


> If we can, would we loose quality/speed of tests?

Not at all.


Tom Breton (Tehom)

PS: One more piece of advice: Make an auto-insert or at least a
skeleton that inserts test headers, like the "(when (not (fboundp ..."
stuff above.  You'll be using it a fair bit, and it's real nice to
have it just automatically be there when you start a new file.  This
advice is geared toward my way of working, though, with the test code
in the same file as the functional code.