emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* Testing --- again...
@ 2010-10-02  3:51 Sebastian Rose
  2010-10-02  8:09 ` Carsten Dominik
  0 siblings, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-02  3:51 UTC (permalink / raw)
  To: Emacs-orgmode mailing list; +Cc: carsten.dominik

Hi,


I thought about testing again recently.  This is something, that never
really got started.  For a reason:  there's no framework for testing.

I therefore wrote a very rough proposal,  found  on
http://github.com/SebastianRose/org-test

The idea is, to provide two simple commands:


  *  org-test-test-current-defun
     will search for tests for the defun point is in or behind
     (`beginning-of-defun') and execute them surrounded by

  (let ((select (or selector "^org"))
   	(deactivate-mark nil))
    (save-excursion
      (save-match-data


  *  org-test-test-buffer-file
     will search for tests for the entire file and execute them the same
     way.

If you use one of these commands, all currently registered ERT tests are
deleted, and files are reloaded (since you're likely to work on the
tests, too).  To repeat the tests without reloading, you will use the
ERT commands like `ert-results-rerun-all-tests', bound to `r' in the ERT
results buffer.



I choose ERT (git clone http://github.com/ohler/ert.git) because that's
likely to go into Emacs core (or elpa.gnu.org).




The idea is to search the directory structure from the current source
file upwards for a directory named "tests/" if it exists.  Else ask the
user.  Similar to what `add-change-log-entry' does.

Below that directory, a tree like the source tree exists:

project
   +-- lisp/
   |     +-- a.el
   |     `-- b/
   |         +-- b.el
   |
   `-- tests/
         +-- a.el/
         |     +-- tests.el
         |     `-- a-defun.el
         `-- b/
             +-- b.el/
                   +-- tests.el
                   `-- b-defun.el

If this setup exists, when editing defun-x in lisp/a.el,
`M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
(fallback: tests.el there) and execute all tests with selector
"^a-defun".
`M-x org-test-test-buffer-file' in that same source file will load all
*.el files in tests/a.el/ and execute all ERT tests for selector "^a".


Thus tests for
    org-mode/lisp/org-protocol.el
will be searched in the directory
    org-mode/tests/lisp/org-protocol.el/*.el


Once the basic route of testing is clear, I'd like to "translate" the
existing tests for org-html.el to work with ERT, which will involve
writing more tools (create output buffers, compare output with control
files using ediff etc.).  I know Lennart Borgman has wrote that stuff
for nXhtml already.  I hope we can use his stuff and help here.

The directory org-mode/lisp/tests/ would not need to be part of the
"official" Org mode package.  It could as well be checked out
separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).


Any thoughts?




  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-02  3:51 Testing --- again Sebastian Rose
@ 2010-10-02  8:09 ` Carsten Dominik
  2010-10-02 14:53   ` Sebastian Rose
                     ` (2 more replies)
  0 siblings, 3 replies; 18+ messages in thread
From: Carsten Dominik @ 2010-10-02  8:09 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list

Hi Sebastian,

the lack of a testing suite for Org-mode is really frustrating,
and even more frustrating is that we have had like seven attempts
to start one, and each of these lead to nothing.  So I would
be perfectly happy to give a free hand, write access to the repo
and a full directory in the distribution to implement one.
Once there is a framework, I am sure many people would be
willing to contribute tests.

More comments below.

On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:

> Hi,
>
>
> I thought about testing again recently.  This is something, that never
> really got started.  For a reason:  there's no framework for testing.
>
> I therefore wrote a very rough proposal,  found  on
> http://github.com/SebastianRose/org-test
>
> The idea is, to provide two simple commands:
>
>
>  *  org-test-test-current-defun
>     will search for tests for the defun point is in or behind
>     (`beginning-of-defun') and execute them surrounded by
>
>  (let ((select (or selector "^org"))
>   	(deactivate-mark nil))
>    (save-excursion
>      (save-match-data
>
>
>  *  org-test-test-buffer-file
>     will search for tests for the entire file and execute them the  
> same
>     way.

FIrst:  I have *no* clue about testing.

Second, I am surprised that you want to structure it by function.  I  
would have
thought that it could be structure by file at the most.  And then  
there will
be tests that involve code from many files.

But I guess

>
> If you use one of these commands, all currently registered ERT tests  
> are
> deleted, and files are reloaded (since you're likely to work on the
> tests, too).  To repeat the tests without reloading, you will use the
> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the  
> ERT
> results buffer.
>
>
>
> I choose ERT (git clone http://github.com/ohler/ert.git) because  
> that's
> likely to go into Emacs core (or elpa.gnu.org).
>
>
>
>
> The idea is to search the directory structure from the current source
> file upwards for a directory named "tests/" if it exists.  Else ask  
> the
> user.  Similar to what `add-change-log-entry' does.
>
> Below that directory, a tree like the source tree exists:
>
> project
>   +-- lisp/
>   |     +-- a.el
>   |     `-- b/
>   |         +-- b.el
>   |
>   `-- tests/
>         +-- a.el/
>         |     +-- tests.el
>         |     `-- a-defun.el
>         `-- b/
>             +-- b.el/
>                   +-- tests.el
>                   `-- b-defun.el
>
> If this setup exists, when editing defun-x in lisp/a.el,
> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
> (fallback: tests.el there) and execute all tests with selector
> "^a-defun".

Well, OK, this is fine.  But under a.el and b.el there should also be
general tests that are not function dependent, and there should be a  
place
to put tests that you do not want to assign to a specific file.

We do have a "testing" directory already, you can use that.
I would prefer the tests to be in testing, not in lisp/testing
if possible. I would like to have the lisp directory contain
only code.  If possible.

It would be OK to have a lisp subdirectory in testing,
just as it would be OK to have contrib/lisp in testing
for the contributed packages.

PLEASE, go ahead.  I do not think you have write access
yet on repo - give me your user name and I'll activate you.

- Carsten

> `M-x org-test-test-buffer-file' in that same source file will load all
> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>
>
> Thus tests for
>    org-mode/lisp/org-protocol.el
> will be searched in the directory
>    org-mode/tests/lisp/org-protocol.el/*.el
>
>
> Once the basic route of testing is clear, I'd like to "translate" the
> existing tests for org-html.el to work with ERT, which will involve
> writing more tools (create output buffers, compare output with control
> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
> for nXhtml already.  I hope we can use his stuff and help here.
>
> The directory org-mode/lisp/tests/ would not need to be part of the
> "official" Org mode package.  It could as well be checked out
> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>
>
> Any thoughts?
>
>
>
>
>  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-02  8:09 ` Carsten Dominik
@ 2010-10-02 14:53   ` Sebastian Rose
  2010-10-02 15:01   ` Sebastian Rose
  2010-10-02 18:12   ` Eric Schulte
  2 siblings, 0 replies; 18+ messages in thread
From: Sebastian Rose @ 2010-10-02 14:53 UTC (permalink / raw)
  To: Carsten Dominik; +Cc: Emacs-orgmode mailing list

Carsten Dominik <carsten.dominik@gmail.com> writes:
> Hi Sebastian,
>
> the lack of a testing suite for Org-mode is really frustrating,
> and even more frustrating is that we have had like seven attempts
> to start one, and each of these lead to nothing.  So I would
> be perfectly happy to give a free hand, write access to the repo
> and a full directory in the distribution to implement one.
> Once there is a framework, I am sure many people would be
> willing to contribute tests.


There was no support for testing in Emacs and I could not see something
coming up.  I guess that was one of the main reasons, that nothing
happened.  The existing tests use a undocumented framework wich consists
of many source files itself, so that I was not able to get my head
around that in half an hour (which is by far too long for most of us).

I asked on emacs-devel and Stefan Monier told me that there was some
consensus that ERT _should_ go into Emacs (24?) or at least to
elpa.gnu.org.  Christian Ohlert already sprang into action, so I'm
optimistic that ERT is the right choice on the long run.


So.  The framework will be ERT.  I'll just right a few commands and
functions that make it _dead simple to right and execute tests_.


>
> More comments below.
>
> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>
>> Hi,
>>
>>
>> I thought about testing again recently.  This is something, that never
>> really got started.  For a reason:  there's no framework for testing.
>>
>> I therefore wrote a very rough proposal,  found  on
>> http://github.com/SebastianRose/org-test
>>
>> The idea is, to provide two simple commands:
>>
>>
>>  *  org-test-test-current-defun
>>     will search for tests for the defun point is in or behind
>>     (`beginning-of-defun') and execute them surrounded by
>>
>>  (let ((select (or selector "^org"))
>>   	(deactivate-mark nil))
>>    (save-excursion
>>      (save-match-data
>>
>>
>>  *  org-test-test-buffer-file
>>     will search for tests for the entire file and execute them the same
>>     way.
>
> FIrst:  I have *no* clue about testing.
>
> Second, I am surprised that you want to structure it by function.  I would have
> thought that it could be structure by file at the most.  And then there will
> be tests that involve code from many files.


It's both actually (see below).  But a simple function could easily have
5 or more invariants.  For each invariant a test should be written.
That makes 5 tests.  Having those (_optional_) per function test files
will help finding certain test easily.  Some functions will also need
lot's of other test code, e.g.  let bindings that emulate different
setups or code that creates temporary directory structures (publishing)
and the like.

Naming (again: optionally) files like the functions they are written for
and putting them into a directory named after the code file makes things
easy to find --- even without docs.


>> ...

>> The idea is to search the directory structure from the current source
>> file upwards for a directory named "tests/" if it exists.  Else ask the
>> user.  Similar to what `add-change-log-entry' does.
>>
>> Below that directory, a tree like the source tree exists:
>>
>> project
>>   +-- lisp/
>>   |     +-- a.el
>>   |     `-- b/
>>   |         +-- b.el
>>   |
>>   `-- tests/
>>         +-- a.el/
>>         |     +-- tests.el
>>         |     `-- a-defun.el
>>         `-- b/
>>             +-- b.el/
>>                   +-- tests.el
>>                   `-- b-defun.el
>>
>> If this setup exists, when editing defun-x in lisp/a.el,
>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>> (fallback: tests.el there) and execute all tests with selector
>> "^a-defun".
>
> Well, OK, this is fine.  But under a.el and b.el there should also be
> general tests that are not function dependent, and there should be a place
> to put tests that you do not want to assign to a specific file.


That's what tests.el is for currently (besides that it is the fallback
for function tests).

`org-test-test-buffer-file' simply loads _all_ *.el files in a file's
test directory.  With prefix argument, it loads tests.el only.



> We do have a "testing" directory already, you can use that.
> I would prefer the tests to be in testing, not in lisp/testing
> if possible. I would like to have the lisp directory contain
> only code.  If possible.


That's already possible.  The directory "tests" (will rename that to
"testing") is searched in the same directory as `buffer-file-name' and
from there up the directory tree.  It could be place entirely outside
the project tree.

> It would be OK to have a lisp subdirectory in testing,
> just as it would be OK to have contrib/lisp in testing
> for the contributed packages.

That's what it has to be like (it is).  We couldn't use the "autmatic"
test search otherwise.  E.g. emacs/lisp/ has many sub-directories each
of which could hold files with the same basename (in theory).


The above graph was wrong, sorry.  This is the current structure:

...

 `-- tests/
       +-- lisp/
             +-- a.el/
             |     +-- tests.el
             |     `-- a-defun.el
...



It's supposed to work for many projects, too. See how the directory
trees are rebuild below "testing/":

projects
  +-- emacs/
  |     `-- lisp/
  |          `-- org/
  +-- some-project/
  |         `-- code/
  |               `-- a.el
  |
  +-- testing
        +-- org-test.el
        `-- tests/
              +-- emacs/
              |     `-- lisp/
              |          `-- org/
              `-- some-project/
                        `-- code/
                              `-- a.el/
                                    +-- a-defun.el
                                    `-- tests.el

This kind of structure is the only way to prevent file name conflicts
and still act completely automatic.

It will _not_ be possible to have testing/  in a subdirectory
org-mode/lisp/.  But who would want that?   ;)


> PLEASE, go ahead.  I do not think you have write access
> yet on repo - give me your user name and I'll activate you.
>
> - Carsten

I'd be glad to care for the testing.  Albeit I'm not sure if I will do a
"good" job there.  But I see the urge to get that started, and that I
can do.  And, as always, I would enjoy helping where possible ;)

This would involve to scan the bug reports for regressions and right
tests accordingly.

My user name on repo.or.cz  is "SebastianRose".



Here a short (short is good) list of what the "testing" directory should
contain (please edit):

  - A subdirectory "tests" where all the tests go.
  - Some concise docs to get hackers started with testing.
  - The file org-test.el (since it's only needed if "testing/" exists).


What we should do, is to strip the org-mode/testing/ directory from
release packages.

Ahhh! "makerelease" already cares for this :)




  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-02  8:09 ` Carsten Dominik
  2010-10-02 14:53   ` Sebastian Rose
@ 2010-10-02 15:01   ` Sebastian Rose
  2010-10-02 17:05     ` Carsten Dominik
  2010-10-02 18:12   ` Eric Schulte
  2 siblings, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-02 15:01 UTC (permalink / raw)
  To: Carsten Dominik; +Cc: Emacs-orgmode mailing list

Carsten Dominik <carsten.dominik@gmail.com> writes:
> It would be OK to have a lisp subdirectory in testing,
> just as it would be OK to have contrib/lisp in testing
> for the contributed packages.

That's what it is like.

You probably did not notice, because "org-test.el" is supposed to live
inside "testing/", so it's not obvious.



   Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-02 15:01   ` Sebastian Rose
@ 2010-10-02 17:05     ` Carsten Dominik
  0 siblings, 0 replies; 18+ messages in thread
From: Carsten Dominik @ 2010-10-02 17:05 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list


On Oct 2, 2010, at 5:01 PM, Sebastian Rose wrote:

> Carsten Dominik <carsten.dominik@gmail.com> writes:
>> It would be OK to have a lisp subdirectory in testing,
>> just as it would be OK to have contrib/lisp in testing
>> for the contributed packages.
>
> That's what it is like.
>
> You probably did not notice, because "org-test.el" is supposed to live
> inside "testing/", so it's not obvious.

OK.

You can also rename testing to tests or whatever is convenient.

I have given you push access.

- Carsten

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02  8:09 ` Carsten Dominik
  2010-10-02 14:53   ` Sebastian Rose
  2010-10-02 15:01   ` Sebastian Rose
@ 2010-10-02 18:12   ` Eric Schulte
  2010-10-02 19:33     ` Sebastian Rose
  2 siblings, 1 reply; 18+ messages in thread
From: Eric Schulte @ 2010-10-02 18:12 UTC (permalink / raw)
  To: Carsten Dominik; +Cc: Emacs-orgmode mailing list

Hi,

This is exciting.

Rather than impose a complete directory/layout schema before-hand I'd
lean towards starting with a little more chaos and then letting the
structure of the test directory develop /naturally/.  From the
discussion below it sounds like an initial structure of

testing/lisp/
testing/contrib/lisp/

may make sense, reserving the top level for "meta" testing stuff, like
functions for running tests, common fixtures, example files, etc...

I have two questions.

1) while waiting for ert to be included into Emacs, should we include an
   ert distribution as part of the Org-mode repository (maybe using git
   sub-modules) or should we just agree that users should have a certain
   version of ert installed locally?  I'm honestly not sure which of
   these options sounds preferable.

2) should the initial population of the testing/ directory take place in
   a separate branch of the repository or in the master branch?  Again I
   don't know which I would prefer, branches add complication but could
   result in cleaner commit histories.

Best -- Eric

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Hi Sebastian,
>
> the lack of a testing suite for Org-mode is really frustrating,
> and even more frustrating is that we have had like seven attempts
> to start one, and each of these lead to nothing.  So I would
> be perfectly happy to give a free hand, write access to the repo
> and a full directory in the distribution to implement one.
> Once there is a framework, I am sure many people would be
> willing to contribute tests.
>
> More comments below.
>
> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>
>> Hi,
>>
>>
>> I thought about testing again recently.  This is something, that never
>> really got started.  For a reason:  there's no framework for testing.
>>
>> I therefore wrote a very rough proposal,  found  on
>> http://github.com/SebastianRose/org-test
>>
>> The idea is, to provide two simple commands:
>>
>>
>>  *  org-test-test-current-defun
>>     will search for tests for the defun point is in or behind
>>     (`beginning-of-defun') and execute them surrounded by
>>
>>  (let ((select (or selector "^org"))
>>   	(deactivate-mark nil))
>>    (save-excursion
>>      (save-match-data
>>
>>
>>  *  org-test-test-buffer-file
>>     will search for tests for the entire file and execute them the
>> same
>>     way.
>
> FIrst:  I have *no* clue about testing.
>
> Second, I am surprised that you want to structure it by function.  I
> would have
> thought that it could be structure by file at the most.  And then
> there will
> be tests that involve code from many files.
>
> But I guess
>
>>
>> If you use one of these commands, all currently registered ERT tests
>> are
>> deleted, and files are reloaded (since you're likely to work on the
>> tests, too).  To repeat the tests without reloading, you will use the
>> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the
>> ERT
>> results buffer.
>>
>>
>>
>> I choose ERT (git clone http://github.com/ohler/ert.git) because
>> that's
>> likely to go into Emacs core (or elpa.gnu.org).
>>
>>
>>
>>
>> The idea is to search the directory structure from the current source
>> file upwards for a directory named "tests/" if it exists.  Else ask
>> the
>> user.  Similar to what `add-change-log-entry' does.
>>
>> Below that directory, a tree like the source tree exists:
>>
>> project
>>   +-- lisp/
>>   |     +-- a.el
>>   |     `-- b/
>>   |         +-- b.el
>>   |
>>   `-- tests/
>>         +-- a.el/
>>         |     +-- tests.el
>>         |     `-- a-defun.el
>>         `-- b/
>>             +-- b.el/
>>                   +-- tests.el
>>                   `-- b-defun.el
>>
>> If this setup exists, when editing defun-x in lisp/a.el,
>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>> (fallback: tests.el there) and execute all tests with selector
>> "^a-defun".
>
> Well, OK, this is fine.  But under a.el and b.el there should also be
> general tests that are not function dependent, and there should be a
> place
> to put tests that you do not want to assign to a specific file.
>
> We do have a "testing" directory already, you can use that.
> I would prefer the tests to be in testing, not in lisp/testing
> if possible. I would like to have the lisp directory contain
> only code.  If possible.
>
> It would be OK to have a lisp subdirectory in testing,
> just as it would be OK to have contrib/lisp in testing
> for the contributed packages.
>
> PLEASE, go ahead.  I do not think you have write access
> yet on repo - give me your user name and I'll activate you.
>
> - Carsten
>
>> `M-x org-test-test-buffer-file' in that same source file will load all
>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>
>>
>> Thus tests for
>>    org-mode/lisp/org-protocol.el
>> will be searched in the directory
>>    org-mode/tests/lisp/org-protocol.el/*.el
>>
>>
>> Once the basic route of testing is clear, I'd like to "translate" the
>> existing tests for org-html.el to work with ERT, which will involve
>> writing more tools (create output buffers, compare output with control
>> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
>> for nXhtml already.  I hope we can use his stuff and help here.
>>
>> The directory org-mode/lisp/tests/ would not need to be part of the
>> "official" Org mode package.  It could as well be checked out
>> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>>
>>
>> Any thoughts?
>>
>>
>>
>>
>>  Sebastian
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02 18:12   ` Eric Schulte
@ 2010-10-02 19:33     ` Sebastian Rose
  2010-10-02 20:30       ` Eric Schulte
  0 siblings, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-02 19:33 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:
> Hi,
>
> This is exciting.
>
> Rather than impose a complete directory/layout schema before-hand I'd
> lean towards starting with a little more chaos and then letting the
> structure of the test directory develop /naturally/.  From the
> discussion below it sounds like an initial structure of
>
> testing/lisp/
> testing/contrib/lisp/


I believe it makes sense enforcing rules.  Many developers plus power
users will want to be able to use the test system.  No system is what we
had in the past.

The idea is, to have a system to automate the laoding of tests.  How
should a function like `org-test-test-current-defun' find the tests
otherwise?

  sh$ cd org-mode
  sh$ find . -name '*.el' | wc -l
  146

Also, we could provide services like setup temporary directories,
buffers and files for tests.  This cannot be automated in a save way, if
there is no structure.  The tests are written in elisp.  Hence one could
do whatever he likes ;)

It's like Perl.  You don't need to follow the conventions, but it will
make your live easier (hopefully).

Just what I think.


> may make sense, reserving the top level for "meta" testing stuff, like
> functions for running tests, common fixtures, example files, etc...
>
> I have two questions.
>
> 1) while waiting for ert to be included into Emacs, should we include an
>    ert distribution as part of the Org-mode repository (maybe using git
>    sub-modules) or should we just agree that users should have a certain
>    version of ert installed locally?  I'm honestly not sure which of
>    these options sounds preferable.


I thought about this, too.  I guess not.  Developers and users that want
to test will be able to follow the current ERT git repo.

But ERT is just 7 *.el files plus 1 texinfo file.

An what I don't know is:

   How would git submodules work?




> 2) should the initial population of the testing/ directory take place in
>    a separate branch of the repository or in the master branch?  Again I
>    don't know which I would prefer, branches add complication but could
>    result in cleaner commit histories.


I'll start on a branch first and constantly rebase as long as the
structure evolves.  The first simple commit will be what you can see on
github, with some doc strings adjusted.

  

   Sebastian

  

> Best -- Eric
>
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> Hi Sebastian,
>>
>> the lack of a testing suite for Org-mode is really frustrating,
>> and even more frustrating is that we have had like seven attempts
>> to start one, and each of these lead to nothing.  So I would
>> be perfectly happy to give a free hand, write access to the repo
>> and a full directory in the distribution to implement one.
>> Once there is a framework, I am sure many people would be
>> willing to contribute tests.
>>
>> More comments below.
>>
>> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>>
>>> Hi,
>>>
>>>
>>> I thought about testing again recently.  This is something, that never
>>> really got started.  For a reason:  there's no framework for testing.
>>>
>>> I therefore wrote a very rough proposal,  found  on
>>> http://github.com/SebastianRose/org-test
>>>
>>> The idea is, to provide two simple commands:
>>>
>>>
>>>  *  org-test-test-current-defun
>>>     will search for tests for the defun point is in or behind
>>>     (`beginning-of-defun') and execute them surrounded by
>>>
>>>  (let ((select (or selector "^org"))
>>>   	(deactivate-mark nil))
>>>    (save-excursion
>>>      (save-match-data
>>>
>>>
>>>  *  org-test-test-buffer-file
>>>     will search for tests for the entire file and execute them the
>>> same
>>>     way.
>>
>> FIrst:  I have *no* clue about testing.
>>
>> Second, I am surprised that you want to structure it by function.  I
>> would have
>> thought that it could be structure by file at the most.  And then
>> there will
>> be tests that involve code from many files.
>>
>> But I guess
>>
>>>
>>> If you use one of these commands, all currently registered ERT tests
>>> are
>>> deleted, and files are reloaded (since you're likely to work on the
>>> tests, too).  To repeat the tests without reloading, you will use the
>>> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the
>>> ERT
>>> results buffer.
>>>
>>>
>>>
>>> I choose ERT (git clone http://github.com/ohler/ert.git) because
>>> that's
>>> likely to go into Emacs core (or elpa.gnu.org).
>>>
>>>
>>>
>>>
>>> The idea is to search the directory structure from the current source
>>> file upwards for a directory named "tests/" if it exists.  Else ask
>>> the
>>> user.  Similar to what `add-change-log-entry' does.
>>>
>>> Below that directory, a tree like the source tree exists:
>>>
>>> project
>>>   +-- lisp/
>>>   |     +-- a.el
>>>   |     `-- b/
>>>   |         +-- b.el
>>>   |
>>>   `-- tests/
>>>         +-- a.el/
>>>         |     +-- tests.el
>>>         |     `-- a-defun.el
>>>         `-- b/
>>>             +-- b.el/
>>>                   +-- tests.el
>>>                   `-- b-defun.el
>>>
>>> If this setup exists, when editing defun-x in lisp/a.el,
>>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>>> (fallback: tests.el there) and execute all tests with selector
>>> "^a-defun".
>>
>> Well, OK, this is fine.  But under a.el and b.el there should also be
>> general tests that are not function dependent, and there should be a
>> place
>> to put tests that you do not want to assign to a specific file.
>>
>> We do have a "testing" directory already, you can use that.
>> I would prefer the tests to be in testing, not in lisp/testing
>> if possible. I would like to have the lisp directory contain
>> only code.  If possible.
>>
>> It would be OK to have a lisp subdirectory in testing,
>> just as it would be OK to have contrib/lisp in testing
>> for the contributed packages.
>>
>> PLEASE, go ahead.  I do not think you have write access
>> yet on repo - give me your user name and I'll activate you.
>>
>> - Carsten
>>
>>> `M-x org-test-test-buffer-file' in that same source file will load all
>>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>>
>>>
>>> Thus tests for
>>>    org-mode/lisp/org-protocol.el
>>> will be searched in the directory
>>>    org-mode/tests/lisp/org-protocol.el/*.el
>>>
>>>
>>> Once the basic route of testing is clear, I'd like to "translate" the
>>> existing tests for org-html.el to work with ERT, which will involve
>>> writing more tools (create output buffers, compare output with control
>>> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
>>> for nXhtml already.  I hope we can use his stuff and help here.
>>>
>>> The directory org-mode/lisp/tests/ would not need to be part of the
>>> "official" Org mode package.  It could as well be checked out
>>> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>>>
>>>
>>> Any thoughts?
>>>
>>>
>>>
>>>
>>>  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02 19:33     ` Sebastian Rose
@ 2010-10-02 20:30       ` Eric Schulte
  2010-10-02 21:55         ` Eric Schulte
  2010-10-02 22:14         ` Sebastian Rose
  0 siblings, 2 replies; 18+ messages in thread
From: Eric Schulte @ 2010-10-02 20:30 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list, Carsten Dominik

Hi Sebastian,

Sebastian Rose <sebastian_rose@gmx.de> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>> Hi,
>>
>> This is exciting.
>>
>> Rather than impose a complete directory/layout schema before-hand I'd
>> lean towards starting with a little more chaos and then letting the
>> structure of the test directory develop /naturally/.  From the
>> discussion below it sounds like an initial structure of
>>
>> testing/lisp/
>> testing/contrib/lisp/
>
>
> I believe it makes sense enforcing rules.  Many developers plus power
> users will want to be able to use the test system.  No system is what we
> had in the past.
>
> The idea is, to have a system to automate the laoding of tests.  How
> should a function like `org-test-test-current-defun' find the tests
> otherwise?
>
>   sh$ cd org-mode
>   sh$ find . -name '*.el' | wc -l
>   146
>
> Also, we could provide services like setup temporary directories,
> buffers and files for tests.  This cannot be automated in a save way, if
> there is no structure.  The tests are written in elisp.  Hence one could
> do whatever he likes ;)
>
> It's like Perl.  You don't need to follow the conventions, but it will
> make your live easier (hopefully).
>
> Just what I think.
>

Point well taken, I suppose my point is more a matter of personal taste,
and I fully understand if we disagree and would yield to your judgment
as you've put more thought into this.  But having said that here's my
thoughts and opinions :)

Could we just load every test in tests/lisp by default, and then use the
existing `ert' selection method to select and run tests.  For example if
we enforce our conventions on the level of test function name rather
than on file name, we could (I believe) do something like the following.
Say every test for a particular function (say org-export) includes that
name of that function in the test name (say test-org-export), then we
could use something simple like the following to run the tests for the
current function (i.e. function around the point)

--8<---------------cut here---------------start------------->8---
(defun org-test-current-function ()
  "Test the current function."
  (ert (format "%S" (which-function))))
--8<---------------cut here---------------end--------------->8---

This way we could maintain a much simpler directory structure inside of
tests/ (or testing/) in which we don't need a separate file name for
every function, but rather maybe one test file per elisp file
(e.g. test-ob.el for ob.el), and possibly other files for tests
organized around concepts that span multiple files (e.g. test-blocks.el
or somesuch).

I may well be misunderstanding the framework you are proposing, so maybe
the best thing to do is to just get started and then see how things
develop.

BTW: back when I worked on ruby-on-rails projects I developed jump.el
[1] for jumping between functions and their tests, I could probably
fairly easily apply this to the org-mode repo if that's desirable.

>
>
>> may make sense, reserving the top level for "meta" testing stuff, like
>> functions for running tests, common fixtures, example files, etc...
>>
>> I have two questions.
>>
>> 1) while waiting for ert to be included into Emacs, should we include an
>>    ert distribution as part of the Org-mode repository (maybe using git
>>    sub-modules) or should we just agree that users should have a certain
>>    version of ert installed locally?  I'm honestly not sure which of
>>    these options sounds preferable.
>
>
> I thought about this, too.  I guess not.  Developers and users that want
> to test will be able to follow the current ERT git repo.
>
> But ERT is just 7 *.el files plus 1 texinfo file.
>
> An what I don't know is:
>
>    How would git submodules work?
>

using git submodules we could specify a location (e.g. tests/ert) and a
version (some particular git commit) for the org-mode repository.  Then
running

  git submodule init
  git submodule update

would checkout the appropriate version of ert to that location.  Users
who don't want to run tests could just never run the two above commands
and shouldn't notice any difference aside from a .gitmodules file in the
base of their org-mode repo.

>
>
>
>
>> 2) should the initial population of the testing/ directory take place in
>>    a separate branch of the repository or in the master branch?  Again I
>>    don't know which I would prefer, branches add complication but could
>>    result in cleaner commit histories.
>
>
> I'll start on a branch first and constantly rebase as long as the
> structure evolves.  The first simple commit will be what you can see on
> github, with some doc strings adjusted.
>
>

Great, when I get a chance I'll check it out and try to write a couple
of simple Babel tests.

Cheers -- Eric

>
>
>    Sebastian
>
>   
>
>> Best -- Eric
>>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Hi Sebastian,
>>>
>>> the lack of a testing suite for Org-mode is really frustrating,
>>> and even more frustrating is that we have had like seven attempts
>>> to start one, and each of these lead to nothing.  So I would
>>> be perfectly happy to give a free hand, write access to the repo
>>> and a full directory in the distribution to implement one.
>>> Once there is a framework, I am sure many people would be
>>> willing to contribute tests.
>>>
>>> More comments below.
>>>
>>> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> I thought about testing again recently.  This is something, that never
>>>> really got started.  For a reason:  there's no framework for testing.
>>>>
>>>> I therefore wrote a very rough proposal,  found  on
>>>> http://github.com/SebastianRose/org-test
>>>>
>>>> The idea is, to provide two simple commands:
>>>>
>>>>
>>>>  *  org-test-test-current-defun
>>>>     will search for tests for the defun point is in or behind
>>>>     (`beginning-of-defun') and execute them surrounded by
>>>>
>>>>  (let ((select (or selector "^org"))
>>>>   	(deactivate-mark nil))
>>>>    (save-excursion
>>>>      (save-match-data
>>>>
>>>>
>>>>  *  org-test-test-buffer-file
>>>>     will search for tests for the entire file and execute them the
>>>> same
>>>>     way.
>>>
>>> FIrst:  I have *no* clue about testing.
>>>
>>> Second, I am surprised that you want to structure it by function.  I
>>> would have
>>> thought that it could be structure by file at the most.  And then
>>> there will
>>> be tests that involve code from many files.
>>>
>>> But I guess
>>>
>>>>
>>>> If you use one of these commands, all currently registered ERT tests
>>>> are
>>>> deleted, and files are reloaded (since you're likely to work on the
>>>> tests, too).  To repeat the tests without reloading, you will use the
>>>> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the
>>>> ERT
>>>> results buffer.
>>>>
>>>>
>>>>
>>>> I choose ERT (git clone http://github.com/ohler/ert.git) because
>>>> that's
>>>> likely to go into Emacs core (or elpa.gnu.org).
>>>>
>>>>
>>>>
>>>>
>>>> The idea is to search the directory structure from the current source
>>>> file upwards for a directory named "tests/" if it exists.  Else ask
>>>> the
>>>> user.  Similar to what `add-change-log-entry' does.
>>>>
>>>> Below that directory, a tree like the source tree exists:
>>>>
>>>> project
>>>>   +-- lisp/
>>>>   |     +-- a.el
>>>>   |     `-- b/
>>>>   |         +-- b.el
>>>>   |
>>>>   `-- tests/
>>>>         +-- a.el/
>>>>         |     +-- tests.el
>>>>         |     `-- a-defun.el
>>>>         `-- b/
>>>>             +-- b.el/
>>>>                   +-- tests.el
>>>>                   `-- b-defun.el
>>>>
>>>> If this setup exists, when editing defun-x in lisp/a.el,
>>>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>>>> (fallback: tests.el there) and execute all tests with selector
>>>> "^a-defun".
>>>
>>> Well, OK, this is fine.  But under a.el and b.el there should also be
>>> general tests that are not function dependent, and there should be a
>>> place
>>> to put tests that you do not want to assign to a specific file.
>>>
>>> We do have a "testing" directory already, you can use that.
>>> I would prefer the tests to be in testing, not in lisp/testing
>>> if possible. I would like to have the lisp directory contain
>>> only code.  If possible.
>>>
>>> It would be OK to have a lisp subdirectory in testing,
>>> just as it would be OK to have contrib/lisp in testing
>>> for the contributed packages.
>>>
>>> PLEASE, go ahead.  I do not think you have write access
>>> yet on repo - give me your user name and I'll activate you.
>>>
>>> - Carsten
>>>
>>>> `M-x org-test-test-buffer-file' in that same source file will load all
>>>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>>>
>>>>
>>>> Thus tests for
>>>>    org-mode/lisp/org-protocol.el
>>>> will be searched in the directory
>>>>    org-mode/tests/lisp/org-protocol.el/*.el
>>>>
>>>>
>>>> Once the basic route of testing is clear, I'd like to "translate" the
>>>> existing tests for org-html.el to work with ERT, which will involve
>>>> writing more tools (create output buffers, compare output with control
>>>> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
>>>> for nXhtml already.  I hope we can use his stuff and help here.
>>>>
>>>> The directory org-mode/lisp/tests/ would not need to be part of the
>>>> "official" Org mode package.  It could as well be checked out
>>>> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>>>>
>>>>
>>>> Any thoughts?
>>>>
>>>>
>>>>
>>>>
>>>>  Sebastian

Footnotes: 
[1]  http://github.com/eschulte/jump.el

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02 20:30       ` Eric Schulte
@ 2010-10-02 21:55         ` Eric Schulte
  2010-10-03  0:46           ` Sebastian Rose
  2010-10-02 22:14         ` Sebastian Rose
  1 sibling, 1 reply; 18+ messages in thread
From: Eric Schulte @ 2010-10-02 21:55 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:

[...]
>
> Could we just load every test in tests/lisp by default, and then use the
> existing `ert' selection method to select and run tests.  For example if
> we enforce our conventions on the level of test function name rather
> than on file name, we could (I believe) do something like the following.
> Say every test for a particular function (say org-export) includes that
> name of that function in the test name (say test-org-export), then we
> could use something simple like the following to run the tests for the
> current function (i.e. function around the point)
>
> (defun org-test-current-function ()
>   "Test the current function."
>   (ert (format "%S" (which-function))))
>
> This way we could maintain a much simpler directory structure inside of
> tests/ (or testing/) in which we don't need a separate file name for
> every function, but rather maybe one test file per elisp file
> (e.g. test-ob.el for ob.el), and possibly other files for tests
> organized around concepts that span multiple files (e.g. test-blocks.el
> or somesuch).
>

To illustrate my suggestions, I've thrown together a couple simple Babel
tests roughly following this outline, currently up in the
`schulte-testing' branch of the Org-mode repo.

To try it out
1) load the testing/org-test.el file
2) run `org-load-tests' to load up the entire org-mode test suite
3) run `ert' to run the test suite.
4) or jump to the definition of `org-babel-get-src-block-info' and run
   `org-test-current-function' to just run the tests for that function

Best -- Eric

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02 20:30       ` Eric Schulte
  2010-10-02 21:55         ` Eric Schulte
@ 2010-10-02 22:14         ` Sebastian Rose
  1 sibling, 0 replies; 18+ messages in thread
From: Sebastian Rose @ 2010-10-02 22:14 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:
> Hi Sebastian,
>
> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>> Hi,
>>>
>>> This is exciting.
>>>
>>> Rather than impose a complete directory/layout schema before-hand I'd
>>> lean towards starting with a little more chaos and then letting the
>>> structure of the test directory develop /naturally/.  From the
>>> discussion below it sounds like an initial structure of
>>>
>>> testing/lisp/
>>> testing/contrib/lisp/
>>
>>
>> I believe it makes sense enforcing rules.  Many developers plus power
>> users will want to be able to use the test system.  No system is what we
>> had in the past.
>>
>> The idea is, to have a system to automate the laoding of tests.  How
>> should a function like `org-test-test-current-defun' find the tests
>> otherwise?
>>
>>   sh$ cd org-mode
>>   sh$ find . -name '*.el' | wc -l
>>   146
>>
>> Also, we could provide services like setup temporary directories,
>> buffers and files for tests.  This cannot be automated in a save way, if
>> there is no structure.  The tests are written in elisp.  Hence one could
>> do whatever he likes ;)
>>
>> It's like Perl.  You don't need to follow the conventions, but it will
>> make your live easier (hopefully).
>>
>> Just what I think.
>>
>
> Point well taken, I suppose my point is more a matter of personal taste,
> and I fully understand if we disagree and would yield to your judgment
> as you've put more thought into this.  But having said that here's my
> thoughts and opinions :)
>
> Could we just load every test in tests/lisp by default, and then use the
> existing `ert' selection method to select and run tests.  For example if
> we enforce our conventions on the level of test function name rather
> than on file name, we could (I believe) do something like the following.
> Say every test for a particular function (say org-export) includes that
> name of that function in the test name (say test-org-export), then we
> could use something simple like the following to run the tests for the
> current function (i.e. function around the point)
>
>
> (defun org-test-current-function ()
>   "Test the current function."
>   (ert (format "%S" (which-function))))


I have no function called `which-function' !?

But yes, those prefixes are (and have to be) part of our "standard".
The entire Emacs elisp structure is build on that way of "namespacing".

It's basically working like that (but I used more lines to get to the
function name...).

All this is _not_ meant to get in the way of ERT.  I just hope a) to add
some support to get people started and b) provide some help and
maintainance of that testing stuff.


E.g. `org-test-run-tests (&optional selector)'  runs tests enclosed in 

  (let ((deactivate-mark nil))
    (save-excursion
      (save-match-data
      ;; ... run tests here
      )))

But yes, I guess the art is to do not too much.


> This way we could maintain a much simpler directory structure inside of
> tests/ (or testing/) in which we don't need a separate file name for
> every function, but rather maybe one test file per elisp file
> (e.g. test-ob.el for ob.el), and possibly other files for tests
> organized around concepts that span multiple files (e.g. test-blocks.el
> or somesuch).


Hmmmm - I see.  But you'd still test single units of code, and a single
unit of code lives in one source file, not in many source files.  Even
though many source files are involved, which is always the case when you
execute code.


> I may well be misunderstanding the framework you are proposing, so maybe
> the best thing to do is to just get started and then see how things
> develop.

OK. Same here :)

I just pushed my little starter to a new branch "ert-testing".  Pooha,
git always a little challenging in such rare situations :)


> BTW: back when I worked on ruby-on-rails projects I developed jump.el
> [1] for jumping between functions and their tests, I could probably
> fairly easily apply this to the org-mode repo if that's desirable.


That's such a helpful little tool I think of!  I'll check that, too :)


I just created the branch ert-testing and pushed my little starter.


>>> may make sense, reserving the top level for "meta" testing stuff, like
>>> functions for running tests, common fixtures, example files, etc...
>>>
>>> I have two questions.
>>>
>>> 1) while waiting for ert to be included into Emacs, should we include an
>>>    ert distribution as part of the Org-mode repository (maybe using git
>>>    sub-modules) or should we just agree that users should have a certain
>>>    version of ert installed locally?  I'm honestly not sure which of
>>>    these options sounds preferable.
>>
>>
>> I thought about this, too.  I guess not.  Developers and users that want
>> to test will be able to follow the current ERT git repo.
>>
>> But ERT is just 7 *.el files plus 1 texinfo file.
>>
>> An what I don't know is:
>>
>>    How would git submodules work?
>>
>
> using git submodules we could specify a location (e.g. tests/ert) and a
> version (some particular git commit) for the org-mode repository.  Then
> running
>
>   git submodule init
>   git submodule update
>
> would checkout the appropriate version of ert to that location.  Users
> who don't want to run tests could just never run the two above commands
> and shouldn't notice any difference aside from a .gitmodules file in the
> base of their org-mode repo.


Wow.  That sounds _very_ good.  From what I read on emacs-devel,
Christian Ohler is going to adjust the code of ERT to get it into Emacs'
core.  Using this technique, we would ensure that the tests work, even
though ERT has changed.  We'd just need to "update" the sub project and
all developers will pull the right thing then?

That sounds absolute fantastic!  It's the best way to do it I came
across so far :)





   Sebastian



>>> 2) should the initial population of the testing/ directory take place in
>>>    a separate branch of the repository or in the master branch?  Again I
>>>    don't know which I would prefer, branches add complication but could
>>>    result in cleaner commit histories.
>>
>>
>> I'll start on a branch first and constantly rebase as long as the
>> structure evolves.  The first simple commit will be what you can see on
>> github, with some doc strings adjusted.
>>
>>
>
> Great, when I get a chance I'll check it out and try to write a couple
> of simple Babel tests.
>
> Cheers -- Eric





>>
>>
>>    Sebastian
>>
>>   
>>
>>> Best -- Eric
>>>
>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>
>>>> Hi Sebastian,
>>>>
>>>> the lack of a testing suite for Org-mode is really frustrating,
>>>> and even more frustrating is that we have had like seven attempts
>>>> to start one, and each of these lead to nothing.  So I would
>>>> be perfectly happy to give a free hand, write access to the repo
>>>> and a full directory in the distribution to implement one.
>>>> Once there is a framework, I am sure many people would be
>>>> willing to contribute tests.
>>>>
>>>> More comments below.
>>>>
>>>> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> I thought about testing again recently.  This is something, that never
>>>>> really got started.  For a reason:  there's no framework for testing.
>>>>>
>>>>> I therefore wrote a very rough proposal,  found  on
>>>>> http://github.com/SebastianRose/org-test
>>>>>
>>>>> The idea is, to provide two simple commands:
>>>>>
>>>>>
>>>>>  *  org-test-test-current-defun
>>>>>     will search for tests for the defun point is in or behind
>>>>>     (`beginning-of-defun') and execute them surrounded by
>>>>>
>>>>>  (let ((select (or selector "^org"))
>>>>>   	(deactivate-mark nil))
>>>>>    (save-excursion
>>>>>      (save-match-data
>>>>>
>>>>>
>>>>>  *  org-test-test-buffer-file
>>>>>     will search for tests for the entire file and execute them the
>>>>> same
>>>>>     way.
>>>>
>>>> FIrst:  I have *no* clue about testing.
>>>>
>>>> Second, I am surprised that you want to structure it by function.  I
>>>> would have
>>>> thought that it could be structure by file at the most.  And then
>>>> there will
>>>> be tests that involve code from many files.
>>>>
>>>> But I guess
>>>>
>>>>>
>>>>> If you use one of these commands, all currently registered ERT tests
>>>>> are
>>>>> deleted, and files are reloaded (since you're likely to work on the
>>>>> tests, too).  To repeat the tests without reloading, you will use the
>>>>> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the
>>>>> ERT
>>>>> results buffer.
>>>>>
>>>>>
>>>>>
>>>>> I choose ERT (git clone http://github.com/ohler/ert.git) because
>>>>> that's
>>>>> likely to go into Emacs core (or elpa.gnu.org).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The idea is to search the directory structure from the current source
>>>>> file upwards for a directory named "tests/" if it exists.  Else ask
>>>>> the
>>>>> user.  Similar to what `add-change-log-entry' does.
>>>>>
>>>>> Below that directory, a tree like the source tree exists:
>>>>>
>>>>> project
>>>>>   +-- lisp/
>>>>>   |     +-- a.el
>>>>>   |     `-- b/
>>>>>   |         +-- b.el
>>>>>   |
>>>>>   `-- tests/
>>>>>         +-- a.el/
>>>>>         |     +-- tests.el
>>>>>         |     `-- a-defun.el
>>>>>         `-- b/
>>>>>             +-- b.el/
>>>>>                   +-- tests.el
>>>>>                   `-- b-defun.el
>>>>>
>>>>> If this setup exists, when editing defun-x in lisp/a.el,
>>>>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>>>>> (fallback: tests.el there) and execute all tests with selector
>>>>> "^a-defun".
>>>>
>>>> Well, OK, this is fine.  But under a.el and b.el there should also be
>>>> general tests that are not function dependent, and there should be a
>>>> place
>>>> to put tests that you do not want to assign to a specific file.
>>>>
>>>> We do have a "testing" directory already, you can use that.
>>>> I would prefer the tests to be in testing, not in lisp/testing
>>>> if possible. I would like to have the lisp directory contain
>>>> only code.  If possible.
>>>>
>>>> It would be OK to have a lisp subdirectory in testing,
>>>> just as it would be OK to have contrib/lisp in testing
>>>> for the contributed packages.
>>>>
>>>> PLEASE, go ahead.  I do not think you have write access
>>>> yet on repo - give me your user name and I'll activate you.
>>>>
>>>> - Carsten
>>>>
>>>>> `M-x org-test-test-buffer-file' in that same source file will load all
>>>>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>>>>
>>>>>
>>>>> Thus tests for
>>>>>    org-mode/lisp/org-protocol.el
>>>>> will be searched in the directory
>>>>>    org-mode/tests/lisp/org-protocol.el/*.el
>>>>>
>>>>>
>>>>> Once the basic route of testing is clear, I'd like to "translate" the
>>>>> existing tests for org-html.el to work with ERT, which will involve
>>>>> writing more tools (create output buffers, compare output with control
>>>>> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
>>>>> for nXhtml already.  I hope we can use his stuff and help here.
>>>>>
>>>>> The directory org-mode/lisp/tests/ would not need to be part of the
>>>>> "official" Org mode package.  It could as well be checked out
>>>>> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>>>>>
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  Sebastian
>
> Footnotes: 
> [1]  http://github.com/eschulte/jump.el

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-02 21:55         ` Eric Schulte
@ 2010-10-03  0:46           ` Sebastian Rose
  2010-10-04  1:26             ` Eric Schulte
  0 siblings, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-03  0:46 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:
> "Eric Schulte" <schulte.eric@gmail.com> writes:
> To illustrate my suggestions, I've thrown together a couple simple Babel
> tests roughly following this outline, currently up in the
> `schulte-testing' branch of the Org-mode repo.
>
> To try it out
> 1) load the testing/org-test.el file
> 2) run `org-load-tests' to load up the entire org-mode test suite
> 3) run `ert' to run the test suite.
> 4) or jump to the definition of `org-babel-get-src-block-info' and run
>    `org-test-current-function' to just run the tests for that function
>
> Best -- Eric

Yes.  I know how running tests looks like:  It looks good :)

Hahaa, I like that code.  It looks so simple :)


To load all files below testing/  is a good suggestion probably.  The
unloading and loading as I implemented it is simply superfluous, if we
suggest that testers no how to eval a buffer or defun (and if they can
change it, they can eval it).



* Question:

  (defun org-test-current-function ()
    "Test the current function."
    (interactive)
    (ert (car (which-function))))
  
  `org-test-test-current-defun' in the ert-testing branch does that, too.
  (But with that superfluous loading/unloading stuff though :-/) But it
  encloses each test inside save-excursion and Co.
  
  What does which-func.el that this function does not:
  
  <#part type="application/emacs-lisp" disposition=inline>
  (defun org-test-which-func ()
    "Return the name of the current defun."
    (save-excursion
      (save-match-data
        (end-of-line)
        (beginning-of-defun)
        (if (looking-at "(defun[[:space:]]+\\([^([:space:]]*\\)[[:space:]]*(")
          (match-string-no-properties 1)
      (error "No defun found around point.")))))
  <#/part>
  
  ??



* Keymap

  We should add keys to the org-mode-map.
  
   C-c t
  
  is still free here.  Or is it in you use it for babel somehow?
  
   C-c t f     org-test-test-current-function
   C-c t b     org-test-test-current-buffer-file




* ERT Selectors

  I see a little namespace problem coming up.  Imagine testing org.el.
  Which ert selector would we use?
  
  "^org" ???  :-/

  Should we create hashes of filenames as selectors (just kidding)?
  Or use the entire filename "^org.el"?  The relative path
  "^lisp/org.el"? 






* About the directory structure:

  It does not burden the user, as tests are loaded and executed
  automatically (per function, per file or all, depending on the command
  used).  But imagine the entire thing grows and someone would add tests
  for all the stuff in emacs/lisp/.  Or add tests files for single
  functions simply because there is so much to test that one file would
  be hard to handle.
  This could cause a lot of clutter.

  Is this too hypothetical?  Hmmm - it might be...

  If you checkout ert-testing, eval testing/org-test.el and do

     M-x org-test-edit-buffer-file-tests

  you're in your test file in the correct directory (which is created if
  it doesn't exist).

     M-x org-test-edit-current-defuns-tests

  creates a file named after the defun you're in.  You'll need to be in
  an elisp file of course.  The testing/ directory will resemble the
  directory structure of the project.

  Every one who saw that directory structure simply asked "don't you
  think it's overkill?" :D  I'll probably drop that.

  

  We could go on on two rails for a while.  I'll need this week to dig
  deeper into ERT and come to conclusions.

  It would be a good thing if we could agree about the keys and the
  selectors (again: "^org"), so that it's painless to switch branches ;)






  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-03  0:46           ` Sebastian Rose
@ 2010-10-04  1:26             ` Eric Schulte
  2010-10-04  3:59               ` Sebastian Rose
  0 siblings, 1 reply; 18+ messages in thread
From: Eric Schulte @ 2010-10-04  1:26 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list, Carsten Dominik

Hi,

I've taken the liberty of merging our two testing branches into the new
"combined-testing" branch.  This now includes both ert and jump.el as
git submodules, which can be installed with

  git submodule init
  git submodule update

after checking out the branch.  In merging the two org-test.el files, I
was able to remove much of the existing code through using which-func,
jump.el.  The new navigation functionality will work regardless of the
directory structure, so tests for file

  lisp/org-foo.el

can be located in either of the following

  testing/lisp/test-org-foo.el
  testing/lisp/org-foo.el/test.el

Additionally the navigation functions defined in jump.el should be
easily extensible to accommodate new naming schemas, so ideally every
test author for a particular file can use whatever naming system they
prefer.  Note that the navigation function `org-test-jump' when called
with a prefix argument uses your code from
`org-test-edit-buffer-file-tests' to create the test file if it is not
already present.

I think I retained most of the functionality from your version of
org-test.el in the merge but please let me know if I broke something.
In my mind this merged version should be small clean and maintainable
and hopefully flexible enough to handle whatever needs we identify
during the course of writing the tests.  How would you feel about using
this as the new base of test development?

Best -- Eric

also I have some inline comments below

Sebastian Rose <sebastian_rose@gmx.de> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>> To illustrate my suggestions, I've thrown together a couple simple Babel
>> tests roughly following this outline, currently up in the
>> `schulte-testing' branch of the Org-mode repo.
>>
>> To try it out
>> 1) load the testing/org-test.el file
>> 2) run `org-load-tests' to load up the entire org-mode test suite
>> 3) run `ert' to run the test suite.
>> 4) or jump to the definition of `org-babel-get-src-block-info' and run
>>    `org-test-current-function' to just run the tests for that function
>>
>> Best -- Eric
>
> Yes.  I know how running tests looks like:  It looks good :)
>
> Hahaa, I like that code.  It looks so simple :)
>
>
> To load all files below testing/  is a good suggestion probably.  The
> unloading and loading as I implemented it is simply superfluous, if we
> suggest that testers no how to eval a buffer or defun (and if they can
> change it, they can eval it).
>
>
>
> * Question:
>
>   (defun org-test-current-function ()
>     "Test the current function."
>     (interactive)
>     (ert (car (which-function))))
>   
>   `org-test-test-current-defun' in the ert-testing branch does that, too.
>   (But with that superfluous loading/unloading stuff though :-/) But it
>   encloses each test inside save-excursion and Co.
>   

I believe ert takes care of the save-excursion stuff

>   
>   What does which-func.el that this function does not:
>   
>   <#part type="application/emacs-lisp" disposition=inline>
>   (defun org-test-which-func ()
>     "Return the name of the current defun."
>     (save-excursion
>       (save-match-data
>         (end-of-line)
>         (beginning-of-defun)
>         (if (looking-at "(defun[[:space:]]+\\([^([:space:]]*\\)[[:space:]]*(")
>           (match-string-no-properties 1)
>       (error "No defun found around point.")))))
>   <#/part>
>   
>   ??
>

I'm not sure that it does include anything new, but it's nice to re-use
existing packages.

>
>
>
> * Keymap
>
>   We should add keys to the org-mode-map.
>   
>    C-c t
>   
>   is still free here.  Or is it in you use it for babel somehow?
>   
>    C-c t f     org-test-test-current-function
>    C-c t b     org-test-test-current-buffer-file
>
>

Wouldn't we want the keys in the elisp-mode key map, since we'll be
doing the testing work from elisp?  Although I guess we may want to run
tests from within Org-mode files, but then we could do that with a Babel
block and dump the results to a table. :)

>
>
>
> * ERT Selectors
>
>   I see a little namespace problem coming up.  Imagine testing org.el.
>   Which ert selector would we use?
>   
>   "^org" ???  :-/
>
>   Should we create hashes of filenames as selectors (just kidding)?
>   Or use the entire filename "^org.el"?  The relative path
>   "^lisp/org.el"? 
>

oh, good point, maybe we'd need to use the "eql" or "tag" ert selectors
in this case.

>
>
>
> * About the directory structure:
>
>   It does not burden the user, as tests are loaded and executed
>   automatically (per function, per file or all, depending on the command
>   used).  But imagine the entire thing grows and someone would add tests
>   for all the stuff in emacs/lisp/.  Or add tests files for single
>   functions simply because there is so much to test that one file would
>   be hard to handle.
>   This could cause a lot of clutter.
>
>   Is this too hypothetical?  Hmmm - it might be...
>
>   If you checkout ert-testing, eval testing/org-test.el and do
>
>      M-x org-test-edit-buffer-file-tests
>

In the combined-testing branch you'd do this by calling org-test-jump
with a prefix argument.

>
>   you're in your test file in the correct directory (which is created
>   if it doesn't exist).
>
>      M-x org-test-edit-current-defuns-tests
>
>   creates a file named after the defun you're in.  You'll need to be in
>   an elisp file of course.  The testing/ directory will resemble the
>   directory structure of the project.
>
>   Every one who saw that directory structure simply asked "don't you
>   think it's overkill?" :D  I'll probably drop that.
>

heh, yea I'd lean towards getting into the writing of tests and then
leaving our implementation flexible enough so that we can adopt this
more fine-grained directory structure when/if it becomes necessary.

>
>   
>
>   We could go on on two rails for a while.  I'll need this week to dig
>   deeper into ERT and come to conclusions.
>
>   It would be a good thing if we could agree about the keys and the
>   selectors (again: "^org"), so that it's painless to switch branches ;)
>
>
>
>
>
>
>   Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-04  1:26             ` Eric Schulte
@ 2010-10-04  3:59               ` Sebastian Rose
  2010-10-04 15:26                 ` Eric Schulte
  0 siblings, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-04  3:59 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:
> Hi,
>
> I've taken the liberty of merging our two testing branches into the new
> "combined-testing" branch.  This now includes both ert and jump.el as
> git submodules, which can be installed with
>
>   git submodule init
>   git submodule update
>
> after checking out the branch.  In merging the two org-test.el files, I
> was able to remove much of the existing code through using which-func,
> jump.el.  The new navigation functionality will work regardless of the
> directory structure, so tests for file
>
>   lisp/org-foo.el
>
> can be located in either of the following
>
>   testing/lisp/test-org-foo.el
>   testing/lisp/org-foo.el/test.el
>
> Additionally the navigation functions defined in jump.el should be
> easily extensible to accommodate new naming schemas, so ideally every
> test author for a particular file can use whatever naming system they
> prefer.  Note that the navigation function `org-test-jump' when called
> with a prefix argument uses your code from
> `org-test-edit-buffer-file-tests' to create the test file if it is not
> already present.
>
> I think I retained most of the functionality from your version of
> org-test.el in the merge but please let me know if I broke something.
> In my mind this merged version should be small clean and maintainable
> and hopefully flexible enough to handle whatever needs we identify
> during the course of writing the tests.  How would you feel about using
> this as the new base of test development?
>
> Best -- Eric
>
> also I have some inline comments below


Hi Eric,


that's good news!


> Sebastian Rose <sebastian_rose@gmx.de> writes:
>>   What does which-func.el that this function does not:
>>   
>>   <#part type="application/emacs-lisp" disposition=inline>
>>   (defun org-test-which-func ()
>>     "Return the name of the current defun."
>>     (save-excursion
>>       (save-match-data
>>         (end-of-line)
>>         (beginning-of-defun)
>>         (if (looking-at "(defun[[:space:]]+\\([^([:space:]]*\\)[[:space:]]*(")
>>           (match-string-no-properties 1)
>>       (error "No defun found around point.")))))
>>   <#/part>
>>   
>>   ??
>>
>
> I'm not sure that it does include anything new, but it's nice to re-use
> existing packages.


And a dependency.


>
>> * Keymap
>>
>>   We should add keys to the org-mode-map.
>>   
>>    C-c t
>>   
>>   is still free here.  Or is it in you use it for babel somehow?
>>   
>>    C-c t f     org-test-test-current-function
>>    C-c t b     org-test-test-current-buffer-file
>>
>>
>
> Wouldn't we want the keys in the elisp-mode key map, since we'll be
> doing the testing work from elisp?

Errrmmm, yes, we definetively want the keys in the emacs-lisp key map :)


>  Although I guess we may want to run
> tests from within Org-mode files, but then we could do that with a Babel
> block and dump the results to a table. :)

Ho ho ho ho ho!



>> * ERT Selectors
>>
>>   I see a little namespace problem coming up.  Imagine testing org.el.
>>   Which ert selector would we use?
>>   
>>   "^org" ???  :-/
>>
>>   Should we create hashes of filenames as selectors (just kidding)?
>>   Or use the entire filename "^org.el"?  The relative path
>>   "^lisp/org.el"? 
>>
>
> oh, good point, maybe we'd need to use the "eql" or "tag" ert selectors
> in this case.



It's a flaw in Org's namespace actually.  A defun's name in org.el could
clash with the name of a defun in another file in org-mode/lisp/.
The defun `org-imenu-get-tree' in org.el could clash with a defun with
that very name in a new file "org-imenu.el".

I guess we'll have to handle org.el in a special way.



>>   If you checkout ert-testing, eval testing/org-test.el and do
>>
>>      M-x org-test-edit-buffer-file-tests
>>
>
> In the combined-testing branch you'd do this by calling org-test-jump
> with a prefix argument.


Perfect.



>>   Every one who saw that directory structure simply asked "don't you
>>   think it's overkill?" :D  I'll probably drop that.
>>
>
> heh, yea I'd lean towards getting into the writing of tests and then
> leaving our implementation flexible enough so that we can adopt this
> more fine-grained directory structure when/if it becomes necessary.


Eric, so just let's skip the directory structure.  We could always
enforce something like that once we feel it's necessary.

We just want some reliable way to load the sensible tests (for a defun,
file, module or whatever) and execute them with just a key stroke.

Adding new tests for an existing elisp file should be just as easy.

As jump.el and which-func.el do part of that job reliably it's perfectly
fine to go that way.




  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-04  3:59               ` Sebastian Rose
@ 2010-10-04 15:26                 ` Eric Schulte
  2010-10-05  0:21                   ` Sebastian Rose
  2010-10-06  0:48                   ` Bernt Hansen
  0 siblings, 2 replies; 18+ messages in thread
From: Eric Schulte @ 2010-10-04 15:26 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list, Carsten Dominik

Hi,

So, I've been using the framework in the combined-testing branch this
morning writing tests to strap down my daily Babel bug fixes, and I'm
really liking ERT.

I wonder, can we commit to the combined-testing branch, and if so could
we fold it into the master branch?  It would make my test driven bug
fixing a much smoother process, and would remove the need to rebase the
combined-testing branch against master and "git push -f" the changes up
to repo.or.cz which just feels wrong.

Thoughts?

-- Eric

Sebastian Rose <sebastian_rose@gmx.de> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>> Hi,
>>
>> I've taken the liberty of merging our two testing branches into the new
>> "combined-testing" branch.  This now includes both ert and jump.el as
>> git submodules, which can be installed with
>>
>>   git submodule init
>>   git submodule update
>>
>> after checking out the branch.  In merging the two org-test.el files, I
>> was able to remove much of the existing code through using which-func,
>> jump.el.  The new navigation functionality will work regardless of the
>> directory structure, so tests for file
>>
>>   lisp/org-foo.el
>>
>> can be located in either of the following
>>
>>   testing/lisp/test-org-foo.el
>>   testing/lisp/org-foo.el/test.el
>>
>> Additionally the navigation functions defined in jump.el should be
>> easily extensible to accommodate new naming schemas, so ideally every
>> test author for a particular file can use whatever naming system they
>> prefer.  Note that the navigation function `org-test-jump' when called
>> with a prefix argument uses your code from
>> `org-test-edit-buffer-file-tests' to create the test file if it is not
>> already present.
>>
>> I think I retained most of the functionality from your version of
>> org-test.el in the merge but please let me know if I broke something.
>> In my mind this merged version should be small clean and maintainable
>> and hopefully flexible enough to handle whatever needs we identify
>> during the course of writing the tests.  How would you feel about using
>> this as the new base of test development?
>>
>> Best -- Eric
>>
>> also I have some inline comments below
>
>
> Hi Eric,
>
>
> that's good news!
>
>
>> Sebastian Rose <sebastian_rose@gmx.de> writes:
>>>   What does which-func.el that this function does not:
>>>   
>>>   <#part type="application/emacs-lisp" disposition=inline>
>>>   (defun org-test-which-func ()
>>>     "Return the name of the current defun."
>>>     (save-excursion
>>>       (save-match-data
>>>         (end-of-line)
>>>         (beginning-of-defun)
>>>         (if (looking-at "(defun[[:space:]]+\\([^([:space:]]*\\)[[:space:]]*(")
>>>           (match-string-no-properties 1)
>>>       (error "No defun found around point.")))))
>>>   <#/part>
>>>   
>>>   ??
>>>
>>
>> I'm not sure that it does include anything new, but it's nice to re-use
>> existing packages.
>
>
> And a dependency.
>
>
>>
>>> * Keymap
>>>
>>>   We should add keys to the org-mode-map.
>>>   
>>>    C-c t
>>>   
>>>   is still free here.  Or is it in you use it for babel somehow?
>>>   
>>>    C-c t f     org-test-test-current-function
>>>    C-c t b     org-test-test-current-buffer-file
>>>
>>>
>>
>> Wouldn't we want the keys in the elisp-mode key map, since we'll be
>> doing the testing work from elisp?
>
> Errrmmm, yes, we definetively want the keys in the emacs-lisp key map :)
>
>
>>  Although I guess we may want to run
>> tests from within Org-mode files, but then we could do that with a Babel
>> block and dump the results to a table. :)
>
> Ho ho ho ho ho!
>
>
>
>>> * ERT Selectors
>>>
>>>   I see a little namespace problem coming up.  Imagine testing org.el.
>>>   Which ert selector would we use?
>>>   
>>>   "^org" ???  :-/
>>>
>>>   Should we create hashes of filenames as selectors (just kidding)?
>>>   Or use the entire filename "^org.el"?  The relative path
>>>   "^lisp/org.el"? 
>>>
>>
>> oh, good point, maybe we'd need to use the "eql" or "tag" ert selectors
>> in this case.
>
>
>
> It's a flaw in Org's namespace actually.  A defun's name in org.el could
> clash with the name of a defun in another file in org-mode/lisp/.
> The defun `org-imenu-get-tree' in org.el could clash with a defun with
> that very name in a new file "org-imenu.el".
>
> I guess we'll have to handle org.el in a special way.
>
>
>
>>>   If you checkout ert-testing, eval testing/org-test.el and do
>>>
>>>      M-x org-test-edit-buffer-file-tests
>>>
>>
>> In the combined-testing branch you'd do this by calling org-test-jump
>> with a prefix argument.
>
>
> Perfect.
>
>
>
>>>   Every one who saw that directory structure simply asked "don't you
>>>   think it's overkill?" :D  I'll probably drop that.
>>>
>>
>> heh, yea I'd lean towards getting into the writing of tests and then
>> leaving our implementation flexible enough so that we can adopt this
>> more fine-grained directory structure when/if it becomes necessary.
>
>
> Eric, so just let's skip the directory structure.  We could always
> enforce something like that once we feel it's necessary.
>
> We just want some reliable way to load the sensible tests (for a defun,
> file, module or whatever) and execute them with just a key stroke.
>
> Adding new tests for an existing elisp file should be just as easy.
>
> As jump.el and which-func.el do part of that job reliably it's perfectly
> fine to go that way.
>
>
>
>
>   Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-04 15:26                 ` Eric Schulte
@ 2010-10-05  0:21                   ` Sebastian Rose
  2010-10-05 18:57                     ` Eric Schulte
  2010-10-06  0:48                   ` Bernt Hansen
  1 sibling, 1 reply; 18+ messages in thread
From: Sebastian Rose @ 2010-10-05  0:21 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:
> Hi,
>
> So, I've been using the framework in the combined-testing branch this
> morning writing tests to strap down my daily Babel bug fixes, and I'm
> really liking ERT.
>
> I wonder, can we commit to the combined-testing branch, and if so could
> we fold it into the master branch?  It would make my test driven bug
> fixing a much smoother process, and would remove the need to rebase the
> combined-testing branch against master and "git push -f" the changes up
> to repo.or.cz which just feels wrong.


Yes,  commit to that branch.  And yes, fold it into master,  if there
are no objections.  In the worst case there will be more contributions.




Eric, can we keep track with the ERT development for a little while?
That will not be done automatically, right?   But, as you know,
Christian is working on it to get it into Emacs, and we should use the
version that finally will make it there.

  git submodul update

Should that be called once to avoid conflicts?


  Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Re: Testing --- again...
  2010-10-05  0:21                   ` Sebastian Rose
@ 2010-10-05 18:57                     ` Eric Schulte
  0 siblings, 0 replies; 18+ messages in thread
From: Eric Schulte @ 2010-10-05 18:57 UTC (permalink / raw)
  To: Sebastian Rose; +Cc: Emacs-orgmode mailing list, Carsten Dominik

Sebastian Rose <sebastian_rose@gmx.de> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>> Hi,
>>
>> So, I've been using the framework in the combined-testing branch this
>> morning writing tests to strap down my daily Babel bug fixes, and I'm
>> really liking ERT.
>>
>> I wonder, can we commit to the combined-testing branch, and if so could
>> we fold it into the master branch?  It would make my test driven bug
>> fixing a much smoother process, and would remove the need to rebase the
>> combined-testing branch against master and "git push -f" the changes up
>> to repo.or.cz which just feels wrong.
>
>
> Yes,  commit to that branch.  And yes, fold it into master,  if there
> are no objections.  In the worst case there will be more contributions.
>
>

Great, I've just merged this into the master branch.

>
>
>
> Eric, can we keep track with the ERT development for a little while?
> That will not be done automatically, right? 

Right, but at our leasure we can
1) cd into the testing/ert directory
2) git pull the latest version of ert into that directory
3) and on our next git commit the update will be committed to the git
   submodules

> But, as you know, Christian is working on it to get it into Emacs, and
> we should use the version that finally will make it there.
>
>   git submodule update
>
> Should that be called once to avoid conflicts?
>

Most users need never run

  git submodule init

or

  git subomdule update

however for those of us writing and using tests, we will want to run

  git submodule init

at least once, and run

  git submodule update

from time to time, to pull down newer versions of ert which may have
been committed to the org-mode repo by the process described at the top
of this email.

Best -- Eric

>
>
>   Sebastian

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-04 15:26                 ` Eric Schulte
  2010-10-05  0:21                   ` Sebastian Rose
@ 2010-10-06  0:48                   ` Bernt Hansen
  2010-10-06  2:57                     ` Eric Schulte
  1 sibling, 1 reply; 18+ messages in thread
From: Bernt Hansen @ 2010-10-06  0:48 UTC (permalink / raw)
  To: Eric Schulte; +Cc: Emacs-orgmode mailing list, Carsten Dominik

"Eric Schulte" <schulte.eric@gmail.com> writes:

> So, I've been using the framework in the combined-testing branch this
> morning writing tests to strap down my daily Babel bug fixes, and I'm
> really liking ERT.
>
> I wonder, can we commit to the combined-testing branch, and if so could
> we fold it into the master branch?  It would make my test driven bug
> fixing a much smoother process, and would remove the need to rebase the
> combined-testing branch against master and "git push -f" the changes up
> to repo.or.cz which just feels wrong.
>
> Thoughts?

Hi Eric and Sebastian,

There's no technical reason that you need to rebase the combined-testing
branch -- there is nothing wrong with making small commits and merging
master into it to fix any conflicts -- eventually when it gets merged
back to master the history will stay intact.

I personally only rebase local work that hasn't been published.  As soon
as the combined-testing branch is useful and public I think it should no
longer be rebased.

I'm really looking forward to getting my hands dirty in this testing
stuff for the clocking functions (since I seem to break those a lot :-P)

Thanks for all of your efforts so far and please keep up the great work!

Regards,
Bernt

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: Testing --- again...
  2010-10-06  0:48                   ` Bernt Hansen
@ 2010-10-06  2:57                     ` Eric Schulte
  0 siblings, 0 replies; 18+ messages in thread
From: Eric Schulte @ 2010-10-06  2:57 UTC (permalink / raw)
  To: Bernt Hansen; +Cc: Emacs-orgmode mailing list, Carsten Dominik

Bernt Hansen <bernt@norang.ca> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> So, I've been using the framework in the combined-testing branch this
>> morning writing tests to strap down my daily Babel bug fixes, and I'm
>> really liking ERT.
>>
>> I wonder, can we commit to the combined-testing branch, and if so could
>> we fold it into the master branch?  It would make my test driven bug
>> fixing a much smoother process, and would remove the need to rebase the
>> combined-testing branch against master and "git push -f" the changes up
>> to repo.or.cz which just feels wrong.
>>
>> Thoughts?
>
> Hi Eric and Sebastian,
>
> There's no technical reason that you need to rebase the combined-testing
> branch -- there is nothing wrong with making small commits and merging
> master into it to fix any conflicts -- eventually when it gets merged
> back to master the history will stay intact.
>

Noted, I was just shying away from including too many commits in the
revision history.

>
> I personally only rebase local work that hasn't been published.  As
> soon as the combined-testing branch is useful and public I think it
> should no longer be rebased.
>

Ah, that's a good point.  Although this is all moot now as the testing
branch has now been merged into the master branch.

>
> I'm really looking forward to getting my hands dirty in this testing
> stuff for the clocking functions (since I seem to break those a lot :-P)
>

Great, I'm really looking forward to seeing how the test framework
develops as it gets more users.

Best -- Eric

>
> Thanks for all of your efforts so far and please keep up the great
> work!
>
> Regards,
> Bernt

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2010-10-06  3:29 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-10-02  3:51 Testing --- again Sebastian Rose
2010-10-02  8:09 ` Carsten Dominik
2010-10-02 14:53   ` Sebastian Rose
2010-10-02 15:01   ` Sebastian Rose
2010-10-02 17:05     ` Carsten Dominik
2010-10-02 18:12   ` Eric Schulte
2010-10-02 19:33     ` Sebastian Rose
2010-10-02 20:30       ` Eric Schulte
2010-10-02 21:55         ` Eric Schulte
2010-10-03  0:46           ` Sebastian Rose
2010-10-04  1:26             ` Eric Schulte
2010-10-04  3:59               ` Sebastian Rose
2010-10-04 15:26                 ` Eric Schulte
2010-10-05  0:21                   ` Sebastian Rose
2010-10-05 18:57                     ` Eric Schulte
2010-10-06  0:48                   ` Bernt Hansen
2010-10-06  2:57                     ` Eric Schulte
2010-10-02 22:14         ` Sebastian Rose

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).