Compiling it I get ,---- | Compiling /home/nick/src/emacs/org/org-mode/lisp/ob-sql.el... | | In org-babel-execute:sql: | ob-sql.el:143:10:Warning: reference to free variable `cond' | ob-sql.el:170:26:Warning: `t' called as a function | ob-sql.el:181:66:Error: Invalid read syntax: ")" `---- I don't know what the reference to free variable `cond' is all about (sounds bogus to me), but the invalid read syntax is real. Can we please make it an invariable practice to run `make test' before every push? Thanks, Nick Org-mode version 8.0-pre (release_8.0-pre-144-g855dcf.dirty @ /home/nick/elisp/org-mode/lisp/)
Hi Nick,
Nick Dokos <nicholas.dokos@hp.com> writes:
> Compiling it I get
This is fixed, thanks.
--
Bastien
Hi Nick,
Nick Dokos wrote:
> Can we please make it an invariable practice to run `make test' before
> every push?
Isn't it possible to put such in some sort of Git pre-commit hook (or pre-push
hook), so that it gets automatically enforced?
Best regards,
Seb
--
Sebastien Vauban
Hi Sébastien, "Sebastien Vauban" <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes: > Nick Dokos wrote: >> Can we please make it an invariable practice to run `make test' before >> every push? > > Isn't it possible to put such in some sort of Git pre-commit hook (or pre-push > hook), so that it gets automatically enforced? If anyone knows how to setup an automated tests framework for Org, feel free to go ahead, we will use it and monitor broken tests to see what's wrong in the code or in the tests or in the environment running the tests. Testing is a nice habit to have, but let's not make it a coercive pre-requisit before pushing patches. My whole thinking here is well captured by Rich Hickey: http://codequarterly.com/2011/rich-hickey/ Fogus: You have been known to speak out against test-driven development. Do you mind elaborating on your position? Hickey: I never spoke out ‘against’ TDD. What I have said is, life is short and there are only a finite number of hours in a day. So, we have to make choices about how we spend our time. If we spend it writing tests, that is time we are not spending doing something else. Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design. Best, -- Bastien
Am 20.03.2013 14:47, schrieb Bastien: > If anyone knows how to setup an automated tests framework for Org, > feel free to go ahead, we will use it and monitor broken tests to > see what's wrong in the code or in the tests or in the environment > running the tests. We already have one, what Nick and Sebastien are asking is not to push commits that are known to not pass the tests. > Testing is a nice habit to have, but let's not make it a coercive > pre-requisit before pushing patches. Why not? Any broken commits make automatic bisecting impossible and they are a constant source of irritation for folks who forget to test their new Org pulls before using or installing them. > My whole thinking here is well captured by Rich Hickey: The citation you gave doesn't even apply to the question at hand. It is about writing tests, not using the tests you already have. Regards, -- Achim. (on the road :-)
Hi Bastien,
Bastien wrote:
> "Sebastien Vauban" writes:
>> Nick Dokos wrote:
>>> Can we please make it an invariable practice to run `make test' before
>>> every push?
>>
>> Isn't it possible to put such in some sort of Git pre-commit hook (or
>> pre-push hook), so that it gets automatically enforced?
>
> If anyone knows how to setup an automated tests framework for Org, feel free
> to go ahead, we will use it and monitor broken tests to see what's wrong in
> the code or in the tests or in the environment running the tests.
>
> Testing is a nice habit to have, but let's not make it a coercive
> pre-requisit before pushing patches.
>
> My whole thinking here is well captured by Rich Hickey:
>
> http://codequarterly.com/2011/rich-hickey/
>
> Fogus: You have been known to speak out against test-driven development.
> Do you mind elaborating on your position?
>
> Hickey: I never spoke out ‘against’ TDD. What I have said is, life is
> short and there are only a finite number of hours in a day. So, we have to
> make choices about how we spend our time. If we spend it writing tests,
> that is time we are not spending doing something else. Each of us needs to
> assess how best to spend our time in order to maximize our results, both
> in quantity and quality. If people think that spending fifty percent of
> their time writing tests maximizes their results—okay for them. I’m sure
> that’s not true for me—I’d rather spend that time thinking about my
> problem. I’m certain that, for me, this produces better solutions, with
> fewer defects, than any other use of my time. A bad design with a complete
> test suite is still a bad design.
The text you mention refers about time to write extra test suites. I was
referring to simply have "make test" _run_ before being able to push commits.
Best regards,
Seb
--
Sebastien Vauban
Hi Achim, Achim Gratz <Stromeko@Nexgo.DE> writes: >> If anyone knows how to setup an automated tests framework for Org, >> feel free to go ahead, we will use it and monitor broken tests to >> see what's wrong in the code or in the tests or in the environment >> running the tests. > > We already have one, The test are not automatic, they are manually triggered, so we don't have an "automated tests framework" -- or am I misunderstanding what an automated test framework is? > what Nick and Sebastien are asking is not to push > commits that are known to not pass the tests. This I 100% agree with. I don't push commits that are known to me as not passing the tests :) >> Testing is a nice habit to have, but let's not make it a coercive >> pre-requisit before pushing patches. > > Why not? Any broken commits make automatic bisecting impossible and they > are a constant source of irritation for folks who forget to test their new > Org pulls before using or installing them. > >> My whole thinking here is well captured by Rich Hickey: > > The citation you gave doesn't even apply to the question at hand. It is > about writing tests, not using the tests you already have. It is about life being short and time being spent on testing vs coding. If you can come up with a pre-push hook that is clever enough to distinguish trivial-and-safe changes against those who need to be tested, please submit one. A trivial-and-safe change is either: - a change against non-code files; - a change in docstring. I don't think this is easy to do. Rich message wrt tests is: "Life is short, decide whether you want to spend it on testing or coding" -- so I think it's relevant here. I often have only 10 minutes at hand, make a few trivial changes, and push. For me, a mandatory pre-push hook running the test suite would be a useless burden for 50% of my commits. This would irritate me. We might agree to disagree on this. -- Bastien
Bastien <bzg@altern.org> writes:
> I often have only 10 minutes at hand, make a few trivial changes, and
> push. For me, a mandatory pre-push hook running the test suite would
> be a useless burden for 50% of my commits. This would irritate me.
orgmode.org could run a post-receive hook and report any failure to the
committer ? I don't know if orgmode.org can run the testsuite though.
--
N.
Hello Bastien, We can use travis-ci for automated tests, I just run tests for org-mode¹ on travis with emacs-snapshot. Magit has been setup recently to run tests for multiple emacs versions² (emacs23, emacs24 and snapshot). travis has a facility to send mails if a test fails. I see you have org-mode repo on your github account, all we need to do put a post recieve hook to mirror org-mode repo to github. we can adopt magit script³ to org-mode as it won't be synced to emacs trunk. what do you think? Thanks., ¹ https://travis-ci.org/yyr/org-mode/builds/5692183 ² https://travis-ci.org/magit/magit ³ https://github.com/magit/magit/blob/master/.travis.yml -- ఎందరో మహానుభావులు అందరికి వందనములు. YYR
Hello Yagnesh, Yagnesh Raghava Yakkala <hi@yagnesh.org> writes: > We can use travis-ci for automated tests, I just run tests for org-mode¹ on travis > with emacs-snapshot. Magit has been setup recently to run tests for multiple > emacs versions² (emacs23, emacs24 and snapshot). travis has a facility to send > mails if a test fails. Great! > I see you have org-mode repo on your github account, all we need to do put a > post recieve hook to mirror org-mode repo to github. > > we can adopt magit script³ to org-mode as it won't be synced to emacs > trunk. > > what do you think? I think this would be *fantastic*. I have no time for this at the moment, but anyone willing to help setting this up would be my hero. Really :) This is efficient and not intrusive. To give another argument about why I think a pre-push hook on the developer's side is wrong, imagine this scenario: - developer A pushes a commit, all tests pass - developer B works on latest HEAD assuming all tests pass - he wants to push his commits but the tests fail - he naturally thinks it's a problem with *his* changes - ... but it is not ... - M-x doctor RET This happened for real: recently some tests passed under Emacs <24.3 but failed under Emacs >=24.3. If we had a pre-push hook, I would not even be able to push the fix, I would have to deactive the hook first. Best, -- Bastien
Am 21.03.2013 14:41, schrieb Bastien: > The test are not automatic, they are manually triggered, so we don't > have an "automated tests framework" -- or am I misunderstanding what > an automated test framework is? What you probably have in mind is a continuous integration framework that triggers the test framework. > This I 100% agree with. I don't push commits that are known to me as > not passing the tests :) The cavalier attitude is not funny, smiley or not. > It is about life being short and time being spent on testing vs > coding. No, this is about doing it right or doing it twice. I have a fairly slow machine, but running "make test" or even "make test-dirty" with all tests enabled has not consumed an appreciable amount of my lifetime and probably never will. It has however saved me some pushes that had incomplete commits or some "obviously safe" last minute changes that turned out not to be or triggered some unexpected behaviour someplace else. I have also spent a good deal of time weeding out false positives from bisects that could have been automated if one could assume that each commit was always passing its tests. > If you can come up with a pre-push hook that is clever enough to > distinguish trivial-and-safe changes against those who need to be > tested, please submit one. A trivial-and-safe change is either: > > - a change against non-code files; > - a change in docstring. > > I don't think this is easy to do. It is also a waste of time and not necessary. Simply run the tests on each commit touching lisp/ and testing/ and reject the data from the push if any test fails. Not sure if Jason wants to put this on the server, though. Regards, -- Achim. (on the road :-)
There is no need to be unpleasant and to describe my attitude as "cavalier". Please see my reply to Yagnesh. It clearly describes a situation where automatically running tests with a pre-push hook would be a problem. -- Bastien
Hello Bastien, On Mar 22 2013, Bastien <bzg@altern.org> wrote: > I think this would be *fantastic*. I have no time for this at the > moment, but anyone willing to help setting travis is trivial enough to start using right away. I can make a patch for that. > This is efficient and not intrusive. > > To give another argument about why I think a pre-push hook on the > developer's side is wrong, imagine this scenario: > > - developer A pushes a commit, all tests pass > - developer B works on latest HEAD assuming all tests pass > - he wants to push his commits but the tests fail > - he naturally thinks it's a problem with *his* changes > - ... but it is not ... > - M-x doctor RET > > This happened for real: recently some tests passed under Emacs <24.3 > but failed under Emacs >=24.3. If we had a pre-push hook, I would not > even be able to push the fix, I would have to deactive the hook first. IIUC, it can't be done with travis. After little googling around, I found that such a conditional committing can be made if we have a continuous integration server (eg: jenkins) is installed and setup on org-mode server. How about using travis first.? Thanks., -- ఎందరో మహానుభావులు అందరికి వందనములు. YYR
Hi Yagnesh, Yagnesh Raghava Yakkala <hi@yagnesh.org> writes: > travis is trivial enough to start using right away. I can make a > patch for that. Thanks in advance for this. > IIUC, it can't be done with travis. After little googling around, I found that > such a conditional committing can be made if we have a continuous integration > server (eg: jenkins) is installed and setup on org-mode server. Yes. The limitation of the pre-push hook comes from the fact that various developers may have various testing environments, no one should be prevented from pushing by the fact that tests do not pass for someone else. > How about using travis first.? I fully agree. Let me know if there is anything I should do. Best, -- Bastien
[-- Attachment #1: Type: text/plain, Size: 878 bytes --] [CC'ed to Yann Hodique to acknowledge him] Hello Bastien, I am attaching a patch, please have a look. (especially change in org-test.el) It is directly copied from Magit (with one minor change). It uses Yann's virtualenv-emacs¹ python package which creates multiple emacs environments. > Let me know if there is anything I should do. 1) Mirror org-mode to your github repo (https://github.com/bzg/org-mode) with post receive hook. does org-mode.org server uses gitolite to manage git repos? if so https://github.com/miracle2k/gitolite-simple-mirror may be useful. (I use it on my server for mirroring) 2) Allow travis to access your org-mode repository (This must be very easy) - Create an account on https://travis-ci.org., just click "sign in with github" - activate tests org-mode repo at https://travis-ci.org/profile [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-.travis.yml-travis-ci-setup-adopted-from-magit-sourc.patch --] [-- Type: text/x-diff, Size: 1758 bytes --] From 2666c2ec2a85ca5d68f61b49081d82d1e5165a2d Mon Sep 17 00:00:00 2001 From: Yakkala Yagnesh Raghava <hi@yagnesh.org> Date: Fri, 22 Mar 2013 06:37:48 +0900 Subject: [PATCH] * .travis.yml: travis-ci setup (adopted from magit sources). * testing/org-test.el: remove requiring ert-x (needed for < emacs-24) Signed-off-by: Yakkala Yagnesh Raghava <hi@yagnesh.org> --- .travis.yml | 24 ++++++++++++++++++++++++ testing/org-test.el | 1 - 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..2334034 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,24 @@ +language: python +python: + - "2.7" +env: + matrix: + - EMACS=emacs + - EMACS=emacs24 + - EMACS=emacs-snapshot +before_install: + - if [ "$EMACS" = "emacs24" ]; then + sudo add-apt-repository -y ppa:cassou/emacs && + sudo apt-get update -qq && + sudo apt-get install -qq emacs24 emacs24-el; + fi + - if [ "$EMACS" = 'emacs-snapshot' ]; then + sudo add-apt-repository -y ppa:cassou/emacs && + sudo apt-get update -qq && + sudo apt-get install -qq + emacs-snapshot-el emacs-snapshot-gtk emacs-snapshot; + fi + - pip install virtualenv-emacs + - virtualenv_install_emacs --with-emacs=`which "$EMACS"` +script: + make test diff --git a/testing/org-test.el b/testing/org-test.el index 0c9ca58..5f8dd52 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -79,7 +79,6 @@ "Parent major mode from which special major modes should inherit." (setq buffer-read-only t))) (require 'ert) - (require 'ert-x) (when (file-exists-p (expand-file-name "jump/jump.el" org-test-dir)) (require 'jump) -- 1.8.1.5 [-- Attachment #3: Type: text/plain, Size: 479 bytes --] There are few tests failing² with emacs-23, don't know much about them. Finally, I haven't activated (yet) email option to send an email to author of the commit if a test fails. Thanks., ¹ https://github.com/sigma/virtualenv-emacs ² https://travis-ci.org/yyr/org-mode (direct link to log: https://api.travis-ci.org/jobs/5698864/log.txt?deansi=true) -- ఎందరో మహానుభావులు అందరికి వందనములు. YYR
Am 21.03.2013 21:41, schrieb Bastien:
[ ... ]
> Yes. The limitation of the pre-push hook comes from the fact that
> various developers may have various testing environments, no one
> should be prevented from pushing by the fact that tests do not pass
> for someone else.
[ ... ]
Hi,
just to ask about some more aspects:
- Typos often will break it from the beginning, so complete tests must not run all every time.
Will some testing at all is needed to detect the typos...
- org-mode has already a directory testing. It was not obvious for me how to make use of it.
Exists some docu wrt available tests?
- usually you know about the range of changes.
Limited tests checking certain features seem useful
Best,
Andreas
Am 21.03.2013 18:59, schrieb Bastien:
> Please see my reply to Yagnesh.
>
> It clearly describes a situation where automatically running tests
> with a pre-push hook would be a problem.
You keep mentioning a pre-push-hook to be run on the developers machine.
However, the test would run on the server and determine whether to
accept the commit into the repo or not. All your further arguments seem
to be based on that misunderstanding, so I won't comment on them.
As for Travis CI: a website that shows absolutely nothing when
JavaScript is turned off? No, thanks.
Regards,
--
Achim.
(on the road :-)
Achim Gratz <Stromeko@Nexgo.DE> writes: > Am 21.03.2013 18:59, schrieb Bastien: >> Please see my reply to Yagnesh. >> >> It clearly describes a situation where automatically running tests >> with a pre-push hook would be a problem. > > You keep mentioning a pre-push-hook to be run on the developers > machine. That's what have been proposed by Sébastien: Isn't it possible to put such in some sort of Git pre-commit hook (or pre-push hook), so that it gets automatically enforced? He was replying to Nick who proposed that developers always run make test before pushing. > However, the test would run on the server and determine whether to > accept the commit into the repo or not. I'm fine with a server-based solution. > All your further arguments seem to > be based on that misunderstanding, so I won't comment on them. Maybe the server-side solution was so obvious to you that you didn't realize we (Sébastien, Nick, me) were discussing something else. So I'll just assume you agree with my argument against the developer-side pre-push hook then, good. > As for Travis CI: a website that shows absolutely nothing when JavaScript > is turned off? No, thanks. Do you know any free (as-in-speech), easy-to-use alternative? -- Bastien
>>>>> "Yagnesh" == Yagnesh Raghava Yakkala <hi@yagnesh.org> writes: > [CC'ed to Yann Hodique to acknowledge him] > Hello Bastien, > I am attaching a patch, please have a look. (especially change in > org-test.el) > It is directly copied from Magit (with one minor change). It uses > Yann's virtualenv-emacs¹ python package which creates multiple > emacs environments. Hi, overall looks good to me. I'm glad to see that it might be useful for others :) > diff --git a/testing/org-test.el b/testing/org-test.el > index 0c9ca58..5f8dd52 100644 > --- a/testing/org-test.el > +++ b/testing/org-test.el > @@ -79,7 +79,6 @@ > "Parent major mode from which special major modes should inherit." > (setq buffer-read-only t))) > (require 'ert) > - (require 'ert-x) > (when (file-exists-p > (expand-file-name "jump/jump.el" org-test-dir)) > (require 'jump) About that, is it just because my code doesn't install a working ert-x.el for emacs 23 ? If so, it might be worth waiting for me to fix it (I'm afraid it's not there simply because I didn't need it :)). It would be sad to limit yourself just because of that. Assuming ert-x.el can be backported easily, I'll try and work on it in the coming days. Cheers, Yann. -- The greatest and most important problems of life cannot be solved. They can only be outgrown. -- SISTER JESSICA, private journal entry
Am 22.03.2013 08:36, schrieb Bastien:
> Do you know any free (as-in-speech), easy-to-use alternative?
Hudson. However, I don't think that a CI framework is what we need or
want. As I said, simply running the tests (preferrably with two
different versions of Emacs) should be enough for now. Unless we hear
from Jason if he thinks the server can take the extra load its a moot
point to discuss details, but I think this can be done in one of the Git
hooks (much like Worg triggers publishing).
Regards,
--
Achim.
(on the road :-)
Hello Yann, On Mar 22 2013, Yann Hodique <yann.hodique@gmail.com> wrote: > overall looks good to me. I'm glad to see that it might be useful for > others :) Thanks for the review. > About that, is it just because my code doesn't install a working > ert-x.el for emacs 23 ? Probably not. I thought ert-x.el is a split from the original ert.el (I may be wrong). > If so, it might be worth waiting for me to fix it (I'm afraid it's not > there simply because I didn't need it :)). It would be sad to limit > yourself just because of that. Assuming ert-x.el can be backported > easily, I'll try and work on it in the coming days. That would be useful. Thanks., -- ఎందరో మహానుభావులు అందరికి వందనములు. YYR
Hello Achim,
On Mar 22 2013, Achim Gratz <Stromeko@Nexgo.DE> wrote:
> As for Travis CI: a website that shows absolutely nothing when JavaScript is
> turned off? No, thanks.
Agreed although travis-ci source is under FSF approved license.
About hudson/jenkins (any other CI), If we have resources on the server, I
would say we should go for it. That will remove Bastien's concern of slowing
down development because of running tests by hand.
Thanks.,
--
ఎందరో మహానుభావులు అందరికి వందనములు.
YYR
Andreas Röhler <andreas.roehler@easy-emacs.de> wrote: > Am 21.03.2013 21:41, schrieb Bastien: > [ ... ] > > Yes. The limitation of the pre-push hook comes from the fact that > > various developers may have various testing environments, no one > > should be prevented from pushing by the fact that tests do not pass > > for someone else. > > [ ... ] > > Hi, > > just to ask about some more aspects: > > - Typos often will break it from the beginning, so complete tests must not run all every time. > Will some testing at all is needed to detect the typos... > Yes, run `make test'. Bastien already added a note to Worg, asking developers to run `make test' before pushing. That leaves it at the discretion of the developer, rather than having it run automatically from some hook which might have problems of its own. I believe that once developers (any that don't do so already) try it, they will find out the great benefits of doing so. And then both the problem and this discussion will die down :-) > - org-mode has already a directory testing. It was not obvious for me how to make use of it. > Exists some docu wrt available tests? > What we are talking about here is running existing test cases. For that, you just say `make test' to run the test suite. Creating new test cases and adding them to the test suite is a different matter of course. There is a useful README in the testing/ directory and an examples/ subdir, but you can also just pick one of the existing tests in the lisp/ subdir and read it. Perhaps not the best example, but http://article.gmane.org/gmane.emacs.orgmode/62908/match=ert shows a minimal .emacs with a test case added. It should at least clarify the mechanics of creating and running a new test case, stripped down to its bare essentials. > - usually you know about the range of changes. > Limited tests checking certain features seem useful > Not worth bothering about IMO: just run the whole test suite. Nick
Am 22.03.2013 13:55, schrieb Nick Dokos: > Andreas Röhler <andreas.roehler@easy-emacs.de> wrote: [ ... ] > Not worth bothering about IMO: just run the whole test suite. > > Nick > Hi Nick, thanks a lot for your explanation and patience. Still digging in... :) Andreas
On Fri, Mar 22, 2013 at 10:40 AM, Yagnesh Raghava Yakkala
<hi@yagnesh.org> wrote:
>
> On Mar 22 2013, Yann Hodique <yann.hodique@gmail.com> wrote:
>
> > If so, it might be worth waiting for me to fix it (I'm afraid it's not
> > there simply because I didn't need it :)). It would be sad to limit
> > yourself just because of that. Assuming ert-x.el can be backported
> > easily, I'll try and work on it in the coming days.
>
> That would be useful.
It should now be ok, with virtualenv-emacs 0.1.5 (that grabs an
ert-x.el from marmalade).
Cheers,
Yann.
Hi, Achim Gratz <Stromeko@Nexgo.DE> writes: > Hudson. However, I don't think that a CI framework is what we need or > want. As I said, simply running the tests (preferrably with two different > versions of Emacs) should be enough for now. Unless we hear from Jason if > he thinks the server can take the extra load its a moot point to discuss > details, but I think this can be done in one of the Git hooks (much like > Worg triggers publishing). Yagnesh Raghava Yakkala <hi@yagnesh.org> writes: > About hudson/jenkins (any other CI), If we have resources on the server, I > would say we should go for it. That will remove Bastien's concern of slowing > down development because of running tests by hand. I'm copying Jason -- the idea is to run tests on the servers via a Git hook, the same way that a Git hook publishes Worg. If the tests fail, the committer would get a warning and the commit would be discarded. Jason, do you think it's feasible? Enough? I guess hudson/travis is really too much for our needs. Thanks, -- Bastien