* Re: How you can help
@ 2008-10-23 12:04 Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
` (2 more replies)
0 siblings, 3 replies; 35+ messages in thread
From: Ben Alexander @ 2008-10-23 12:04 UTC (permalink / raw)
To: sebastian_rose; +Cc: emacs-orgmode
Sebastian Rose wrote:
> 5. I also think of little packages for testing parts of org.
I'm curious if you or someone else has any ideas for writing automated
tests for org-mode. I haven't the foggiest idea how someone would
write a test for the parts of org that control what is displayed on
the screen. I mean, when the bug is 'it doesn't look right' how can
you tell?
Perhaps the git repository should have a small collection of small org-
mode files that reproduce certain bugs? If there were some examples
of how to create such a test, then perhaps bug reporters would find it
much easier to create them.
I do see some confusing issues due to different configuration files.
So creating a test file might involve making sure org-mode doesn't
read any configuration (how do you do that?) and possible asking org-
mode to extract all the configuration variables it has right now and
dump them into a test file (...and how do you do that?)
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: How you can help 2008-10-23 12:04 How you can help Ben Alexander @ 2008-10-23 13:43 ` Bernt Hansen 2008-10-23 15:04 ` Sebastian Rose 2008-10-23 17:01 ` Jason F. McBrayer 2008-10-23 14:20 ` Sebastian Rose 2008-10-23 15:08 ` Sebastian Rose 2 siblings, 2 replies; 35+ messages in thread From: Bernt Hansen @ 2008-10-23 13:43 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Ben Alexander <bva@alexanderonline.org> writes: > Sebastian Rose wrote: >> 5. I also think of little packages for testing parts of org. > > I'm curious if you or someone else has any ideas for writing automated > tests for org-mode. I haven't the foggiest idea how someone would > write a test for the parts of org that control what is displayed on > the screen. I mean, when the bug is 'it doesn't look right' how can > you tell? > > Perhaps the git repository should have a small collection of small > org- > mode files that reproduce certain bugs? If there were some examples > of how to create such a test, then perhaps bug reporters would find it > much easier to create them. > > I do see some confusing issues due to different configuration files. > So creating a test file might involve making sure org-mode doesn't > read any configuration (how do you do that?) and possible asking org- > mode to extract all the configuration variables it has right now and > dump them into a test file (...and how do you do that?) Running a minimal emacs should suppress custom config files: emacs -q -l yourtest.el Some kind of regression testing framework would be awesome. Org-mode is large enough that this is almost a necessity to keep things stable and bug-free. Maybe something can be put together from the git testing framework and use of emacs -batch to process test org files and verify the output is as expected (with diff or some other tool). -Bernt ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 13:43 ` Bernt Hansen @ 2008-10-23 15:04 ` Sebastian Rose 2008-10-23 15:49 ` Richard Riley 2008-10-23 16:19 ` Bernt Hansen 2008-10-23 17:01 ` Jason F. McBrayer 1 sibling, 2 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 15:04 UTC (permalink / raw) To: Bernt Hansen; +Cc: Ben Alexander, emacs-orgmode Bernt Hansen <bernt@norang.ca> writes: > Running a minimal emacs should suppress custom config files: > emacs -q -l yourtest.el Added this one to the Clippboard section on new org-tests/index.org in Worg.git. (this section will be temporary...) > Some kind of regression testing framework would be awesome. Org-mode is > large enough that this is almost a necessity to keep things stable and > bug-free. It's big and feature-RICH. > Maybe something can be put together from the git testing framework and > use of emacs -batch to process test org files and verify the output is > as expected (with diff or some other tool). Hey, diff is a good idea!! I didn't take the verification of the output into account yet :-) I just pushed a change of Worgs start page, and added a directory 'org-tests'. I've placed an index.org there, which now is just a collection of ideas (I'm on my day job, so I can't really work on it now). Don't know how often the git repo is published. Bernt and Ben, are you 'worgers' allready? Do you think it makes sense to add snippets and ideas to the new page in Worg? I think while the list great to exchange ideas, it's good to have a place, where all those ideas are destilled to one-liners. Cheers, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 15:04 ` Sebastian Rose @ 2008-10-23 15:49 ` Richard Riley 2008-10-23 16:22 ` Ben Alexander ` (2 more replies) 2008-10-23 16:19 ` Bernt Hansen 1 sibling, 3 replies; 35+ messages in thread From: Richard Riley @ 2008-10-23 15:49 UTC (permalink / raw) To: Sebastian Rose; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander Sebastian Rose <sebastian_rose@gmx.de> writes: > Bernt Hansen <bernt@norang.ca> writes: >> Running a minimal emacs should suppress custom config files: >> emacs -q -l yourtest.el > > Added this one to the Clippboard section on new org-tests/index.org in > Worg.git. (this section will be temporary...) Something like the above should only be a link (at most) to the emacs manual. Reproducing standard info is bad in the long run in case things in the base product (emacs) change for example. > >> Some kind of regression testing framework would be awesome. Org-mode is >> large enough that this is almost a necessity to keep things stable and >> bug-free. > > It's big and feature-RICH. The nature of OSS means that the community using the product keep it stable and bug free. I dont think the efforts to produce meaningful regression tests would be beneficial in an ever morphing product like org-mode. Clearly my humble opinion on that one :-; > >> Maybe something can be put together from the git testing framework and >> use of emacs -batch to process test org files and verify the output is >> as expected (with diff or some other tool). > > > Hey, diff is a good idea!! > > I didn't take the verification of the output into account yet :-) > > I just pushed a change of Worgs start page, and added a directory > 'org-tests'. I've placed an index.org there, which now is just a > collection of ideas (I'm on my day job, so I can't really work on it > now). > > Don't know how often the git repo is published. > Bernt and Ben, are you 'worgers' allready? > > Do you think it makes sense to add snippets and ideas to the new page in > Worg? I think while the list great to exchange ideas, it's good to have > a place, where all those ideas are destilled to one-liners. I must say I am dubious about this. It means, for the tests to be meaningful, that the output must be a fixed format in base org. I doubt this will ever be the case. The presentation will fluctuate while the core information (dates, schedules periods etc) will remain pretty much constant. The majority of bugs that I see are often down to people misusing or using things in the base which are not fully explored. No amount of regression testing can cover things like that unless the regression tests include everyones customisations. Do I think regression testing is important? Yes - in certain environments. But every time Carsten, you, myself or anyone else fires up org-mode we are already doing just that. -- Inventor: A person who makes an ingenious arrangement of wheels, levers and springs, and believes it civilization. ~Ambrose Bierce, The Devil's Dictionary ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 15:49 ` Richard Riley @ 2008-10-23 16:22 ` Ben Alexander 2008-10-23 17:02 ` Sebastian Rose 2008-10-23 19:13 ` Avdi Grimm 2 siblings, 0 replies; 35+ messages in thread From: Ben Alexander @ 2008-10-23 16:22 UTC (permalink / raw) To: Richard Riley; +Cc: emacs-orgmode Org-Mode On 2008-Oct-23, at 16:49, Richard Riley wrote: > Sebastian Rose <sebastian_rose@gmx.de> writes: > >> Bernt Hansen <bernt@norang.ca> writes: >>> Running a minimal emacs should suppress custom config files: >>> emacs -q -l yourtest.el >> >> Added this one to the Clippboard section on new org-tests/index.org >> in >> Worg.git. (this section will be temporary...) > > Something like the above should only be a link (at most) to the emacs > manual. Reproducing standard info is bad in the long run in case > things > in the base product (emacs) change for example. > >> >>> Some kind of regression testing framework would be awesome. Org- >>> mode is >>> large enough that this is almost a necessity to keep things stable >>> and >>> bug-free. >> >> It's big and feature-RICH. > > The nature of OSS means that the community using the product keep it > stable and bug free. I dont think the efforts to produce meaningful > regression tests would be beneficial in an ever morphing product like > org-mode. Clearly my humble opinion on that one :-; > I'm a bit clue-free on what the words mean. I think of regression tests as ways of defining the way "Things Should Work(TM)" so if someone commits as fix to org-mode that works fine on Debian, but breaks on Windows, there's a way for someone on Windows to easily (brainlessly) run the test and report back to the developer. I don't think of org-mode as 'ever morphing'. There are some features that are stable, and should remain so. A regression test would be valuable, presumably, for developers working on a new feature (don't screw up important existing functionality!) > > I must say I am dubious about this. It means, for the tests to be > meaningful, that the output must be a fixed format in base org. I > doubt > this will ever be the case. The presentation will fluctuate while the > core information (dates, schedules periods etc) will remain pretty > much > constant. > I agree that the presentation is fluid and it was hard for me to imagine how to test it in an automated way. But I disagree (I think) in that having a test on the presentation is extremely powerful. I'm thinking that one example of how to write such a test could be copied when someone is trying to explain what they want to happen, and how it is not happening right now. Perhaps such a test would have a very limited useful life. I'd like to think that it could live and be useful for a very long time (that's why thinking about configuration/user customization issues in a test is important). But even if the test is only useful until the bug is fixed (or the feature implemented) , so be it. > The majority of bugs that I see are often down to people misusing or > using things in the base which are not fully explored. No amount of > regression testing can cover things like that unless the regression > tests include everyones customisations. > Not everyone's customizations, just every customization org-mode has. I mean, if the feature is there, it should work. If the feature is important, it should work in the same way on every platform. The trick -- and I hope it's either solved or easy -- is exactly how to show that "the feature works the same way as it should, right now, on this platform" If I understand you properly, you're just saying that the user customization creates so much variability in the output, there isn't anyway you could capture all the possible presentations in a single test. Well, yeah, so user customizations need to be left out of the test case. Except that in the scenario I'm thinking of where someone (a mere mortal using org-mode, with limited lisp knowledge) is trying to report a bug, user customization will be critical. My first thought was asking org-mode to dump all the customization variables it has into a buffer (as buffer-local variables maybe?) would help someone else recreate the bug. (By starting org-mode with NO customizations, except what's found in the test case) Or are you suggesting that the source of presentation variation is due mostly to OS, Emacs version, or something else beyond the scope of our tests, so they couldn't be replicated on another machine? > Do I think regression testing is important? Yes - in certain > environments. But every time Carsten, you, myself or anyone else fires > up org-mode we are already doing just that. > > > -- > Inventor: A person who makes an ingenious arrangement of wheels, > levers and springs, and believes it civilization. ~Ambrose Bierce, > The Devil's Dictionary ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 15:49 ` Richard Riley 2008-10-23 16:22 ` Ben Alexander @ 2008-10-23 17:02 ` Sebastian Rose 2008-10-24 12:13 ` Richard Riley 2008-10-23 19:13 ` Avdi Grimm 2 siblings, 1 reply; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 17:02 UTC (permalink / raw) To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander Hi Richard, Richard Riley <rileyrgdev@googlemail.com> writes: >> Added this one to the Clippboard section on new org-tests/index.org in >> Worg.git. (this section will be temporary...) > > Something like the above should only be a link (at most) to the emacs > manual. Reproducing standard info is bad in the long run in case things > in the base product (emacs) change for example. On the long run, yes. I just added this section, to start the page. And since it's just STARTing... this was in the first or second reaction that came in. Feel free to edit ('go wild' ;-))! I wouldn't bother to have several pages like this one will be (it's supposed to be an index on the long run), each covering another way of testing. I just meant to take _ACT_. Scepticism is a good thing, as long as it doesn't stop you from doing. >> >>> Some kind of regression testing framework would be awesome. Org-mode is >>> large enough that this is almost a necessity to keep things stable and >>> bug-free. >> >> It's big and feature-RICH. > > The nature of OSS means that the community using the product keep it > stable and bug free. I dont think the efforts to produce meaningful > regression tests would be beneficial in an ever morphing product like > org-mode. Clearly my humble opinion on that one :-; We do test our software by using it. But the bug in the HTML-exporter that Carsten has fixed two days ago, was introduced in early September and would be in 6.10, which is supposed to be in the emacs 23 release. A very simple test plan would have revealed it. >>> Maybe something can be put together from the git testing framework and >>> use of emacs -batch to process test org files and verify the output is >>> as expected (with diff or some other tool). >> >> >> Hey, diff is a good idea!! >> >> I didn't take the verification of the output into account yet :-) >> >> I just pushed a change of Worgs start page, and added a directory >> 'org-tests'. I've placed an index.org there, which now is just a >> collection of ideas (I'm on my day job, so I can't really work on it >> now). >> >> Don't know how often the git repo is published. >> Bernt and Ben, are you 'worgers' allready? >> >> Do you think it makes sense to add snippets and ideas to the new page in >> Worg? I think while the list great to exchange ideas, it's good to have >> a place, where all those ideas are destilled to one-liners. > > I must say I am dubious about this. It means, for the tests to be > meaningful, that the output must be a fixed format in base org. Why? If the test bails out 'ERROR', I will have to look for the reason. If the format changed in a legal way => adjust the test. > I doubt > this will ever be the case. The presentation will fluctuate while the > core information (dates, schedules periods etc) will remain pretty much > constant. > > The majority of bugs that I see are often down to people misusing or > using things in the base which are not fully explored. No amount of > regression testing can cover things like that unless the regression > tests include everyones customisations. Yes, because Carsten add features en masse :-) I see the testing differently. In the first place we need THINK of testing. New Org-revision out? Ahh, OK, I have to the HTML-exports, I want to be working. To do this, I need several different setups, several different data directories (Org-files) and an easy way to test, that doesn't eat my time and gives a result. The quality may vary, but ERRORs will be detected for shure. Not one or mounths later - immediately. If someone installs emacs 23, tries to export to HTML and gets an error.... > Do I think regression testing is important? Yes - in certain > environments. But every time Carsten, you, myself or anyone else fires > up org-mode we are already doing just that. Yes, but we can do better, easier and more complete. -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 17:02 ` Sebastian Rose @ 2008-10-24 12:13 ` Richard Riley 2008-10-24 15:39 ` Sebastian Rose 2008-10-24 16:27 ` Manish 0 siblings, 2 replies; 35+ messages in thread From: Richard Riley @ 2008-10-24 12:13 UTC (permalink / raw) To: Sebastian Rose; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander Sebastian Rose <sebastian_rose@gmx.de> writes: > Hi Richard, > > > Richard Riley <rileyrgdev@googlemail.com> writes: >>> Added this one to the Clippboard section on new org-tests/index.org in >>> Worg.git. (this section will be temporary...) >> >> Something like the above should only be a link (at most) to the emacs >> manual. Reproducing standard info is bad in the long run in case things >> in the base product (emacs) change for example. > > > On the long run, yes. I just added this section, to start the page. And > since it's just STARTing... this was in the first or second reaction > that came in. Feel free to edit ('go wild' ;-))! > > I wouldn't bother to have several pages like this one will be (it's > supposed to be an index on the long run), each covering another way of > testing. > > I just meant to take _ACT_. Scepticism is a good thing, as long as it > doesn't stop you from doing. Of course not. > > >>> >>>> Some kind of regression testing framework would be awesome. Org-mode is >>>> large enough that this is almost a necessity to keep things stable and >>>> bug-free. >>> >>> It's big and feature-RICH. >> >> The nature of OSS means that the community using the product keep it >> stable and bug free. I dont think the efforts to produce meaningful >> regression tests would be beneficial in an ever morphing product like >> org-mode. Clearly my humble opinion on that one :-; > > > We do test our software by using it. But the bug in the HTML-exporter > that Carsten has fixed two days ago, was introduced in early September > and would be in 6.10, which is supposed to be in the emacs 23 release. > > A very simple test plan would have revealed it. The question is : define simple. I am not against test plans, but the nature of such means a process in place where such test plans are carried out. Carsten knows his SW inside out and would find it hard to apply test cases every time a bug is fixed for time alone. Bugs happen in SW and the nature (excuse the repetition) of OSS is that we, the users, are kind of the testers and users. >>>> Maybe something can be put together from the git testing framework and >>>> use of emacs -batch to process test org files and verify the output is >>>> as expected (with diff or some other tool). >>> >>> >>> Hey, diff is a good idea!! >>> >>> I didn't take the verification of the output into account yet :-) >>> >>> I just pushed a change of Worgs start page, and added a directory >>> 'org-tests'. I've placed an index.org there, which now is just a >>> collection of ideas (I'm on my day job, so I can't really work on it >>> now). >>> >>> Don't know how often the git repo is published. >>> Bernt and Ben, are you 'worgers' allready? >>> >>> Do you think it makes sense to add snippets and ideas to the new page in >>> Worg? I think while the list great to exchange ideas, it's good to have >>> a place, where all those ideas are destilled to one-liners. >> >> I must say I am dubious about this. It means, for the tests to be >> meaningful, that the output must be a fixed format in base org. > > > Why? If the test bails out 'ERROR', I will have to look for the > reason. If the format changed in a legal way => adjust the test. What is an error? And for whom? That the categories are one more tab to the right? I know I am probably sounding a bit of a killjoy here but having been involved in closed source SW development with dedicated QA and testing teams it was hard enough then. Maybe I am just being a bit of an old curmudgeon here :-( > > >> I doubt >> this will ever be the case. The presentation will fluctuate while the >> core information (dates, schedules periods etc) will remain pretty much >> constant. >> >> The majority of bugs that I see are often down to people misusing or >> using things in the base which are not fully explored. No amount of >> regression testing can cover things like that unless the regression >> tests include everyones customisations. > > Yes, because Carsten add features en masse :-) > I see the testing differently. In the first place we need THINK of > testing. I think we all do. And the main contributors do too. But it is unrealistic to expect a full regression test for each and every release unless there is someone willing to step up to the mark and take the time to perform them. I personally feel that enough take the "cutting edge" and report back before the "official" cuts. We are all beat testers. Its the nature of being an emacs and OSS user in the first place :-; > > New Org-revision out? > Ahh, OK, I have to the HTML-exports, I want to be working. > > To do this, I need several different setups, several different data > directories (Org-files) and an easy way to test, that doesn't eat my > time and gives a result. The quality may vary, but ERRORs will be > detected for shure. > > Not one or mounths later - immediately. If you don't find it immediately then how often do you use the feature? Now I know I probably sound negative. I assure I am not. Maybe the earlier word "sceptic" is more applicable .. > > If someone installs emacs 23, tries to export to HTML and gets an > error.... I have been there too. In fact you found a bug that I left hanging for ages because I found a work around ( dont publish from anywhere other than the root of the project) - so thanks for your perseverance on that. I think the best thing people can do is keep a stable version and also test the CVS (oops) GIT version too very now and again. > > >> Do I think regression testing is important? Yes - in certain >> environments. But every time Carsten, you, myself or anyone else fires >> up org-mode we are already doing just that. > > Yes, but we can do better, easier and more complete. Possibly. I look forward to seeing proposed solutions and would gladly lend some time to applying them. -- God never made his work for man to mend. ~John Dryden ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-24 12:13 ` Richard Riley @ 2008-10-24 15:39 ` Sebastian Rose 2008-10-24 16:27 ` Manish 1 sibling, 0 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-24 15:39 UTC (permalink / raw) To: emacs-orgmode Hi Richard, Richard Riley wrote: > Sebastian Rose <sebastian_rose@gmx.de> writes: > >> Hi Richard, >> >> >> Richard Riley <rileyrgdev@googlemail.com> writes: >>>> Added this one to the Clippboard section on new org-tests/index.org in >>>> Worg.git. (this section will be temporary...) >>> Something like the above should only be a link (at most) to the emacs >>> manual. Reproducing standard info is bad in the long run in case things >>> in the base product (emacs) change for example. >> >> On the long run, yes. I just added this section, to start the page. And >> since it's just STARTing... this was in the first or second reaction >> that came in. Feel free to edit ('go wild' ;-))! >> >> I wouldn't bother to have several pages like this one will be (it's >> supposed to be an index on the long run), each covering another way of >> testing. >> >> I just meant to take _ACT_. Scepticism is a good thing, as long as it >> doesn't stop you from doing. > > Of course not. Glad you replied. While I speek a little English, I'm not alwas shure how it _feels_ for native speakers. For example this one about scepticism. >> >>>>> Some kind of regression testing framework would be awesome. Org-mode is >>>>> large enough that this is almost a necessity to keep things stable and >>>>> bug-free. >>>> It's big and feature-RICH. >>> The nature of OSS means that the community using the product keep it >>> stable and bug free. I dont think the efforts to produce meaningful >>> regression tests would be beneficial in an ever morphing product like >>> org-mode. Clearly my humble opinion on that one :-; >> >> We do test our software by using it. But the bug in the HTML-exporter >> that Carsten has fixed two days ago, was introduced in early September >> and would be in 6.10, which is supposed to be in the emacs 23 release. >> >> A very simple test plan would have revealed it. > > The question is : define simple. I am not against test plans, but the > nature of such means a process in place where such test plans are > carried out. Carsten knows his SW inside out and would find it hard to > apply test cases every time a bug is fixed for time alone. Bugs happen > in SW and the nature (excuse the repetition) of OSS is that we, the > users, are kind of the testers and users. OK, simplicity. That's an important point I'd like to make clear. I _want_ simplicity in two ways: a) This here is _not_ meant to make Carsten work _more_, but to _help_ Carsten and all contributors. We must keep that in mind, while searching for the right way to test. Not Carsten, but _we_ collect the bugs, _we_ write the tests (whithout interfering with Carstens code), _we_ setup the testenvironment, and _we_ finally _do_ the testing. b) I _want_ tests, that everyone can do. Nobody knows, who of us is still here in a year or two. I like the idea of automated testing, but I'd also like to get every org-user to test. Imagine to have org-test.el in contrib (split into modules maybe). (setq org/test-html-export-base-dir "~/org-test/data/") (setq org/test-html-export-publising-dir "~/public_html/org-test/") M-x org/test-test-html-export The whole thing automatically executed once a week/day/mounth, after every git-pull, <SCHEDULED +1d>, sending a backtrace on failure. Once tests exist, we will find out how the interaction with Carsten works best / where neccessary. I know it will be hard to adjust all those tests to 'movin target' called Org. >>>>> Maybe something can be put together from the git testing framework and >>>>> use of emacs -batch to process test org files and verify the output is >>>>> as expected (with diff or some other tool). >>>> >>>> Hey, diff is a good idea!! >>>> >>>> I didn't take the verification of the output into account yet :-) >>>> >>>> I just pushed a change of Worgs start page, and added a directory >>>> 'org-tests'. I've placed an index.org there, which now is just a >>>> collection of ideas (I'm on my day job, so I can't really work on it >>>> now). >>>> >>>> Don't know how often the git repo is published. >>>> Bernt and Ben, are you 'worgers' allready? >>>> >>>> Do you think it makes sense to add snippets and ideas to the new page in >>>> Worg? I think while the list great to exchange ideas, it's good to have >>>> a place, where all those ideas are destilled to one-liners. >>> I must say I am dubious about this. It means, for the tests to be >>> meaningful, that the output must be a fixed format in base org. >> >> Why? If the test bails out 'ERROR', I will have to look for the >> reason. If the format changed in a legal way => adjust the test. > > > What is an error? And for whom? That the categories are one more tab to > the right? In this case: if I have a project defined in org-publish-projects-alist, and :base-dir contains a file /foo/bar.org, and, when publishing, this file goes to /bar.html (instead of /foo/bar.html), it is definitively an ERROR. At least in my eyes :-) > I know I am probably sounding a bit of a killjoy here but > having been involved in closed source SW development with dedicated QA > and testing teams it was hard enough then. Maybe I am just being a bit > of an old curmudgeon here :-( It will _never_ be that hard, since we are who we are :-D >> >>> I doubt >>> this will ever be the case. The presentation will fluctuate while the >>> core information (dates, schedules periods etc) will remain pretty much >>> constant. >>> >>> The majority of bugs that I see are often down to people misusing or >>> using things in the base which are not fully explored. No amount of >>> regression testing can cover things like that unless the regression >>> tests include everyones customisations. Yes, the customizations: M-x org/test-test-html-export That's why I wrote (in another mai), we shouldn't urge users to use a bug tracker. We would just fill the bug tracker with lots of useless information (from the bug trackers point of view). I love the bug tracking on the list the way it is. >> Yes, because Carsten add features en masse :-) >> I see the testing differently. In the first place we need THINK of >> testing. > > I think we all do. And the main contributors do too. I didn't. I didn't even recognize the bug for a while, until I found, that one of the published org files had the wrong contents. A simple test would have revealed the bug. > But it is > unrealistic to expect a full regression test for each and every release > unless there is someone willing to step up to the mark and take the time > to perform them. I personally feel that enough take the "cutting edge" > and report back before the "official" cuts. We are all beat testers. Its > the nature of being an emacs and OSS user in the first place :-; There will always remain parts of the system untested. It'd be funny to calculate all possible combinations of emacs configuration, emacs versions and OS version Org-mode is supposed to run on, org mode configurations and invariants. Seems like a big big number :-D But the number of invariants itself is not endless! Differences between OSs are, from an elisp view, minor. Not all the possible combinations of configuration values are sensible. Those configs could be added as needed (or simply left up to the users). Shure, tests are just one piece of the puzzle. But I'm so happy to see this thread grow, see new names here, see people editing Worg, learn something myself - that makes me hope, we can make the best out of it. >> New Org-revision out? >> Ahh, OK, I have to the HTML-exports, I want to be working. >> >> To do this, I need several different setups, several different data >> directories (Org-files) and an easy way to test, that doesn't eat my >> time and gives a result. The quality may vary, but ERRORs will be >> detected for shure. >> >> Not one or mounths later - immediately. > > If you don't find it immediately then how often do you use the feature? > Now I know I probably sound negative. I assure I am not. Maybe the > earlier word "sceptic" is more applicable .. I use that feature nearly every day. But the project I export is quite big and I happend to not visit one of the affected files - hm, maybe I did, but the contents where not updated (the exporter doesn't delete files in :publishing-dir - which is OK btw - a test should do that). When I finally found something was wrong, I did an rm -r ~/public_html/or-notes rm ~/.org-timestamps/* and exported again, I found it was a desaster. Many files missing or in wrong directories (I think about 20%). HTML export worked like a charme 6 mounth for me. Suddenly it doesn't and none of us even noticed. A test for this should be easy to write. >> If someone installs emacs 23, tries to export to HTML and gets an >> error.... > > I have been there too. In fact you found a bug that I left hanging for > ages because I found a work around ( dont publish from anywhere other > than the root of the project) - so thanks for your perseverance on that. > > I think the best thing people can do is keep a stable version and also > test the CVS (oops) GIT version too very now and again. > >> >>> Do I think regression testing is important? Yes - in certain >>> environments. But every time Carsten, you, myself or anyone else fires >>> up org-mode we are already doing just that. >> Yes, but we can do better, easier and more complete. > > Possibly. I look forward to seeing proposed solutions and would gladly > lend some time to applying them. Ahh, Great! Thanks! Sebastian ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-24 12:13 ` Richard Riley 2008-10-24 15:39 ` Sebastian Rose @ 2008-10-24 16:27 ` Manish 2008-10-24 18:41 ` Avdi Grimm 1 sibling, 1 reply; 35+ messages in thread From: Manish @ 2008-10-24 16:27 UTC (permalink / raw) To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander On Fri, Oct 24, 2008 at 5:43 PM, Richard Riley wrote: [snip] > I think the best thing people can do is keep a stable version and also > test the CVS (oops) GIT version too very now and again. +1 That's exactly what I try to do. I update almost every other day, go through (using tig) the diffs of code, changelog, documentation, comments.. and see if I can try out and learn something. I also try and report if something doesn't work right for me with whatever details I can provide. It really doesn't take too long. I definitely need to learn more about generating backtraces and debuggers so I could do a better job at providing right kind and level of details. With the kind of people we have on the list, we already have a high quallity Volunteer Powered Massively Parallel Testing System - VPMPTS (tm) in place. :) Let me take a step back and think aloud why we need a bug-tracking and testing system (if only to clarify my understanding) and who/when does it help. Following scenarios come to mind (and how they can be handled best (again only my limited understanding)): 1. Someone sees something odd, assumes it's a bug and wants/needs to report it. Report it to list, people will help out if it's a configuration or understanding issue or confirm if it's a repeatble issue. 2. A bug report needs to be confirmed. VPMPTS goes to work. :) 3. A bug needs to be communicated to the developers. Developers look at the bug confirmation reports on the list and decide whether to accept it as a bug or improve our collective understanding. 4. New functionality needs to be tested for regression. VPMPTS to rescue! 5. Bug reporter needs to know when the bug is fixed. Read the list! (Possibly track git repo as well.) 6. Developers needs to sync up about who is working on a specific bug. Seems like bug tracker will be useful. This is for developers to comment upon but bug acknowledging developer can be safely assumed to know the reasons behind the bug and assumed to own it. I confess I do not have programming or testing background so I may be completely out of whack here. IMHO, we should go in the direction of taking on the overhead of maintaining additional infrastructure only when it's inevitable or really valuable. Let me take another step back and recall Carsten't first email on this thread; he asked for help in 5 specific areas: 1. Providing sufficient details when reporting bugs. 2. Reproducing reported bugs. 3. Sub-system adoption. 4. Answering emails on list. 5. Adding to Worg in form of FAQ, tutorials, and hacks. I think #1 needs a tutorial (at least for me.) We already do #2 and #4. #5 needs more work to cover as many features as possible (mea culpa.) #3 is up to elisp gurus here. I believe Carsten thought long and hard before he came up with this list of very specific areas which, IMHO, covers most (if not all) of the scenarios I conjured up above. >> >>> Do I think regression testing is important? Yes - in certain >>> environments. But every time Carsten, you, myself or anyone else fires >>> up org-mode we are already doing just that. >> >> Yes, but we can do better, easier and more complete. > > Possibly. I look forward to seeing proposed solutions and would gladly > lend some time to applying them. Ditto. Please do not assume that I am against any change in our systems and processes. I am just trying to see the system working in my mind's eye. -- Manish ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-24 16:27 ` Manish @ 2008-10-24 18:41 ` Avdi Grimm 0 siblings, 0 replies; 35+ messages in thread From: Avdi Grimm @ 2008-10-24 18:41 UTC (permalink / raw) To: Manish; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander On Fri, Oct 24, 2008 at 12:27 PM, Manish <mailtomanish.sharma@gmail.com> wrote: > Let me take a step back and think aloud why we need a bug-tracking and > testing system (if only to clarify my understanding) and who/when does > it help. Following scenarios come to mind (and how they can be > handled best (again only my limited understanding)): First of all, bug tracking are completely separate beasts, and should be considered separately. You can have either one without the other. Second: I don't care as much about bug tracking, except for the following case: if there is a bug tracker and I'm feeling bored and want to contribute a little, I'll check it for something that I think I can fix. If there is no such list, I won't do this, and I'll contribute less. I do *not* follow the mailing list closely enough to pick up bugs from there except by sheer chance, and this is not going to change. I really don't care where or how the list is kept, so long as there is a list. Third, regarding testing. I'm a coder, I know some ELisp, I *love* org-mode, and I would like to contribute. That said, I am completely uninterested in doing manual QA for Org. I am not going to go clicking around just to look for broken behavior. I have neither the time nor the interest. However, if there is an automated test suite I can promise you that a) I will run it before submitting any change I make; and b) I will add new tests to cover any functionality I add. However, you should take all of this with a grain of salt because I haven't submitted any code at all yet. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 15:49 ` Richard Riley 2008-10-23 16:22 ` Ben Alexander 2008-10-23 17:02 ` Sebastian Rose @ 2008-10-23 19:13 ` Avdi Grimm 2008-10-24 12:19 ` Richard Riley 2 siblings, 1 reply; 35+ messages in thread From: Avdi Grimm @ 2008-10-23 19:13 UTC (permalink / raw) To: Richard Riley; +Cc: Bernt Hansen, emacs-orgmode, Ben Alexander On Thu, Oct 23, 2008 at 11:49 AM, Richard Riley <rileyrgdev@googlemail.com> wrote: > The nature of OSS means that the community using the product keep it > stable and bug free. I dont think the efforts to produce meaningful > regression tests would be beneficial in an ever morphing product like > org-mode. Clearly my humble opinion on that one :-; I don't know of a single open-source project with more than one or two developers which doesn't have a suite of regression tests. In fact, I know of a number of OSS projects which *will not* accept a patch unless tests accompany it. Once a codebase grows beyond a one-person pet project, tests become increasingly necessary for maintainability. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 19:13 ` Avdi Grimm @ 2008-10-24 12:19 ` Richard Riley 0 siblings, 0 replies; 35+ messages in thread From: Richard Riley @ 2008-10-24 12:19 UTC (permalink / raw) To: Avdi Grimm; +Cc: Bernt Hansen, emacs-orgmode, Richard Riley, Ben Alexander "Avdi Grimm" <avdi@avdi.org> writes: > On Thu, Oct 23, 2008 at 11:49 AM, Richard Riley > <rileyrgdev@googlemail.com> wrote: >> The nature of OSS means that the community using the product keep it >> stable and bug free. I dont think the efforts to produce meaningful >> regression tests would be beneficial in an ever morphing product like >> org-mode. Clearly my humble opinion on that one :-; > > > I don't know of a single open-source project with more than one or two > developers which doesn't have a suite of regression tests. In fact, I > know of a number of OSS projects which *will not* accept a patch > unless tests accompany it. Once a codebase grows beyond a one-person > pet project, tests become increasingly necessary for maintainability. Then I have to disagree. There are many OSS projects with no regression tests. Is this a good thing? Not necessarily. But the nature of emacs, elisp and the customization features make it, for me, a slightly different kettle of fish. In fact the nature of this thread and the nature of most emacs customizations suggests to me that the regression tests are merely using the SW and seeing if it works as expected - not an automated process. This is the important piece : automated regression tests. Regression testing as in "run it and see if it works" is fine - automated ones are, well, another issue. Lets see how this pans out. Be sure I am not against testing, regression testing etc, I am merely suggesting that automated tests are possibly less effective than a proper beta release to users which is currently provided. -- Do you realize if it weren't for Edison we'd be watching TV by candlelight? ~Al Boliska ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 15:04 ` Sebastian Rose 2008-10-23 15:49 ` Richard Riley @ 2008-10-23 16:19 ` Bernt Hansen 2008-10-24 5:05 ` Carsten Dominik 1 sibling, 1 reply; 35+ messages in thread From: Bernt Hansen @ 2008-10-23 16:19 UTC (permalink / raw) To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode Sebastian Rose <sebastian_rose@gmx.de> writes: > Don't know how often the git repo is published. > Bernt and Ben, are you 'worgers' allready? I think it's published hourly (maybe) and yes I have access to Worg. -Bernt ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 16:19 ` Bernt Hansen @ 2008-10-24 5:05 ` Carsten Dominik 0 siblings, 0 replies; 35+ messages in thread From: Carsten Dominik @ 2008-10-24 5:05 UTC (permalink / raw) To: Bernt Hansen; +Cc: Ben Alexander, emacs-orgmode On Oct 23, 2008, at 6:19 PM, Bernt Hansen wrote: > Sebastian Rose <sebastian_rose@gmx.de> writes: > >> Don't know how often the git repo is published. >> Bernt and Ben, are you 'worgers' allready? > > I think it's published hourly (maybe) and yes I have access to Worg. The webserver pulls changes from the Worg git repo on the full hour and publishes these changes on the half hour. So changes will appear on the web after 30-90 minutes. If necessary, this procedure can be upgraded. - Carsten ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 13:43 ` Bernt Hansen 2008-10-23 15:04 ` Sebastian Rose @ 2008-10-23 17:01 ` Jason F. McBrayer 2008-10-23 23:46 ` Eric Schulte 1 sibling, 1 reply; 35+ messages in thread From: Jason F. McBrayer @ 2008-10-23 17:01 UTC (permalink / raw) To: emacs-orgmode Bernt Hansen <bernt@norang.ca> writes: > Some kind of regression testing framework would be awesome. Org-mode is > large enough that this is almost a necessity to keep things stable and > bug-free. > Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ? -- +-----------------------------------------------------------+ | Jason F. McBrayer jmcbray@carcosa.net | | If someone conquers a thousand times a thousand others in | | battle, and someone else conquers himself, the latter one | | is the greatest of all conquerors. --- The Dhammapada | ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Re: How you can help 2008-10-23 17:01 ` Jason F. McBrayer @ 2008-10-23 23:46 ` Eric Schulte 0 siblings, 0 replies; 35+ messages in thread From: Eric Schulte @ 2008-10-23 23:46 UTC (permalink / raw) To: Jason F. McBrayer; +Cc: emacs-orgmode jmcbray@carcosa.net (Jason F. McBrayer) writes: > Bernt Hansen <bernt@norang.ca> writes: > >> Some kind of regression testing framework would be awesome. Org-mode is >> large enough that this is almost a necessity to keep things stable and >> bug-free. >> > > Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ? The author of elunit (Phil) has abandoned it in favor of ert. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 12:04 How you can help Ben Alexander 2008-10-23 13:43 ` Bernt Hansen @ 2008-10-23 14:20 ` Sebastian Rose 2008-10-23 14:50 ` Manish 2008-10-23 14:55 ` Ben Alexander 2008-10-23 15:08 ` Sebastian Rose 2 siblings, 2 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 14:20 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode Hi Ben, I cc'ed the list. The tests I described in my email to the list are not automated. The reason for that is my lack of (e)lisp knowledge. BUT, they where easy to handle for non programmers. I think the little test will make it to the worg site this week, when all private data is removed. You could take a look at it by then. And: improve, improve, improve... :-) If you know of someone who knows how to do automated tests in elisp, or some technique, package, whatever, please post it to the list, so we all can take a look at it and comment->decide something. This is _highly_ _appreciated_. Ben Alexander <bva@alexanderonline.org> writes: > Sebastian Rose wrote: >> 5. I also think of little packages for testing parts of org. > > I'm curious if you or someone else has any ideas for writing automated tests for > org-mode. I haven't the foggiest idea how someone would write a test for the > parts of org that control what is displayed on the screen. I mean, when the > bug is 'it doesn't look right' how can you tell? Believe me, my idea is the foggiest of all possible ideas ;-) (so where my ideas of JavaScript some mounth ago), so I won't be of much help, I fear. > Perhaps the git repository should have a small collection of small org- > mode files that reproduce certain bugs? If there were some examples of how to > create such a test, then perhaps bug reporters would find it much easier to > create them. YES! Exactly. Every corner case an Org-file, every bug an Org-file. _DATA_ for testing is something, everyone _can_ provide. But git later, yes, maybe. Since this would need Carsten to think and act on this, I feel Worg is a nice place to start the first expieriments. We need Carstens power for other things (when will Org-mode finaly wash my car? It's repeater-TODO, but nothing happens!!!) Basically, I'd try to keep the the testing as simple as possible, and try to get elisp programmers to help with the code. We should try to keep the hurdles for testers as humble (?) as possible, and put all that's needed to be helpfull on one page in worg. I recently discovered the very unautomated `(print object)' in the elisp reference. Not everything can be done automated, maybe, but if I would have known this stupid `print' function, I would know more about elisp and some files in org already. And it would have been faster and easier to create the two minor patches I wrote. This is, where the 'links to elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy. Willingness to help is no problem, as the Org-community reveals. As for me, it's often a lack of time and/or knowledge, that prevents it. And the aim of all this is to help helping, in means of good and detailed bug reports in the first place. > I do see some confusing issues due to different configuration files. So > creating a test file might involve making sure org-mode doesn't read any > configuration (how do you do that?) and possible asking org- > mode to extract all the configuration variables it has right now and dump them > into a test file (...and how do you do that?) True. Hm - to test without configuration, we already have `emacs -q'. Dumping the variables is just a list of (print var) statements. A (quite) complete list of variables could be extracted with grep? e.g.: grep -r defvar lisp/ Once we have such a list, we could set/reset some/all variables. This will not be perfect, but could work reasonable well. And yes, it would indeed be nice, to have elisp to reliably reset emacs/org configuration, so one could do several different test in sequence. Preferably they would _always_ run and dump all errors to some file or buffer, even if one or more tests fail. I think elsip provides funtionalities to handle those errors and skip to the next test. This would be a huge step. But still, we have to test with different _DATA_ too! The test I described is just a quick hack I did to do the testing, while Carsten was bug-hunting on the other end of my internet connection. The XHTML-export test was much easier to do, then some other tests we might need. In the end, it was a test, automated or not. And I had to do it, because I'm one out of a few who use org to maintain a web-site (locally in my case) and export an entire directory tree with lots of _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This became obvious in this case. I was the one who even noticed the bug, which means, no one else was using recursive XHTML export for a while, or their setup didn't reveal that bug. So clearly I was the one to provide some support for this very part of the system. I can't rely on the assumption, that some maintainer has all possible setups at hand all the time (maybe this was possible years ago). While the testing of the HTML-export was quite simple, it requires a lot of _DATA_. Namly files and directories + different setups, per file setups, #+SETUPFILE, images, with/without org-info.js, extra styles set or not... to have a realistic testing environment (the test didn't have all of these...). A summary could be, that it's nice to provide different setups for maintainers and testers. - *Easy to install* (unpack and done), - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set everything up) - and *easy to download* once they are removed again (a central place on Worg). - *Corner cases*, like the 'empty line before first heading bug' in the LaTeX exporter recently. Regards, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.roseATemma-stil.de, sebastian_roseATgmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 14:20 ` Sebastian Rose @ 2008-10-23 14:50 ` Manish 2008-10-23 15:46 ` Eric Schulte 2008-10-23 14:55 ` Ben Alexander 1 sibling, 1 reply; 35+ messages in thread From: Manish @ 2008-10-23 14:50 UTC (permalink / raw) To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode Org-Mode On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote: [snip] > > If you know of someone who knows how to do automated tests in > elisp, or some technique, package, whatever, please post it to the > list, so we all can take a look at it and comment->decide > something. This is _highly_ _appreciated_. We could start here: http://www.emacswiki.org/emacs-fr/UnitTesting#toc2 -- Manish ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 14:50 ` Manish @ 2008-10-23 15:46 ` Eric Schulte 2008-10-23 16:18 ` Avdi Grimm 0 siblings, 1 reply; 35+ messages in thread From: Eric Schulte @ 2008-10-23 15:46 UTC (permalink / raw) To: Manish; +Cc: Ben Alexander, emacs-orgmode Org-Mode Hi, I'd recommend ert.el, it is actively maintained, and I have it on good authority that this the tool of choice for elisp unit testing. http://github.com/ohler/ert/tree/master ,----[from ert.el] | ;;; Commentary: | | ;; ERT is a tool for automated testing in Emacs Lisp. Its main | ;; features are facilities for defining and running test cases and | ;; reporting the results as well as for debugging test failures | ;; interactively. | ;; | ;; The main entry points are `ert-deftest', which is similar to | ;; `defun' but defines a test, and `ert-run-tests-interactively', | ;; which runs tests and offers an interactive interface for inspecting | ;; results and debugging. There is also `ert-run-tests-batch' for | ;; non-interactive use. | ;; | ;; The body of `ert-deftest' forms resembles a function body, but the | ;; additional operators `should', `should-not' and `should-error' are | ;; available. `should' is similar to cl's `assert', but signals a | ;; different error when its condition is violated that is caught and | ;; processed by ERT. In addition, it analyzes its argument form and | ;; records information that helps debugging (`assert' tries to do | ;; something similar when its second argument SHOW-ARGS is true, but | ;; `should' is more sophisticated). For information on `should-not' | ;; and `should-error', see their docstrings. `---- Manish <mailtomanish.sharma@gmail.com> writes: > On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote: > [snip] > > > > If you know of someone who knows how to do automated tests in > > elisp, or some technique, package, whatever, please post it to the > > list, so we all can take a look at it and comment->decide > > something. This is _highly_ _appreciated_. > > We could start here: > http://www.emacswiki.org/emacs-fr/UnitTesting#toc2 > > -- Manish > > > _______________________________________________ > Emacs-orgmode mailing list > Remember: use `Reply All' to send replies to the list. > Emacs-orgmode@gnu.org > http://lists.gnu.org/mailman/listinfo/emacs-orgmode ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 15:46 ` Eric Schulte @ 2008-10-23 16:18 ` Avdi Grimm 0 siblings, 0 replies; 35+ messages in thread From: Avdi Grimm @ 2008-10-23 16:18 UTC (permalink / raw) To: emacs-orgmode Org-Mode I've been considering buckling down and doing some more ELisp coding, which would mean org-mode coding because I've become ridiculously dependent on org-mode. If I do, I would only do it in a test-first way, because that's the only way I'll write code anymore; so it would definitely be good to know that unit tests will be be accepted into the codebase. Personally, if I did this I think I would like to use Emacs Lisp Expectations: http://www.emacswiki.org/cgi-bin/emacs/EmacsLispExpectations I'm a big fan of Jay Field's work, I like the way the original Expectations for Ruby is structured, and I like the looks of the EL Expectations API. I'm open to other libraries though - so long as SOME unit-testing standard it accepted. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 14:20 ` Sebastian Rose 2008-10-23 14:50 ` Manish @ 2008-10-23 14:55 ` Ben Alexander 2008-10-23 16:26 ` Sebastian Rose 1 sibling, 1 reply; 35+ messages in thread From: Ben Alexander @ 2008-10-23 14:55 UTC (permalink / raw) To: Sebastian Rose; +Cc: emacs-orgmode Org-Mode Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting Unfortunately for me, I can't tell if Emacs comes with any builtin framework already, so I downloaded one of the many options listed on that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el The personal issue I have is that I'm on a Mac, using Aquamacs, and the command line version of emacs is a different binary, so there might be trouble when a test passes at the command line, but fails where it matters to me. I don't even make sure that org-mode is up to date at the command line. I thought it wasn't, but I just checked and now it is. Plus, I don't really understand internals of emacs (like basic internals: I understand point and mark, buffer and file, but not transient mark, indirect buffer, symbols vs strings, window vs tab vs frame) The tutorial I'd need to write a test is one which lays out code I could copy and paste to do the following * setup the test environment - create a test directory - create a sample test.org file - put the cursor in a particular place * run the command we need to test - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to find out exactly what command is actually being run when you hit a keystroke. And some of me might need to be told what lisp-code to use when the keystroke runs different commands at different places in a file) - reformat a table - clock in/out - create the agenda - export .html .ics .dvi file * How do we specify the correct result??? - check that the headline folded properly. What's the lisp code for getting the folded string as displayed? - check that the cursor is where it should be? especially when the cursor is near elipses... - check that the agenda is built properly. What's the lisp code for getting the agenda as a string? - check that the exported files are correct. Maybe the right suggestion is to run the export on two different files, so the test can focus on the 'diff' between them. That way different people who run the same test on different hosts can get the same result. More experienced folks must suggest whether it's better to have the expectation of a test be specified as a string in the test code or as a separate file in the test directory. It might depend on context, so feel free to lay out three or four cases. If even one kind of test for each of these 'cases' exists, hopefully we can encourage people to report bugs with the test. I mean, if people are already going through the effort to create the small sample org file to demonstrate their bug, and it's only a short (obvious) step to translating their english description of what should happen into equivalent lisp code.... that seems like a big temptation for someone to do it. Plus it might help them convince themselves that it really is a bug, and not that they are going crazy. I gotta move on for the day here, but I'll keep thinking a bit about this. On 2008-Oct-23, at 15:20, Sebastian Rose wrote: > Hi Ben, > > I cc'ed the list. > > > The tests I described in my email to the list are not automated. The > reason for that is my lack of (e)lisp knowledge. > > BUT, they where easy to handle for non programmers. I think the little > test will make it to the worg site this week, when all private data is > removed. You could take a look at it by then. And: improve, improve, > improve... :-) > > If you know of someone who knows how to do automated tests in elisp, > or > some technique, package, whatever, please post it to the list, so we > all > can take a look at it and comment->decide something. This is _highly_ > _appreciated_. > > > Ben Alexander <bva@alexanderonline.org> writes: >> Sebastian Rose wrote: >>> 5. I also think of little packages for testing parts of org. >> >> I'm curious if you or someone else has any ideas for writing >> automated tests for >> org-mode. I haven't the foggiest idea how someone would write a >> test for the >> parts of org that control what is displayed on the screen. I >> mean, when the >> bug is 'it doesn't look right' how can you tell? > > Believe me, my idea is the foggiest of all possible ideas ;-) (so > where > my ideas of JavaScript some mounth ago), so I won't be of much help, I > fear. > > >> Perhaps the git repository should have a small collection of small >> org- >> mode files that reproduce certain bugs? If there were some >> examples of how to >> create such a test, then perhaps bug reporters would find it much >> easier to >> create them. > > YES! Exactly. Every corner case an Org-file, every bug an Org-file. > _DATA_ for testing is something, everyone _can_ provide. > > But git later, yes, maybe. > > Since this would need Carsten to think and act on this, I feel Worg > is a > nice place to start the first expieriments. We need Carstens power for > other things (when will Org-mode finaly wash my car? It's repeater- > TODO, > but nothing happens!!!) > > Basically, I'd try to keep the the testing as simple as possible, and > try to get elisp programmers to help with the code. We should try to > keep the hurdles for testers as humble (?) as possible, and put all > that's needed to be helpfull on one page in worg. > > I recently discovered the very unautomated `(print object)' in the > elisp > reference. Not everything can be done automated, maybe, but if I would > have known this stupid `print' function, I would know more about elisp > and some files in org already. And it would have been faster and > easier > to create the two minor patches I wrote. This is, where the 'links to > elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy. > > Willingness to help is no problem, as the Org-community reveals. As > for > me, it's often a lack of time and/or knowledge, that prevents it. > > And the aim of all this is to help helping, in means of good and > detailed bug reports in the first place. > > > > >> I do see some confusing issues due to different configuration >> files. So >> creating a test file might involve making sure org-mode doesn't >> read any >> configuration (how do you do that?) and possible asking org- >> mode to extract all the configuration variables it has right now >> and dump them >> into a test file (...and how do you do that?) > > True. > > Hm - to test without configuration, we already have `emacs -q'. > Dumping the variables is just a list of (print var) statements. A > (quite) complete list of variables could be extracted with grep? > > e.g.: > > grep -r defvar lisp/ > > Once we have such a list, we could set/reset some/all variables. > > This will not be perfect, but could work reasonable well. > > And yes, it would indeed be nice, to have elisp to reliably reset > emacs/org configuration, so one could do several different test in > sequence. Preferably they would _always_ run and dump all errors to > some > file or buffer, even if one or more tests fail. > > I think elsip provides funtionalities to handle those errors and > skip to > the next test. This would be a huge step. > > > > > > > But still, we have to test with different _DATA_ too! The test I > described is just a quick hack I did to do the testing, while Carsten > was bug-hunting on the other end of my internet connection. > > The XHTML-export test was much easier to do, then some other tests we > might need. In the end, it was a test, automated or not. And I had > to do > it, because I'm one out of a few who use org to maintain a web-site > (locally in my case) and export an entire directory tree with lots of > _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This > became obvious in this case. I was the one who even noticed the bug, > which means, no one else was using recursive XHTML export for a while, > or their setup didn't reveal that bug. > > So clearly I was the one to provide some support for this very part of > the system. I can't rely on the assumption, that some maintainer has > all > possible setups at hand all the time (maybe this was possible years > ago). > > While the testing of the HTML-export was quite simple, it requires a > lot > of _DATA_. Namly files and directories + different setups, per file > setups, #+SETUPFILE, images, with/without org-info.js, extra styles > set > or not... to have a realistic testing environment (the test didn't > have > all of these...). > > > > > A summary could be, that it's nice to provide different setups for > maintainers and testers. > > - *Easy to install* (unpack and done), > - *easy to use* ('emacs -q' and click a few [[elisp:]] links to set > everything up) > - and *easy to download* once they are removed again (a central > place on > Worg). > - *Corner cases*, like the 'empty line before first heading bug' in > the > LaTeX exporter recently. > > > > > Regards, > > > -- > Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 > Hannover > > Tel.: +49 (0)511 - 36 58 472 > Fax: +49 (0)1805 - 233633 - 11044 > mobil: +49 (0)173 - 83 93 417 > Email: s.roseATemma-stil.de, sebastian_roseATgmx.de > Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 14:55 ` Ben Alexander @ 2008-10-23 16:26 ` Sebastian Rose 2008-10-23 16:42 ` Avdi Grimm [not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org> 0 siblings, 2 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 16:26 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode Ben, I added this to the new page on worg. Please tell me: is it OK for you, if I simply add your ideas there?? What do the others think? Ben Alexander <bva@alexanderonline.org> writes: > The tutorial I'd need to write a test is one which lays out code I could copy > and paste to do the following > > * setup the test environment > - create a test directory > - create a sample test.org file > - put the cursor in a particular place > > * run the command we need to test > - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to > find out exactly what command is actually being run when you hit a keystroke. > And some of me might need to be told what lisp-code to use when the keystroke > runs different commands at different places in a file) > - reformat a table > - clock in/out > - create the agenda > - export .html .ics .dvi file > * How do we specify the correct result??? > - check that the headline folded properly. What's the lisp code for getting > the folded string as displayed? > - check that the cursor is where it should be? especially when the cursor is > near elipses... > - check that the agenda is built properly. What's the lisp code for getting > the agenda as a string? > - check that the exported files are correct. Maybe the right suggestion is to > run the export on two different files, so the test can focus on the 'diff' > between them. That way different people who run the same test on different > hosts can get the same result. Please, if you find time, add the rest somewhere there? If not, I'll take a closer look tonight and check in what's interesting (which seems to be all of it, actually). http://www.emacswiki.org/emacs-fr/UnitTesting: Does this mean to add code to Org's *.el files? This would mean a lot of work for Carsten :-( OK, once done... But: sh> grep -r defun lisp/ | wc -l 1290 Checking in/out of every function makes 2580 places to check. I can't estimate the effort, actually, because I never did Unit testing. I guess this number much higher than needed? Could anyone with testing expierience estimate? While I think automated tests are a great thing to do and I'm happy to see this reaction on the list :-), I still think we should also provide the 'simple' tests without additional dependencies (when will package management make it into emacs...). And I'd like to keep the effort and impact for Carsten as small as possible. He will have to check each and every patch because of feeling, and actually being, responsible for the code. Another question: will the code compile in emacs without the testing framework installed? Probably yes. Should we try to think of those 'simple test packages' as the more _DATA_ part of other tests? This would mean one could code an automated test without the need to create data too, since we have the data seperated. Again, I'd like to keep the hurdles low for everyone involved. When automated testing turns out to be a foolish easy and rock solid, this could change. * Three basic approaches: 1 Simple data and setup packages. 2 Code, that executes org commands and test for correct output, no changes in Org's code needed. 3 Test framework with changes to Org's code. I'd promote a combination from 1 and 2, and see, how far it goes. DATA +--- d1 +--- d2 +--- d3 TESTS +---t-a +---t-b +---t-c Now one would run test t-a over d1, d2, d3. Still a programmer could use d1 to fix a bug in d1 and finally push the fix for automated testing against d2 and d3. bug X tester verifies (is ist bug? how to reproduce...) tester adds X to dX (message -> list or programmer) programmer installs dX programmer fixes bugX programmer push (message -> list or tester) tester tests against d1, d2, d3 where tester = [human|machine] preferable human (simplicity) Should we track the dependencies between code and tests? Suppose one has worked on a.el. Shure more than one test has to run. Example: org-publish.el contains code common to LaTeX-export, HTML-export... Regards, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 16:26 ` Sebastian Rose @ 2008-10-23 16:42 ` Avdi Grimm 2008-10-23 17:33 ` Sebastian Rose 2008-10-24 18:33 ` Ben Alexander [not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org> 1 sibling, 2 replies; 35+ messages in thread From: Avdi Grimm @ 2008-10-23 16:42 UTC (permalink / raw) To: Sebastian Rose; +Cc: Ben Alexander, emacs-orgmode Org-Mode A few points, from someone with a decent amount of testing background: * As someone who has contributed to OS projects, the lack of a pre-existing set of regression tests in org-mode is actually *the* most significant blocker to my getting involved in org-mode development. I do all of my coding - both professional and personal - in the context of tests, and not having the existing framework means that before I can start working on new features I first need to spend time yak-shaving on testing infrastructure. * I think there's a lot of over-thinking going on here. Here's the test-first coding discipline in a nutshell: 1. Identify a problem/missing feature. 2. Write a test (possibly using a unit-testing framework to help) which will pass when the bug has been fixed or feature added. This can be as simple as calling a function and validating its return value. 3. Run the test. Verify it FAILS. 4. Write code to make the test PASS. 5. Refactor, if you introduced any code duplication in step 4. 6. Run all the tests, to make sure you didn't break anything else. 7. Commit. If someone would be so kind as to identify a small bug or feature, I would be happy to demonstrate this workflow in the form of code, time permitting. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 16:42 ` Avdi Grimm @ 2008-10-23 17:33 ` Sebastian Rose 2008-10-23 19:10 ` Avdi Grimm 2008-10-24 21:09 ` Tom Breton (Tehom) 2008-10-24 18:33 ` Ben Alexander 1 sibling, 2 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 17:33 UTC (permalink / raw) To: emacs-orgmode Org-Mode "Avdi Grimm" <avdi@avdi.org> writes: > A few points, from someone with a decent amount of testing background: Jippie! Please stay with us for a few days :-D > * As someone who has contributed to OS projects, the lack of a > pre-existing set of regression tests in org-mode is actually *the* > most significant blocker to my getting involved in org-mode > development. I do all of my coding - both professional and personal - > in the context of tests, and not having the existing framework means > that before I can start working on new features I first need to spend > time yak-shaving on testing infrastructure. > > * I think there's a lot of over-thinking going on here. Here's the > test-first coding discipline in a nutshell: > 1. Identify a problem/missing feature. > 2. Write a test (possibly using a unit-testing framework to help) > which will pass when the bug has been fixed or feature added. This can > be as simple as calling a function and validating its return value. > 3. Run the test. Verify it FAILS. > 4. Write code to make the test PASS. > 5. Refactor, if you introduced any code duplication in step 4. > 6. Run all the tests, to make sure you didn't break anything else. > 7. Commit. > > If someone would be so kind as to identify a small bug or feature, I > would be happy to demonstrate this workflow in the form of code, time > permitting. Sounds great! Not shure if we can provide a bug though :-D Bug anyone? Hm - how about: 1. A not yet existent elisp file test-worg.el, that defines a function hello-worg, and a variable lang, and simply puts "Hello Worg"into the minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang "de")) ? Did you work with unit-testing frameworks for elisp already? Which one? Recommendations? Could say something about the effort to get started with such a framework? Can we add it without changing Org's code? If understand 2. correctly - yes? If we can, would we loose quality/speed of tests? How much? Tell us all :-) Oha - I have an appointment (concert) - back tonight! All the best, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 17:33 ` Sebastian Rose @ 2008-10-23 19:10 ` Avdi Grimm 2008-10-24 21:09 ` Tom Breton (Tehom) 1 sibling, 0 replies; 35+ messages in thread From: Avdi Grimm @ 2008-10-23 19:10 UTC (permalink / raw) To: Sebastian Rose; +Cc: emacs-orgmode Org-Mode On Thu, Oct 23, 2008 at 1:33 PM, Sebastian Rose <sebastian_rose@gmx.de> wrote: > Hm - how about: > > 1. A not yet existent elisp file test-worg.el, that defines a function > hello-worg, and a variable lang, and simply puts "Hello Worg"into the > minibuffer (if (string= lang "en")) and "Hallo Worg" (if (string= lang > "de")) ? I'd prefer something that would be of concrete value to someone in using org-mode - made-up scenarios tend to lead to unrealistic tests. > Did you work with unit-testing frameworks for elisp already? > Which one? I have not. I've done test-driven development in Java, C#, Ruby, and C++(!) though, and if I can do it in C++ I think I can get up to speed in any language pretty quickly ;-) > Recommendations? As I said in another comment, I'm liking the look of Emacs Lisp Expectations a *lot*. Partly because I think Jay Fields is bang on the money for how maintain good habits in your tests, and partly because it has a mocking framework to go with it. Testing without a mocking framework is less fun than testing with. However, I haven't actually used it yet, so I can't responsibly recommend it. Let's just say that the discovery of it recently was the thing that got me excited about doing some serious ELisp development. > Could say something about the effort to get started with such a > framework? I think if you have some *running* examples to crib from it shouldn't take someone who is already ELisp-savvy more than a few hours to get the hang of it. > > Can we add it without changing Org's code? If understand 2. correctly - yes? In a dynamic language like ELisp, any testing framework that requires you to change the code just to get it under test isn't worth using. No, you should not have to change org-mode's code to get started with tests. That said, developing in concert with a test suite will definitely tend to influence how you write the code. Generally speaking, developing with tests encourages smaller functions which do one thing and one thing alone and don't have a lot of non-obvious inputs and outputs. Since this is usually the direction you *want* your code to go in anyway, it's a virtuous cycle. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 17:33 ` Sebastian Rose 2008-10-23 19:10 ` Avdi Grimm @ 2008-10-24 21:09 ` Tom Breton (Tehom) 1 sibling, 0 replies; 35+ messages in thread From: Tom Breton (Tehom) @ 2008-10-24 21:09 UTC (permalink / raw) Cc: emacs-orgmode Org-Mode > "Avdi Grimm" <avdi@avdi.org> writes: > > A few points, from someone with a decent amount of testing background: > > Jippie! Please stay with us for a few days :-D [...] As someone with also a decent amount of testing background, I'd like to answer also. > Did you work with unit-testing frameworks for elisp already? > Which one? I co-wrote regress.el (Wayne Mesard wrote the first version) and I wrote rtest. I've written some testing for allout.el before I switched over to org-mode (which is very nice, BTW, Carsten, and provides pretty much everything I wish-listed for allout mode). Of particular interest for testing org-mode, I wrote mockbuf.el, which mocks files and buffers so that tests can deal with files without changing anything permanently. > Recommendations? Hard for me to say objectively. I prefer my rtest but it's in flux right now. I wouldn't recommend the current public version, rtest 3.0 I've designed and half written a better version but I can't really ask you to wait until I get around to making a better version of rtest. There is a development version that works and does nice things; it's messy as hell right now and I'm embarrassed about that. It isn't public but I could put it up if you'll forgive its messy state, It has mockbuf, which I found indispensable in writing tests for allout.el. In fact, I mostly developed it for that, though all the allout-specific code is elsewhere. Also useful for writing tests: safe-equal.el, caselet.el, and select.cl, all in the ancillary directory. I would steer clear of frameworks that provide many of synonyms for equality tests, inequality tests, etc. You already know how to write (not (equal ...)) in elisp. Why learn the name and syntax of something like (shouldnt-be-equal A B)? Having a lot of that makes a package LOOK heftier but doesn't do anything more for you. IMO analyzing test failures should be done by inspecting the test form, not by making you write things in non-standard elisp that could be written in standard elisp. Using ert would mean requiring ewoc, which means tests wouldn't work on emacs 21. Too bad, because it has some appeal. > Could say something about the effort to get started with such a > framework? For rtest, right now you just need (a) the development version, (b) unzip it in your emacs-site-lisp or set the paths and autoloads manually if you prefer, (c) include the following in a file to be tested: (when (not (fboundp 'rtest:deftest)) (defmacro rtest:deftest (&rest dummy)) (defmacro rtest:if-avail (&rest dummy))) Then write tests as: (rtest:deftest function-name ("Docstring" (test-form) ;;Return a boolean ) ;;Repeat as wanted ) That will get you started, but really you'd want to get familiar with the stuff in mockbuf.el - which is so messy right now that I'm embarrassed about it, because it contains both old and new versions of many things in flux. Other advice: Make a directory to hold short examples of files that illustrate "before" and "after" states for the various transformations. Each unit (eg org-agenda.el, org-archive.el etc) should have its own subtree of examples. One thing I'd do early is build the following test helper functions: A macro that runs a test body inside a "let" of all the configuration options set to expected values. A function to compare vector-style strings. `equal' doesn't suffice. rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and safe-equal-compare-vectors, which IIRC suffice for this. The ability to write only the visible parts to a separate buffer was indispensable in testing allout's folding. You already have it in org-export-visible with the type that keeps the buffer, "?\ ". Then you can just compare contents. You will likely have a lot of what I call "output tests", tests of code that has the property that if the output looks right, it is right. IMO with output tests, you need to make it easy on yourself. TDD is all well and fine, but not if it makes you repeat yourself. If there is available output that looks right, capture it and use it as a base for comparison. That's not "pure" but it's far easier and it works right. If the available output isn't yet right, capture it anyways and tweak it to make it right - but this rarely happens. I automated that sort of thing for a Prolog test harness I wrote, works fine. You'd just write a test of a special type that produced and captured output. It would originally be ungraded, but you would then grade it, and it would remember the correct output and compare against it. I was planning to add the same to rtest and I partly did, but never finished it. Still on my todo list. > Can we add it without changing Org's code? If understand 2. correctly - yes? > rtest's tests like to be in the same file as the code. If you use the form above, the file still compiles and runs even when rtest is not available. It's possible to write the tests in another file, but then either tests aren't grouped together (must be run one by one) or you have to do extra work to group them. If they're in file, you can run every test in a package with `rtest:library'. The other really convenient way to run a test is rtest:defun-at-point. > If we can, would we loose quality/speed of tests? Not at all. Tom Breton (Tehom) PS: One more piece of advice: Make an auto-insert or at least a skeleton that inserts test headers, like the "(when (not (fboundp ..." stuff above. You'll be using it a fair bit, and it's real nice to have it just automatically be there when you start a new file. This advice is geared toward my way of working, though, with the test code in the same file as the functional code. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 16:42 ` Avdi Grimm 2008-10-23 17:33 ` Sebastian Rose @ 2008-10-24 18:33 ` Ben Alexander 2008-10-24 18:44 ` Avdi Grimm 1 sibling, 1 reply; 35+ messages in thread From: Ben Alexander @ 2008-10-24 18:33 UTC (permalink / raw) To: Avdi Grimm; +Cc: emacs-orgmode Org-Mode On 2008-Oct-23, at 17:42, Avdi Grimm wrote: > > If someone would be so kind as to identify a small bug or feature, I > would be happy to demonstrate this workflow in the form of code, time > permitting. > > Ok, here's your chance. This is something that has bothered me for quite some time, but I've never been able to reliably reproduce the problem. And it's such a small issue. Sometimes, when it looks like my cursor is at the end of a headline and I hit tab, the headline does not cycle. Normally it does. It hit me that the point was not on the headline, it only looked like it. So here's the sample test.org file * First Headline Some body text * Second Headline (goto-char 17) ;that's the end of the first line (org-cycle) At this point the buffer looks like: * First Headline... * Second Headline (goto-char 25) (org-cycle) I think the buffer should look like this * First Headline Some body text * Second Headline but it still looks like: * First Headline... * Second Headline * ABOUT THIS BEHAVIOR I should be honest here. This may not be a bug. As I created this test data, it became a bit more clear that exactly where the cursor is matters a lot here. I *promise* that I've seen this behavior even when the cursor shows up before the ellipses, but in the test case I set up here, I could not make that happen. I also found that if I used the key-chord M-x goto-char <RETURN> 25, everything worked fine. The cursor stayed in front of the ellipses and the tab key worked. But when I used M-: (goto-char 25) the cursor moved to the same line as the ellipses, but after them. and the <TAB> key stopped 'working'. I finally figured this out while playing with git. I switched branches at the command line and needed to perform a 'revert-buffer'. This left the cursor before the ellipses, but unable to properly (org- cycle) using the <TAB> key. More experimenting.... I *think* the revert-buffer command tries to keep point close to the same place, and the org buffer had automatic folding. The bad thing about this for me is that I'll hit the <TAB> key four times trying to make sure it's not just an empty tree. Meanwhile, (org-cycle) has indented the first line of the body, but hasn't shown me the text I'm changing. Then I get confused about why the buffer needs saving, probably 10 minutes later when I've forgotten all about hitting the tab key. But I'm way out of my depth to try to understand the relationship between (revert-buffer) (org-cycle) the arrow keys, and all. I just hit M-< or S-<TAB> and try again. Or C-a, which sometimes works. Sometimes I grab the mouse as a last resort. * ABOUT TESTING ISSUES Just from this example, I'm eager to understand: 1. How can we differentiate between where the cursor appears and the value of point 2. How can we tell what the users sees on the screen versus what the buffer contains So the value of a testing framework is that this: if I'm going to announce to the world that I can't get something to work, (like the tab key), I'm going to make darn sure it's a real problem. I'm going to spend the time to create a small sample file, that reliable creates the problem, and I'm going to try a few different scenarios. But if I start to get lost or confused about which settings I've fiddled with or what is supposed to happen, I'll wander away from the problem and I won't submit the bug. We all lose in that scenario. But if the testing framework exists, and it is lightweight enough for novice emacs users (advanced enough to use M-x but not advanced enough to read lisp) then maybe I would have used it for this example. -Ben ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-24 18:33 ` Ben Alexander @ 2008-10-24 18:44 ` Avdi Grimm 2008-10-24 19:02 ` Jeff Mickey 2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander 0 siblings, 2 replies; 35+ messages in thread From: Avdi Grimm @ 2008-10-24 18:44 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org> wrote: > Ok, here's your chance. This is something that has bothered me for quite > some time, but I've never been able to reliably reproduce the problem. And > it's such a small issue. Thanks! I'll try to take a look tonight or this weekend. > So the value of a testing framework is that this: if I'm going to announce > to the world that I can't get something to work, (like the tab key), I'm > going to make darn sure it's a real problem. I'm going to spend the time to > create a small sample file, that reliable creates the problem, and I'm going > to try a few different scenarios. But if I start to get lost or confused > about which settings I've fiddled with or what is supposed to happen, I'll > wander away from the problem and I won't submit the bug. We all lose in > that scenario. But if the testing framework exists, and it is lightweight > enough for novice emacs users (advanced enough to use M-x but not advanced > enough to read lisp) then maybe I would have used it for this example. For the record, I normally would never expect the user to submit tests. A description of the problem and some sample data is all I expect. Writing the a test to reproduce the behavior is the programmer's responsibility as far as I'm concerned. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-24 18:44 ` Avdi Grimm @ 2008-10-24 19:02 ` Jeff Mickey 2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander 1 sibling, 0 replies; 35+ messages in thread From: Jeff Mickey @ 2008-10-24 19:02 UTC (permalink / raw) To: Avdi Grimm; +Cc: Ben Alexander, emacs-orgmode Org-Mode On Fri, Oct 24, 2008 at 14:44, Avdi Grimm <avdi@avdi.org> wrote: > For the record, I normally would never expect the user to submit > tests. A description of the problem and some sample data is all I > expect. Writing the a test to reproduce the behavior is the > programmer's responsibility as far as I'm concerned. That's the best part of open source software! A surprising amount of the time, the user IS a programmer, and thus is able to provide tests. And it's not that we should require anyone to provide tests before they report a bug, but they should be required to outline the steps to reproduce the behavior, so any programmer could take 5 minutes and quickly whip up a test. ps: I almost have a baby ert framework for org-mode. It doesn't do anything yet, but the design is you write a test, and add it to the org-test-hook to be run by org-test-all. -- . : [ + carpe diem totus tuus + ] : . ^ permalink raw reply [flat|nested] 35+ messages in thread
* org-cycle broken when cursor is at ellipses 2008-10-24 18:44 ` Avdi Grimm 2008-10-24 19:02 ` Jeff Mickey @ 2008-10-26 19:49 ` Ben Alexander 2008-10-26 21:31 ` Cameron Horsburgh 2008-10-27 8:47 ` Carsten Dominik 1 sibling, 2 replies; 35+ messages in thread From: Ben Alexander @ 2008-10-26 19:49 UTC (permalink / raw) To: Avdi Grimm; +Cc: emacs-orgmode Org-Mode So about this bug... I think I've figured out why I get this so often. I use Aquamacs on MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs Emacs/places.el) a list of files and values for point. Something about the timing of this means that point is left somewhere in the body (or maybe subheading?) of a headline, even though org-mode presents the file in 'Overview'. Since the point is in an unexpected location, even though the cursor looks fine, (org-cycle) doesn't work as it looks like it should. I think I'd like to turn off the 'places.el' thing. And I reiterate: this is really a tiny issue. Just hitting C-a fixes the state of point. On 2008-Oct-24, at 19:44, Avdi Grimm wrote: > On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org > > wrote: >> Ok, here's your chance. This is something that has bothered me for >> quite >> some time, but I've never been able to reliably reproduce the >> problem. And >> it's such a small issue. > > Thanks! I'll try to take a look tonight ... -Ben ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses 2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander @ 2008-10-26 21:31 ` Cameron Horsburgh 2008-10-27 8:47 ` Carsten Dominik 2008-10-27 8:47 ` Carsten Dominik 1 sibling, 1 reply; 35+ messages in thread From: Cameron Horsburgh @ 2008-10-26 21:31 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode On Sun, Oct 26, 2008 at 07:49:46PM +0000, Ben Alexander wrote: > So about this bug... > > I think I've figured out why I get this so often. I use Aquamacs on > MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs > Emacs/places.el) a list of files and values for point. > For the record, I'm using emacs-snapshot on Debian sid and I have the same (or a very similar) issue. -- Cameron Horsburgh Blog: http://spiritcry.wordpress.com/ ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses 2008-10-26 21:31 ` Cameron Horsburgh @ 2008-10-27 8:47 ` Carsten Dominik 0 siblings, 0 replies; 35+ messages in thread From: Carsten Dominik @ 2008-10-27 8:47 UTC (permalink / raw) To: Cameron Horsburgh; +Cc: Ben Alexander, emacs-orgmode Org-Mode [-- Attachment #1.1: Type: text/plain, Size: 780 bytes --] On Oct 26, 2008, at 10:31 PM, Cameron Horsburgh wrote: > On Sun, Oct 26, 2008 at 07:49:46PM +0000, Ben Alexander wrote: >> So about this bug... >> >> I think I've figured out why I get this so often. I use Aquamacs on >> MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs >> Emacs/places.el) a list of files and values for point. >> > > For the record, I'm using emacs-snapshot on Debian sid and I have the > same (or a very similar) issue. Is that also related to saveplace? In general, in outlines there is the danger that the cursor can be located at an invisible point, but in practice that should not happen too often. I myself never press TAB with the cursor at the ellipsis dots, because you cannot be sure where the cursor truly is. - Carsten [-- Attachment #1.2: Type: text/html, Size: 1330 bytes --] [-- Attachment #2: Type: text/plain, Size: 204 bytes --] _______________________________________________ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: org-cycle broken when cursor is at ellipses 2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander 2008-10-26 21:31 ` Cameron Horsburgh @ 2008-10-27 8:47 ` Carsten Dominik 1 sibling, 0 replies; 35+ messages in thread From: Carsten Dominik @ 2008-10-27 8:47 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode [-- Attachment #1.1: Type: text/plain, Size: 2304 bytes --] Hi Ben this is what I do not like about AquaEmacs, that it installs all kinds of defaults which you might not want. You can turn off saveplace with (setq-default save-place nil) You might still have to remove places.el. Or you can make sure that the cursor always starts out at the beginning of a visible line with (add-hook 'org-mode-hook (lambda () (let (org-special-ctrl-a/e) (org-beginning-of-line)))) Then saveplace will get its will by moving the cursor to the recent location, but the cursor would be moved to the beginning of a visible line near that location. Or you can make sure that the point visited by saveplace will be visible, with (eval-after-load "saveplace" '(defadvice save-place-find-file-hook (after org-make-visible activate) "Make the position visible." (org-bookmark-jump-unhide))) I am actually putting this last piece of code into org.el. HTH - Carsten On Oct 26, 2008, at 8:49 PM, Ben Alexander wrote: > So about this bug... > > I think I've figured out why I get this so often. I use Aquamacs on > MacOS X, and it seems to store (in ~/Library/Preferences/Aquamacs > Emacs/places.el) a list of files and values for point. > > Something about the timing of this means that point is left > somewhere in the body (or maybe subheading?) of a headline, even > though org-mode presents the file in 'Overview'. Since the point is > in an unexpected location, even though the cursor looks fine, (org- > cycle) doesn't work as it looks like it should. > > I think I'd like to turn off the 'places.el' thing. > > And I reiterate: this is really a tiny issue. Just hitting C-a > fixes the state of point. > > On 2008-Oct-24, at 19:44, Avdi Grimm wrote: > >> On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander <bva@alexanderonline.org >> > wrote: >>> Ok, here's your chance. This is something that has bothered me >>> for quite >>> some time, but I've never been able to reliably reproduce the >>> problem. And >>> it's such a small issue. >> >> Thanks! I'll try to take a look tonight > ... > > -Ben > > _______________________________________________ > Emacs-orgmode mailing list > Remember: use `Reply All' to send replies to the list. > Emacs-orgmode@gnu.org > http://lists.gnu.org/mailman/listinfo/emacs-orgmode [-- Attachment #1.2: Type: text/html, Size: 3428 bytes --] [-- Attachment #2: Type: text/plain, Size: 204 bytes --] _______________________________________________ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode ^ permalink raw reply [flat|nested] 35+ messages in thread
[parent not found: <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>]
* Re: How you can help [not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org> @ 2008-10-23 17:12 ` Sebastian Rose 0 siblings, 0 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 17:12 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode (cc'ing the list again) Ben Alexander <bva@alexanderonline.org> writes: > On 2008-Oct-23, at 17:26, Sebastian Rose wrote: > >> Ben, >> >> I added this to the new page on worg. Please tell me: is it OK for you, >> if I simply add your ideas there?? >> > Sure put what you want there. Great. > FWIW, I'm brand new (today!) to git and I've cloned Worg today too. But I not a > worger' -- I'm not sure I've read much of it online. So I'll look what else should be added? Hm - this thread is so busy, that I'd say I take a few hours on weekend to sum it up. >> What do the others think? >> >> >> Ben Alexander <bva@alexanderonline.org> writes: >>> way too much >>> >> >> >> Please, if you find time, add the rest somewhere there? >> If not, I'll take a closer look tonight and check in what's interesting >> (which seems to be all of it, actually). >> >> >> http://www.emacswiki.org/emacs-fr/UnitTesting: >> >> Does this mean to add code to Org's *.el files? This would mean a lot of >> work for Carsten :-( OK, once done... >> > > Uh, I hope that's not necessary. I'd hope that the tests could be written > completely independently of any org-code. If the basic idea is to launch emacs > in batch mode, then the test could redefine any function locally, in the test > code, instead of screwing around with what Carsten has. > > If every test is written this way, it'll be very slow to run all the tests. > Maybe that doesn't matter for bug-report tests? But here's where I should shut > up and listen instead of talk. Great! I hope so too!! If slow, it's still no problem to run all those tests before a release or once a day, whatever. Glad to here your interested in keeping impact on Carsten low! Cheers, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: How you can help 2008-10-23 12:04 How you can help Ben Alexander 2008-10-23 13:43 ` Bernt Hansen 2008-10-23 14:20 ` Sebastian Rose @ 2008-10-23 15:08 ` Sebastian Rose 2 siblings, 0 replies; 35+ messages in thread From: Sebastian Rose @ 2008-10-23 15:08 UTC (permalink / raw) To: Ben Alexander; +Cc: emacs-orgmode Org-Mode Ben Alexander <bva@alexanderonline.org> writes: > Perhaps the git repository should have a small collection of small org- > mode files that reproduce certain bugs? Ah, stupid me, Worg IS a git repo :-D -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: s.rose@emma-stil.de, sebastian_rose@gmx.de Http: www.emma-stil.de ^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2008-10-27 8:47 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-10-23 12:04 How you can help Ben Alexander
2008-10-23 13:43 ` Bernt Hansen
2008-10-23 15:04 ` Sebastian Rose
2008-10-23 15:49 ` Richard Riley
2008-10-23 16:22 ` Ben Alexander
2008-10-23 17:02 ` Sebastian Rose
2008-10-24 12:13 ` Richard Riley
2008-10-24 15:39 ` Sebastian Rose
2008-10-24 16:27 ` Manish
2008-10-24 18:41 ` Avdi Grimm
2008-10-23 19:13 ` Avdi Grimm
2008-10-24 12:19 ` Richard Riley
2008-10-23 16:19 ` Bernt Hansen
2008-10-24 5:05 ` Carsten Dominik
2008-10-23 17:01 ` Jason F. McBrayer
2008-10-23 23:46 ` Eric Schulte
2008-10-23 14:20 ` Sebastian Rose
2008-10-23 14:50 ` Manish
2008-10-23 15:46 ` Eric Schulte
2008-10-23 16:18 ` Avdi Grimm
2008-10-23 14:55 ` Ben Alexander
2008-10-23 16:26 ` Sebastian Rose
2008-10-23 16:42 ` Avdi Grimm
2008-10-23 17:33 ` Sebastian Rose
2008-10-23 19:10 ` Avdi Grimm
2008-10-24 21:09 ` Tom Breton (Tehom)
2008-10-24 18:33 ` Ben Alexander
2008-10-24 18:44 ` Avdi Grimm
2008-10-24 19:02 ` Jeff Mickey
2008-10-26 19:49 ` org-cycle broken when cursor is at ellipses Ben Alexander
2008-10-26 21:31 ` Cameron Horsburgh
2008-10-27 8:47 ` Carsten Dominik
2008-10-27 8:47 ` Carsten Dominik
[not found] ` <D43ED86C-EFD4-4BA8-8528-4F82DB11D625@alexanderonline.org>
2008-10-23 17:12 ` How you can help Sebastian Rose
2008-10-23 15:08 ` Sebastian Rose
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).