* Regression testing for org-mode @ 2008-10-23 16:03 Robert Goldman 2008-10-23 23:57 ` Eric Schulte 0 siblings, 1 reply; 10+ messages in thread From: Robert Goldman @ 2008-10-23 16:03 UTC (permalink / raw) To: emacs-orgmode There's a page on the EmacsWiki describing schemes for unit/regression testing in emacs lisp: http://www.emacswiki.org/cgi-bin/wiki/UnitTesting Perhaps we could pick up one of these? I haven't evaluated any of them. In response to Ben Alexander's question, yes, sometimes the screen output is what you care about, but a big advantage of emacs is that the screen output is a big array of text, so we could check it (it's a lot worse, if it's just a bit map...). Sure, we can't always do that, but I think in many cases we could just grab up a buffer's contents and compare it against a "right answer." best, r ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Regression testing for org-mode 2008-10-23 16:03 Regression testing for org-mode Robert Goldman @ 2008-10-23 23:57 ` Eric Schulte 2008-10-24 0:03 ` Avdi Grimm 0 siblings, 1 reply; 10+ messages in thread From: Eric Schulte @ 2008-10-23 23:57 UTC (permalink / raw) To: Robert Goldman; +Cc: emacs-orgmode I've added a "testing tools" section and a simple example of an org-mode test using ert to Sebastian's org testing page in worg. I don't mean to advocate for ert, it is simply the only elisp testing framework that I have any experience with. I'm pessimistic that the colossal amount of work involved in wrapping all of org-modes extensive and varied functionality into a test framework can/should actually be completed. Would it be sufficient to simply begin dealing with bug reports in a test-first manner? Also, should we start tracking bug reports somewhere (worg), so that they can be claimed, tested against, and repaired? Thanks -- Eric Robert Goldman <rpgoldman@sift.info> writes: > There's a page on the EmacsWiki describing schemes for unit/regression > testing in emacs lisp: > > http://www.emacswiki.org/cgi-bin/wiki/UnitTesting > > Perhaps we could pick up one of these? I haven't evaluated any of them. > > In response to Ben Alexander's question, yes, sometimes the screen > output is what you care about, but a big advantage of emacs is that the > screen output is a big array of text, so we could check it (it's a lot > worse, if it's just a bit map...). Sure, we can't always do that, but I > think in many cases we could just grab up a buffer's contents and > compare it against a "right answer." > > best, > r > > > > _______________________________________________ > Emacs-orgmode mailing list > Remember: use `Reply All' to send replies to the list. > Emacs-orgmode@gnu.org > http://lists.gnu.org/mailman/listinfo/emacs-orgmode ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Regression testing for org-mode 2008-10-23 23:57 ` Eric Schulte @ 2008-10-24 0:03 ` Avdi Grimm 2008-10-24 2:41 ` Robert Goldman 0 siblings, 1 reply; 10+ messages in thread From: Avdi Grimm @ 2008-10-24 0:03 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode, Robert Goldman On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote: > > I'm pessimistic that the colossal amount of work involved in wrapping > all of org-modes extensive and varied functionality into a test > framework can/should actually be completed. Would it be sufficient to > simply begin dealing with bug reports in a test-first manner? Absolutely. I do NOT recommend making an effort to get test coverage over existing code just for the sake of coverage - in my experience this usually leads a lot of work for brittle and not terribly useful tests. Put tests only around the areas you are about to change. Over time the coverage will grow. > Also, should we start tracking bug reports somewhere (worg), so that > they can be claimed, tested against, and repaired? Not a bad idea. Normally I'd recommend just going with an established bug tracker like Trac or Lighthouse, but since Org is so great for managing tasks it seems only right that the developers should "eat their own dogfood" by using Org to track tickets :-) -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Regression testing for org-mode 2008-10-24 0:03 ` Avdi Grimm @ 2008-10-24 2:41 ` Robert Goldman 2008-10-24 3:04 ` worg for bug reports and feature requests was: (Regression testing for org-mode) Eric Schulte 0 siblings, 1 reply; 10+ messages in thread From: Robert Goldman @ 2008-10-24 2:41 UTC (permalink / raw) To: Avdi Grimm; +Cc: emacs-orgmode Avdi Grimm wrote: > On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote: >> I'm pessimistic that the colossal amount of work involved in wrapping >> all of org-modes extensive and varied functionality into a test >> framework can/should actually be completed. Would it be sufficient to >> simply begin dealing with bug reports in a test-first manner? > > Absolutely. I do NOT recommend making an effort to get test coverage > over existing code just for the sake of coverage - in my experience > this usually leads a lot of work for brittle and not terribly useful > tests. Put tests only around the areas you are about to change. Over > time the coverage will grow. > >> Also, should we start tracking bug reports somewhere (worg), so that >> they can be claimed, tested against, and repaired? > > Not a bad idea. Normally I'd recommend just going with an established > bug tracker like Trac or Lighthouse, but since Org is so great for > managing tasks it seems only right that the developers should "eat > their own dogfood" by using Org to track tickets :-) > Actually, I'm not sure I necessarily agree with the notion of using Org to track tickets. The reason is not that org mightn't be up to the job, but that the use of org with git won't be up to it. systems like trac and bugzilla are set up to allow outsiders to post bugs, but if we use git, then we're really raising the bar for bug submission. Instead of filling out a form, a bug reporter would now have to figure out how to use git, pull the org file, modify it, and then either push it (which would require someone to authorize him or her) or submit it to someone else who would push it. That seems inappropriate to me --- when you're developing software a good bug report is very valuable, and one shouldn't turn them away. Unless someone can figure out an easier way for people to submit bugs with what worg has now, I'm inclined to say that trac or bugzilla would be better. Best, r ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 2:41 ` Robert Goldman @ 2008-10-24 3:04 ` Eric Schulte 2008-10-24 4:42 ` Robert Goldman 0 siblings, 1 reply; 10+ messages in thread From: Eric Schulte @ 2008-10-24 3:04 UTC (permalink / raw) To: Robert Goldman; +Cc: emacs-orgmode Robert Goldman <rpgoldman@sift.info> writes: > Avdi Grimm wrote: >> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote: >>> Also, should we start tracking bug reports somewhere (worg), so that >>> they can be claimed, tested against, and repaired? >> >> Not a bad idea. Normally I'd recommend just going with an established >> bug tracker like Trac or Lighthouse, but since Org is so great for >> managing tasks it seems only right that the developers should "eat >> their own dogfood" by using Org to track tickets :-) >> > > Actually, I'm not sure I necessarily agree with the notion of using Org > to track tickets. The reason is not that org mightn't be up to the job, > but that the use of org with git won't be up to it. systems like trac > and bugzilla are set up to allow outsiders to post bugs, but if we use > git, then we're really raising the bar for bug submission. Instead of > filling out a form, a bug reporter would now have to figure out how to > use git, pull the org file, modify it, and then either push it (which > would require someone to authorize him or her) or submit it to someone > else who would push it. That seems inappropriate to me --- when you're > developing software a good bug report is very valuable, and one > shouldn't turn them away. > > Unless someone can figure out an easier way for people to submit bugs > with what worg has now, I'm inclined to say that trac or bugzilla would > be better. > Yes, you're correct the current method of contributing to worg is definitely too high of a bar for bug reports, or feature requests. That said once they were submitted, worg would be a good mechanism for tracking reported bugs/features, and for publishing lists of said reports on the web. Worg/org has the added advantage that it is already familiar to the entire org community. As you mention, the question seem to be: can we implement a simple interface for reporting bugs/feature-recs which will? 1) work well with worg as it's currently used, and 2) which won't constantly be begging for enhancements until we're re-implementing a full bug tracking system from scratch I don't think this would be too difficult, say... a simple web form, which could be embedded into one of the worg pages, and could drop bug/feature-recs into an org-mode file under git. Would anyone be up to trying to throw such a thing together? If only I had some more time... Thanks -- Eric ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 3:04 ` worg for bug reports and feature requests was: (Regression testing for org-mode) Eric Schulte @ 2008-10-24 4:42 ` Robert Goldman 2008-10-24 14:12 ` Sebastian Rose 0 siblings, 1 reply; 10+ messages in thread From: Robert Goldman @ 2008-10-24 4:42 UTC (permalink / raw) To: Eric Schulte; +Cc: emacs-orgmode Eric Schulte wrote: > Robert Goldman <rpgoldman@sift.info> writes: > >> Avdi Grimm wrote: >>> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote: >>>> Also, should we start tracking bug reports somewhere (worg), so that >>>> they can be claimed, tested against, and repaired? >>> Not a bad idea. Normally I'd recommend just going with an established >>> bug tracker like Trac or Lighthouse, but since Org is so great for >>> managing tasks it seems only right that the developers should "eat >>> their own dogfood" by using Org to track tickets :-) >>> >> Actually, I'm not sure I necessarily agree with the notion of using Org >> to track tickets. The reason is not that org mightn't be up to the job, >> but that the use of org with git won't be up to it. systems like trac >> and bugzilla are set up to allow outsiders to post bugs, but if we use >> git, then we're really raising the bar for bug submission. Instead of >> filling out a form, a bug reporter would now have to figure out how to >> use git, pull the org file, modify it, and then either push it (which >> would require someone to authorize him or her) or submit it to someone >> else who would push it. That seems inappropriate to me --- when you're >> developing software a good bug report is very valuable, and one >> shouldn't turn them away. >> >> Unless someone can figure out an easier way for people to submit bugs >> with what worg has now, I'm inclined to say that trac or bugzilla would >> be better. >> > > Yes, you're correct the current method of contributing to worg is > definitely too high of a bar for bug reports, or feature requests. That > said once they were submitted, worg would be a good mechanism for > tracking reported bugs/features, and for publishing lists of said > reports on the web. Worg/org has the added advantage that it is already > familiar to the entire org community. > > As you mention, the question seem to be: can we implement a simple > interface for reporting bugs/feature-recs which will? > > 1) work well with worg as it's currently used, and > 2) which won't constantly be begging for enhancements until we're > re-implementing a full bug tracking system from scratch > > I don't think this would be too difficult, say... > > a simple web form, which could be embedded into one of the worg pages, > and could drop bug/feature-recs into an org-mode file under git. > Would anyone be up to trying to throw such a thing together? If only > I had some more time... > I don't mean to seem like a party-pooper, but I would suggest that we not do this. Getting even the simple web form done right involves thinking hard about dealing with spam bots, people trying to hack your site, etc. The people who built bugzilla and trac (and apache) have a lot more experience with this kind of issue than we do (at least speaking for myself), and have had a lot of time to shake the kinks out. If you want a real ticket system, I'd suggest using one of these rather than rolling another one. Best, r ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 4:42 ` Robert Goldman @ 2008-10-24 14:12 ` Sebastian Rose 2008-10-24 14:27 ` Avdi Grimm 2008-10-24 20:44 ` Eric Schulte 0 siblings, 2 replies; 10+ messages in thread From: Sebastian Rose @ 2008-10-24 14:12 UTC (permalink / raw) To: emacs-orgmode While full fletched bug tracker is a _very_ usefull tool for developers, it's a drag for users. As for the users, I'm not shure if we should urge them to use a bug tracking system at all. The list works so fine and natural. It's is quite low/medium traffic with those bug reports. As Richard Riley statud lately, it often turns out, that the 'bug' boiles down to some configuration problem. Lots of testing is done by the users, simply by using org. Since we still don't have a complete set of automated tests, this will be the case for quite some time. And, the list is, what everyone reads. Sometimes new people show up, read about a bug and provide patches. Why 'hide' the bugs somewhere? That said, I still believe a bug tracker as an 'internal' tool for the testers/developers would be of _great_ help. IMO, it would definitively make sense to use bugzilla or track for that. I don't want to implement all those features from scratch, that make a bug tracker usefull. Where will such a system live? Who installs and maintains it? Robert Goldman wrote: > Eric Schulte wrote: >> Robert Goldman <rpgoldman@sift.info> writes: >> >>> Avdi Grimm wrote: >>>> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote: >>>>> Also, should we start tracking bug reports somewhere (worg), so that >>>>> they can be claimed, tested against, and repaired? >>>> Not a bad idea. Normally I'd recommend just going with an established >>>> bug tracker like Trac or Lighthouse, but since Org is so great for >>>> managing tasks it seems only right that the developers should "eat >>>> their own dogfood" by using Org to track tickets :-) >>>> >>> Actually, I'm not sure I necessarily agree with the notion of using Org >>> to track tickets. The reason is not that org mightn't be up to the job, >>> but that the use of org with git won't be up to it. systems like trac >>> and bugzilla are set up to allow outsiders to post bugs, but if we use >>> git, then we're really raising the bar for bug submission. Instead of >>> filling out a form, a bug reporter would now have to figure out how to >>> use git, pull the org file, modify it, and then either push it (which >>> would require someone to authorize him or her) or submit it to someone >>> else who would push it. That seems inappropriate to me --- when you're >>> developing software a good bug report is very valuable, and one >>> shouldn't turn them away. >>> >>> Unless someone can figure out an easier way for people to submit bugs >>> with what worg has now, I'm inclined to say that trac or bugzilla would >>> be better. >>> >> Yes, you're correct the current method of contributing to worg is >> definitely too high of a bar for bug reports, or feature requests. That >> said once they were submitted, worg would be a good mechanism for >> tracking reported bugs/features, and for publishing lists of said >> reports on the web. Worg/org has the added advantage that it is already >> familiar to the entire org community. >> >> As you mention, the question seem to be: can we implement a simple >> interface for reporting bugs/feature-recs which will? >> >> 1) work well with worg as it's currently used, and >> 2) which won't constantly be begging for enhancements until we're >> re-implementing a full bug tracking system from scratch >> >> I don't think this would be too difficult, say... >> >> a simple web form, which could be embedded into one of the worg pages, >> and could drop bug/feature-recs into an org-mode file under git. >> Would anyone be up to trying to throw such a thing together? If only >> I had some more time... >> > > > > I don't mean to seem like a party-pooper, but I would suggest that we > not do this. Getting even the simple web form done right involves > thinking hard about dealing with spam bots, people trying to hack your > site, etc. The people who built bugzilla and trac (and apache) have a > lot more experience with this kind of issue than we do (at least > speaking for myself), and have had a lot of time to shake the kinks out. > If you want a real ticket system, I'd suggest using one of these rather > than rolling another one. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 14:12 ` Sebastian Rose @ 2008-10-24 14:27 ` Avdi Grimm 2008-10-24 20:44 ` Eric Schulte 1 sibling, 0 replies; 10+ messages in thread From: Avdi Grimm @ 2008-10-24 14:27 UTC (permalink / raw) To: sebastian_rose; +Cc: emacs-orgmode On Fri, Oct 24, 2008 at 10:12 AM, Sebastian Rose <sebastian_rose@gmx.de> wrote: > Where will such a system live? > Who installs and maintains it? If someone actually wants to install and host the service, I recommend using Trac. It's open source, mature, full-featured, and used on gobs of projects. My place of employment uses it and we are quite happy with it. If you want to go the hosted ruote, there are a few possibilities: - A SourceForge[1] account includes a ticket tracker. - Presumably Savannah[2] (a GNU SourceForge clone) provides the same. - Lighthouse[3] provides free hosted ticket tracking for open-source projects. Personally I'm not a huge fan of Lighthouse, but they've been gaining popularity recently. [1] http://sourceforge.net/ [2] http://savannah.nongnu.org/ [3] http://lighthouseapp.com/ -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 14:12 ` Sebastian Rose 2008-10-24 14:27 ` Avdi Grimm @ 2008-10-24 20:44 ` Eric Schulte 2008-10-24 23:54 ` Eric Schulte 1 sibling, 1 reply; 10+ messages in thread From: Eric Schulte @ 2008-10-24 20:44 UTC (permalink / raw) To: sebastian_rose; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1366 bytes --] Sebastian Rose <sebastian_rose@gmx.de> writes: > And, the list is, what everyone reads. Sometimes new people show up, > read about a bug and provide patches. Why 'hide' the bugs somewhere? > Point taken, the list has worked very well, and there is probably no need to change it. That said I couldn't help myself... and implemented a very simple little web-form which can accept error reports, and append them to a "bugs.org" org-file. This gives every new report it's own headline in the outline, uses TODO keywords to track the status of the report, uses properties to track information such as the type of the report, and who is responsible for completion, and it comments out the users input using the "^: " syntax, to thwart any malicious inputs. I know, I know, we should use a mature bug tracking system or none at all, but org seems so well suited, and this is so small. How much trouble could it cause? :) > Where will such a system live? > Who installs and maintains it? also, this solves the above problems, because if such a page were to live as part of worg, then the resulting bugs.org files could live in the worg git repo, and be maintained by worgers... Thanks -- Eric The attached mini-web-application is a ruby "camping" application, it will build and maintain a bug file named "bugs.org" in the same directory the script is located in. [-- Attachment #2: org_bugs.rb --] [-- Type: application/octet-stream, Size: 3198 bytes --] #!/usr/bin/env ruby require 'rubygems' require 'camping' ## Setup Camping.goes :OrgBugs ORG_FILE = File.join(File.expand_path(File.dirname(__FILE__)), "bugs.org") DAYS_OF_THE_WEEK = %w[Sun Mon Tue Wed Thu Fri Sat] ## Views module OrgBugs::Views def index html do head do title "Org-Mode Bug Reports and Feature Requests" style "label {font-weight: bold;}" end body do h2 "Org-Mode Bug Reports and Feature Requests" form(:action => "submit", :method => "post", :enctype => "multipart/form-data") do p do label "Title (required) " input({:name => "title", :id => "title", :type => 'text_field', :size => 52}) br label "Name (required)" input({:name => "name", :id => "name", :type => 'text_field', :size => 52}) br label "Email (required)" input({:name => "email", :id => "email", :type => 'text_field', :size => 52}) br label "Type (required) " tag! :select, :name => "report_type", :id => "report_type" do tag! :option, "bug" tag! :option, "feature request" tag! :option, "other" end br p do "Please see the <a href=\"http://orgmode.org/org.html#Feedback\">Feedback</a> "+ "section of the org-mode manual for information on </br> how to submit a useful bug report. "+ "Don't forget to mention..." end ol do li {"What exactly did you do? "} li {"What did you expect to happen? "} li {"What happened instead? "} end label "Description (required)" br textarea({:name => 'desc', :id => 'desc', :cols => 80, :rows => 16}) br input(:type => "submit", :value => "Submit") end end end end end def thanks html do head do title "Thanks for the Report!" style "label {font-weight: bold;}" end body do h1 "Report Submitted" p {"Thanks <i>#{@input.name}</i> your report <i>#{@input.title}</i> "+ "has been submitted, and will be reviewed."} end end end end ## Controllers module OrgBugs::Controllers class Index < R '/' def get; render :index end end class Submit < R '/submit' def post report_to_org(@input) render :thanks end end end def write_prop(prop, value) " :#{prop}: #{value}\n" end def report_to_org(input) File.open(ORG_FILE, "a") do |f| f << "** SUBMITTED #{input.title}\n" f << " :PROPERTIES:\n" f << write_prop(:submitter, input.name) f << write_prop(:submitter_email, input.email) f << write_prop(:date, DateTime.now.strftime("<%Y-%m-%d -%w- %H:%M>"). sub(/-\d-/, DAYS_OF_THE_WEEK[Integer($1)])) f << write_prop(:type, input.report_type) f << write_prop(:assigned_to, "no-one") f << " :END:\n" f << "\n" f << input.desc.split(/[\n\r]/).map{|line| ": #{line.chomp}\n" if line.size > 0}.compact.join("") f << "\n" end end [-- Attachment #3: Type: text/plain, Size: 204 bytes --] _______________________________________________ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode) 2008-10-24 20:44 ` Eric Schulte @ 2008-10-24 23:54 ` Eric Schulte 0 siblings, 0 replies; 10+ messages in thread From: Eric Schulte @ 2008-10-24 23:54 UTC (permalink / raw) To: sebastian_rose; +Cc: emacs-orgmode "Eric Schulte" <schulte.eric@gmail.com> writes: > Sebastian Rose <sebastian_rose@gmx.de> writes: >> And, the list is, what everyone reads. Sometimes new people show up, >> read about a bug and provide patches. Why 'hide' the bugs somewhere? >> > > Point taken, the list has worked very well, and there is probably no > need to change it. > > That said I couldn't help myself... and implemented a very simple little > web-form which can accept error reports, and append them to a "bugs.org" > org-file. This gives every new report it's own headline in the outline, > uses TODO keywords to track the status of the report, uses properties to > track information such as the type of the report, and who is responsible > for completion, and it comments out the users input using the "^: " > syntax, to thwart any malicious inputs. I know, I know, we should use a > mature bug tracking system or none at all, but org seems so well suited, > and this is so small. How much trouble could it cause? :) > >> Where will such a system live? >> Who installs and maintains it? > > also, this solves the above problems, because if such a page were to > live as part of worg, then the resulting bugs.org files could live in > the worg git repo, and be maintained by worgers... > > Thanks -- Eric > > The attached mini-web-application is a ruby "camping" application, it > will build and maintain a bug file named "bugs.org" in the same > directory the script is located in. figured I'd actually put this up, as a better demonstration than just the source. It's just running on a server in my study, so please if you see any glaring security holes, don't kill my machine. http://org-bug.suey.ath.cx/ Thanks -- Eric ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2008-10-24 23:54 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2008-10-23 16:03 Regression testing for org-mode Robert Goldman 2008-10-23 23:57 ` Eric Schulte 2008-10-24 0:03 ` Avdi Grimm 2008-10-24 2:41 ` Robert Goldman 2008-10-24 3:04 ` worg for bug reports and feature requests was: (Regression testing for org-mode) Eric Schulte 2008-10-24 4:42 ` Robert Goldman 2008-10-24 14:12 ` Sebastian Rose 2008-10-24 14:27 ` Avdi Grimm 2008-10-24 20:44 ` Eric Schulte 2008-10-24 23:54 ` Eric Schulte
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).