Re: RFC: Improvements to org-remember

From: Carsten Dominik <dominik@science.uva.nl>
To: James TD Smith <ahktenzero@mohorovi.cc>
Cc: emacs-orgmode@gnu.org
Subject: Re: RFC: Improvements to org-remember
Date: Mon, 24 Nov 2008 09:58:49 +0100	[thread overview]
Message-ID: <EF11EE54-90F7-41DA-8134-68DAA73EC7F9@uva.nl> (raw)
In-Reply-To: <20081123232557.GC62148@yog-sothoth.mohorovi.cc>

Hi James, I do like all this.  A few comments:

On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:

> I have a number of improvements to org-remember I am planning to  
> implement. I
> have briefly discussed some of them with Carsten, and I thought I  
> should post a
> detailed description here for discussion as I'm sure you will have  
> ideas,
> suggestions and criticisms of the proposed changes.
>
> * Preliminary work
> ** Refactoring
>
> Carsten suggested that the remember handler could do with some  
> refactoring
> before adding any new features, as it has gotten quite complicated  
> in places.
>
> The longest functions are `org-remember-apply-template' and
> `org-remember-handler', so these are probably the best place to start.
>
> I think it would make sense to move the code to get values for  
> remember
> expansions out of `org-remember-apply-template' into separate  
> functions. These
> could be added to an association list indexed by the expansion  
> character. This
> would also make it easier to add new expansions.

Yes.  However, it is necessary o keep the sequence of handling the  
escapes,
in particular first filling in all non-interactive ones, and only then
doing the interactive ones.

>
>
> ** Unit tests
>
>  If we can decide on a unit testing framework, this would be an ideal
>  opportunity to add some test coverage, as I'll need to do extensive  
> testing
>  to ensure I don't break anything with the refactoring or the new  
> features. I
>  have some experience with unit testing using JUnit.
>
> ** Plists for remember templates

Ah, this will be a big relieve when it is implemented,
should have been like this from the start.

>
>
>  I want to change the format of remember templates to use plists.  
> This is to
>  allow introducing a number of optional parameters to control the  
> new features
>  I want to add. Org uses plists elsewhere, for example in the # 
> +OPTIONS:
>  configuration header, and #+PLOT: lines, so the syntax should  
> already be
>  familiar to org users.
>
>  I also think it would make sense to to move some options which are  
> currently
>  set using escapes outside of the template, specifically "%!" (store
>  template immediately) and "%&" (jump to entry after storing).

Yes, this wil be much better.

>  For backwards compatibility, the current template format would  
> still be
>  supported, but the additional options would not be available.  
> Defaults for the
>  extra parameters would be set so if they are not present the  
> templates would
>  work as they do currently.
>
>  The current options would be represented as below:
>  - :template :: the template itself
>  - :name :: the name of the template
>  - :target :: The :target parameter takes the place of the current  
> file and
> 		headline target specification. The parameter specifies only the
> 		default target; all the other options will remain available via
> 		numeric prefixes to C-c C-c. The available options are:
>    - "<filename>:<heading>" :: a file target. If the heading is  
> omitted,
> 	  			 a top-level heading will be created.

That should be a double colon, for symmetry with org-archive-location,
and to avoid problems with Windows paths.

>
>    - clock :: currently clocked task
>    - current :: add under the the org headline where the point is.
>    - interactive :: select a location interactively using the  
> appropriate
> 		      interface
>    - <function> :: call function to get the target. The function can  
> return
> 		     either a marker, or a file/headline string.
>  - :test :: a function, or list of major modes. The template is only
> 	      available if the function returns true, or the current
> 	      buffer is in one of the appropriate modes
>  - :immediate :: replaces the %! escape; if t, the template is  
> stored as soon
> 		   as all escapes are replaced.
>  - :jumpto :: replaces the %& escape; if t, org jumps to the target  
> location
> 		when the note is stored.
>
> * New features
> ** Adding non-headline items
>
>  For some time I have wanted to be able to use remember to add  
> checklist
>  entries and table rows as well as org headlines. To configure this,  
> a :type
>  parameter will be added to the template, which can be either  
> headline (the
>  default), list, checklist or table.
>
>  - Table rows.

This is an awesome idea, as are plain list items and checkboxes.

For plain list items, I guess the right thing would be to select the
first plain list under the headline, there might be several.
Also, the first table under a headline, in case there are more.

While one could have a property for explicitly selecting a type
like table row or plain-list item or checkbox,  it would also be
possible to derive this from the Remember buffer content automatically.
Which method is better?

>
>
>    Currently if you want to use org to record periodic measurements  
> (for
>    example see the thread about using org to manage fitness  
> training), you
>    have to use properties and column view, which has a number of  
> limitations
>    (speed, calculations). Being able to add table rows via remember  
> would make
>    it much easier to do this.
>
>    The simplest implementation would use a template containing the  
> appropriate
>    number of table columns, for example something like
>
>    "| %U | %^{Value 1} | %^{Value 2} |"
>
>    This would be added to the table at the appropriate position  
> (depending on
>    the :prepend value for the template, and then formatted properly  
> using
>    `org-table-align'.
>
>    The handler would also need to ensure that table formulae get  
> updated
>    (increment row ranges etc) and that values are recalculated (if  
> automatic
>    recalculation is enabled) after the line is added.
>
>    An extension to this would be to include a truncated copy of the  
> table in
>    the remember buffer, with just the headers (and possibly  
> formulae) from the
>    target table, so the user could add multiple lines in the  
> remember buffer
>    and then add them to the table.

I guess you mean here to retrive that truncated copy from the target  
location?

>
>
>  - Checklist entries
>
>    I use checklists a lot for breaking down tasks. Being able to add  
> checklist
>    entries using remember, with the clocked task as a target, would  
> make it
>    possible to do this without having to switch away from the buffer  
> I was
>    working in.
>
>  - Plain list entries.
>
>    It also be useful to be able to add plain list items. This would  
> be like a
>    more featureful version of `org-add-note', with the full range of  
> remember
>    expansions available.

I would like to add here

   - Non-org items, to simply be appended to a file.
     I believe Russel Adams had a request about this, on how to use
     remember to add entries to a ledger file, for which using a
     template would be nice.  The problem is that Org currently requires
     the target file to be in Org-mode - for good reasons, because all
     kinds of Org stuff will be executed when the new entry is added.
     We should allow for the results of a remember template to be added
     to a non-Org file.

   - With these new options, will we need to have optional two-key
     access to templates, or is single key enough?  Maybe only at a  
later
     stage if it is felt that this is really necessary.

> ** Updating completion statistics
>
>  If the target for a remember template contains either of the  
> completion
>  status tracking cookies ([/}, [%]), these should be updated after  
> the entry
>  has been added. This should apply to both entries containing  
> checklists, and
>  entries containing other headlines.
>
>  Also, target matching for templates should ignore the completion  
> status
>  cookies.
>
> ** Per-template insertion order
>
>  It is currently possible to configure a remember template to add  
> entries at
>  the start or end of the target only if the target is a file, or set  
> globally
>  with `org-reverse-note-order'. It would be useful to be able to set  
> this for
>  individual templates.
>
>  An optional :prepend parameter would control where the entry would be
>  inserted. If set to t the new entry would be the first under the  
> heading, and
>  the last the if nil. The default would be whatever `org-reverse- 
> note-order'
>  is set to.
>
> ** Automatic sorting
>
>  It would be useful to be able to automatically sort the target's  
> entries
>  after adding one using remember, for example if you want to keep  
> entries
>  sorted by TODO keyword or in alphabetical order.
>
>  This would require two parameters:
>  - :sort :: if non-nil, sort the contents of the headline the item was
> 	       added under with the specified function. valid options are
>    - todoorder :: sort by TODO state
>    - alpha :: sort alphabetically
>    - numeric :: sort numerically
>    - priority :: sort by priority
>    - time :: sort by time
>    - property :: sort by property
>    - <function> :: use function to determine sort order
>    - Additionally, any of the previous items postfixed with '-rev'  
> will sort
>      in reverse
>  - :sortkey :: for tables, and property searches, this specifies  
> either the
> 		 column or property to sort by.

Another option I would like to see is, how many empty lines should
be inserted before the entry.  Because sometimes it is nice to have
an empty line between entries, and sometimes not.  Default should be
no empty lines.

> * Development plan
>
> It will take me quite a while to implement all this so I plan on  
> doing it in
> stages. After the preliminary work is done, each new feature will be
> implemented in a separate git branch which can be merged into the  
> master
> branch when it's finished and properly tested. I'll post an update  
> to the list
> when each feature is ready for testing.

Yes, this sounds good.  The one thing we need to keep in mind is
that a significant number of people by now really depend on the
templates working, so we cannot break org-remember at any stage for
the normal user.  So my proposal would be to create a new branch in our
main repository, a branch James will be the only one to push to, using
patches and ideas from the rest of us.  If this branch is frequently
rebased to the main Org-mode master, testers could easily and without
loss of other new features switch between the Org master and the
remember branch.

Does that sound good, or would you prefer to use your own repo
for this, James?  I believe the number of testers will be larger
if you use our main repo.  If you give me your user name on
repo.or.cz, I will give you push access.

- Carsten

>
>
> James
>
> --
> |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode