RFC: Improvements to org-remember

RFC: Improvements to org-remember

[James TD Smith]

I have a number of improvements to org-remember I am planning to implement. I
have briefly discussed some of them with Carsten, and I thought I should post a
detailed description here for discussion as I'm sure you will have ideas,
suggestions and criticisms of the proposed changes.

* Preliminary work
** Refactoring

  Carsten suggested that the remember handler could do with some refactoring
  before adding any new features, as it has gotten quite complicated in places.

  The longest functions are `org-remember-apply-template' and
  `org-remember-handler', so these are probably the best place to start.

  I think it would make sense to move the code to get values for remember
  expansions out of `org-remember-apply-template' into separate functions. These
  could be added to an association list indexed by the expansion character. This
  would also make it easier to add new expansions.

** Unit tests

   If we can decide on a unit testing framework, this would be an ideal
   opportunity to add some test coverage, as I'll need to do extensive testing
   to ensure I don't break anything with the refactoring or the new features. I
   have some experience with unit testing using JUnit.
   
** Plists for remember templates

   I want to change the format of remember templates to use plists. This is to
   allow introducing a number of optional parameters to control the new features
   I want to add. Org uses plists elsewhere, for example in the #+OPTIONS:
   configuration header, and #+PLOT: lines, so the syntax should already be
   familiar to org users.

   I also think it would make sense to to move some options which are currently
   set using escapes outside of the template, specifically "%!" (store
   template immediately) and "%&" (jump to entry after storing).

   For backwards compatibility, the current template format would still be
   supported, but the additional options would not be available. Defaults for the
   extra parameters would be set so if they are not present the templates would
   work as they do currently.

   The current options would be represented as below:
   - :template :: the template itself
   - :name :: the name of the template
   - :target :: The :target parameter takes the place of the current file and
		headline target specification. The parameter specifies only the
		default target; all the other options will remain available via
		numeric prefixes to C-c C-c. The available options are:
     - "<filename>:<heading>" :: a file target. If the heading is omitted,
	  			 a top-level heading will be created.
     - clock :: currently clocked task
     - current :: add under the the org headline where the point is.
     - interactive :: select a location interactively using the appropriate
		      interface
     - <function> :: call function to get the target. The function can return
		     either a marker, or a file/headline string.
   - :test :: a function, or list of major modes. The template is only
	      available if the function returns true, or the current
	      buffer is in one of the appropriate modes
   - :immediate :: replaces the %! escape; if t, the template is stored as soon
		   as all escapes are replaced.
   - :jumpto :: replaces the %& escape; if t, org jumps to the target location
		when the note is stored.

* New features
** Adding non-headline items

   For some time I have wanted to be able to use remember to add checklist
   entries and table rows as well as org headlines. To configure this, a :type
   parameter will be added to the template, which can be either headline (the
   default), list, checklist or table.

   - Table rows.

     Currently if you want to use org to record periodic measurements (for
     example see the thread about using org to manage fitness training), you
     have to use properties and column view, which has a number of limitations
     (speed, calculations). Being able to add table rows via remember would make
     it much easier to do this.

     The simplest implementation would use a template containing the appropriate
     number of table columns, for example something like

     "| %U | %^{Value 1} | %^{Value 2} |"

     This would be added to the table at the appropriate position (depending on
     the :prepend value for the template, and then formatted properly using
     `org-table-align'.

     The handler would also need to ensure that table formulae get updated
     (increment row ranges etc) and that values are recalculated (if automatic
     recalculation is enabled) after the line is added.

     An extension to this would be to include a truncated copy of the table in
     the remember buffer, with just the headers (and possibly formulae) from the
     target table, so the user could add multiple lines in the remember buffer
     and then add them to the table.

   - Checklist entries

     I use checklists a lot for breaking down tasks. Being able to add checklist
     entries using remember, with the clocked task as a target, would make it
     possible to do this without having to switch away from the buffer I was
     working in.

   - Plain list entries.

     It also be useful to be able to add plain list items. This would be like a
     more featureful version of `org-add-note', with the full range of remember
     expansions available.

** Updating completion statistics

   If the target for a remember template contains either of the completion
   status tracking cookies ([/}, [%]), these should be updated after the entry
   has been added. This should apply to both entries containing checklists, and
   entries containing other headlines.

   Also, target matching for templates should ignore the completion status
   cookies.

** Per-template insertion order

   It is currently possible to configure a remember template to add entries at
   the start or end of the target only if the target is a file, or set globally
   with `org-reverse-note-order'. It would be useful to be able to set this for
   individual templates.

   An optional :prepend parameter would control where the entry would be
   inserted. If set to t the new entry would be the first under the heading, and
   the last the if nil. The default would be whatever `org-reverse-note-order'
   is set to.

** Automatic sorting

   It would be useful to be able to automatically sort the target's entries
   after adding one using remember, for example if you want to keep entries
   sorted by TODO keyword or in alphabetical order.

   This would require two parameters:
   - :sort :: if non-nil, sort the contents of the headline the item was
	       added under with the specified function. valid options are
     - todoorder :: sort by TODO state
     - alpha :: sort alphabetically
     - numeric :: sort numerically
     - priority :: sort by priority
     - time :: sort by time
     - property :: sort by property
     - <function> :: use function to determine sort order
     - Additionally, any of the previous items postfixed with '-rev' will sort
       in reverse
   - :sortkey :: for tables, and property searches, this specifies either the
		 column or property to sort by.

* Development plan

  It will take me quite a while to implement all this so I plan on doing it in
  stages. After the preliminary work is done, each new feature will be
  implemented in a separate git branch which can be merged into the master
  branch when it's finished and properly tested. I'll post an update to the list
  when each feature is ready for testing.
  
James

--
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|

Re: RFC: Improvements to org-remember

[Samuel Wales]

All of this looks great.  I especially like code integrity, plist
syntax, and :prefix.

Do you want more ideas for remember?

Re: RFC: Improvements to org-remember

[James TD Smith]

Hi Samuel,

On 2008-11-23 17:23:15(-0700), Samuel Wales wrote:
> All of this looks great.  I especially like code integrity, plist
> syntax, and :prefix.
> 
> Do you want more ideas for remember?

Yes. Any suggestions for new features or ways the behaviour of the remember
handler could be improved are most welcome. 
 
James

-- 
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|

Re: RFC: Improvements to org-remember

[Samuel Wales]

On Mon, Nov 24, 2008 at 03:02, James TD Smith <ahktenzero@mohorovi.cc> wrote:
> Yes. Any suggestions for new features or ways the behaviour of the remember
> handler could be improved are most welcome.

In addition to the stuff already mentioned:

************* TODO remember ideas
*************** respect pop-up-windows
*************** if the target file for org-remember is already known
use the control variables from that file
that way you can have your todo sequence available
*************** if org-remember does not recognize the type,
abort completely
*************** org-remember should be reentrant
able to call itself from inside itself if you have a note
you want to add that is not related to the one you are
adding.
*************** org-remember-templates takes a character
can it take a function key?
*************** emacs-w3m tight integration with org-mode
  - might be interesting to use an org-mode file to store
    bookmarks.  this would require changing the way
    bookmarks are added, to store them in a way similar to
    org-remember.
  - perhaps antenna can also be integrated with org-mode.

[2008-09-20 Sat 18:26]
*************** org merge org-annotate-file with remember code
to allow annotating anything
also have a hook for opening files and w3m pages etc. that
will print in the minibuffer "this file/page/directory is
annotated.  press ... to see the annotation".

[2008-10-27 Mon 22:02]
[[http://orgmode.org/manual/Extensions-in-the-contrib-directory.html][Extensions
in the contrib directory - The Org Manual]]
*************** brainstorm: support asking for the template
after the note was entered.

this might complicate things too much.

this is a tricky one to design, but the philosophy is that
the time between having an idea and entering it should be
minimal.  choosing the template type is a cognitive burden
before you enter the idea.

so dedicate a remember shortcut to the concept of "let me
enter this now".  c-c c-c, it asks you details like is this
a todo item?  and which file does it go to?

you would have a remember template that allows for other
remember templates to be chosen after you enter the note.

ideally, it works like this:

  1.  call org-remember like that
  2.  enter note
  3.  c-c c-c
  4.  choose whether it's a note, journal, or shopping item
  5.  if it's note or journal, choose whether it's todo
  6.  if it's note or journal, choose tags (RET for none)
  7.  show completed buffer
  8.  y to accept; n to edit

nested plists?

P.S.  :prepend -- org-refile also needs this, separately from adding
notes in reverse order.

Re: RFC: Improvements to org-remember

[Carsten Dominik]

Hi Samuel,

this is hard to read.  Try, with the cursor on the first line:

C-c @        to select the subtree
C-c C-e a    to export to ASCII

The result is more readable as an Email text.

HTH

- Carsten


On Nov 24, 2008, at 8:29 PM, Samuel Wales wrote:

> On Mon, Nov 24, 2008 at 03:02, James TD Smith  
> <ahktenzero@mohorovi.cc> wrote:
>> Yes. Any suggestions for new features or ways the behaviour of the  
>> remember
>> handler could be improved are most welcome.
>
> In addition to the stuff already mentioned:
>
> ************* TODO remember ideas
> *************** respect pop-up-windows
> *************** if the target file for org-remember is already known
> use the control variables from that file
> that way you can have your todo sequence available
> *************** if org-remember does not recognize the type,
> abort completely
> *************** org-remember should be reentrant
> able to call itself from inside itself if you have a note
> you want to add that is not related to the one you are
> adding.
> *************** org-remember-templates takes a character
> can it take a function key?
> *************** emacs-w3m tight integration with org-mode
>  - might be interesting to use an org-mode file to store
>    bookmarks.  this would require changing the way
>    bookmarks are added, to store them in a way similar to
>    org-remember.
>  - perhaps antenna can also be integrated with org-mode.
>
> [2008-09-20 Sat 18:26]
> *************** org merge org-annotate-file with remember code
> to allow annotating anything
> also have a hook for opening files and w3m pages etc. that
> will print in the minibuffer "this file/page/directory is
> annotated.  press ... to see the annotation".
>
> [2008-10-27 Mon 22:02]
> [[http://orgmode.org/manual/Extensions-in-the-contrib-directory.html] 
> [Extensions
> in the contrib directory - The Org Manual]]
> *************** brainstorm: support asking for the template
> after the note was entered.
>
> this might complicate things too much.
>
> this is a tricky one to design, but the philosophy is that
> the time between having an idea and entering it should be
> minimal.  choosing the template type is a cognitive burden
> before you enter the idea.
>
> so dedicate a remember shortcut to the concept of "let me
> enter this now".  c-c c-c, it asks you details like is this
> a todo item?  and which file does it go to?
>
> you would have a remember template that allows for other
> remember templates to be chosen after you enter the note.
>
> ideally, it works like this:
>
>  1.  call org-remember like that
>  2.  enter note
>  3.  c-c c-c
>  4.  choose whether it's a note, journal, or shopping item
>  5.  if it's note or journal, choose whether it's todo
>  6.  if it's note or journal, choose tags (RET for none)
>  7.  show completed buffer
>  8.  y to accept; n to edit
>
> nested plists?
>
> P.S.  :prepend -- org-refile also needs this, separately from adding
> notes in reverse order.
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: RFC: Improvements to org-remember

[Samuel Wales]

I was trying to contribute while reducing typing to a minimum to
reduce pain, and ended up making it hard for you to read.  Apologies.

======

        remember ideas i've gathered over the past few months
        =====================================================

Author: tom <tom@noplace.invalid>
Date: 2008-11-24 14:38:05 MST


Table of Contents
=================


        + respect pop-up-windows
        + if the target file for org-remember is already known
          use the control variables from that file
          that way you can have your todo sequence available
        + if org-remember does not recognize the type,
          abort completely
        + org-remember should be reentrant
          able to call itself from inside itself if you have a note
          you want to add that is not related to the one you are
          adding.
        + org-remember-templates takes a character
          can it take a function key?
        + emacs-w3m tight integration with org-mode
          - might be interesting to use an org-mode file to store
            bookmarks.  this would require changing the way
            bookmarks are added, to store them in a way similar to
            org-remember.
          - perhaps antenna can also be integrated with org-mode.

          [2008-09-20 Sat 18:26]
        + org merge org-annotate-file with remember code
          to allow annotating anything
          also have a hook for opening files and w3m pages etc. that
          will print in the minibuffer "this file/page/directory is
          annotated.  press ... to see the annotation".

          [2008-10-27 Mon 22:02] [Extensions in the contrib directory
- The Org Manual]
        + brainstorm: support asking for the template
          after the note was entered.

          this might complicate things too much.

          this is a tricky one to design, but the philosophy is that
          the time between having an idea and entering it should be
          minimal.  choosing the template type is a cognitive burden
          before you enter the idea.

          so dedicate a remember shortcut to the concept of "let me
          enter this now".  c-c c-c, it asks you details like is this
          a todo item?  and which file does it go to?

          you would have a remember template that allows for other
          remember templates to be chosen after you enter the note.

          ideally, it works like this:

            1.  call org-remember like that
            2.  enter note
            3.  c-c c-c
            4.  choose whether it's a note, journal, or shopping item
            5.  if it's note or journal, choose whether it's todo
            6.  if it's note or journal, choose tags (RET for none)
            7.  show completed buffer
            8.  y to accept; n to edit

          nested plists?

-- 
Myalgic encephalomyelitis denialists are knowingly causing further
suffering and death by opposing biomedical research on this
fast-spreading serious disease.  Do you care about the world?
http://www.meactionuk.org.uk/What_Is_ME_What_Is_CFS.htm

Re: RFC: Improvements to org-remember

[James TD Smith]

Hi Samuel, 

I wasn't expecting this many ideas, thanks. There are some great suggestions
here, though I think some of them are outside the scope of what I am planning on
doing.

On 2008-11-24 14:41:00(-0700), Samuel Wales wrote:
> + respect pop-up-windows

Could you explain this in more detail? 

> + if the target file for org-remember is already known use the control
>   variables from that file that way you can have your todo sequence available

I'm definately adding this. Perhaps there should also be a parameter to set the
todo keywords to use for templates which don't have a known target.

> + if org-remember does not recognize the type, abort completely

Personally I'd prefer it to keep asking for a valid template if you enter a
character that doesn't correspond to any of the templates as most of the time I
do that I've hit the wrong key. You do know you can abort at the template
selection stage with C-g?

> + org-remember-templates takes a character
>   can it take a function key?

It would be possible to add this. Do you specifically want to use the F-keys, or
have you just run out of keys for templates? I'm planning on adding two-stage
selection for remember templates, which will make it possible to have many more
templates.

> + org-remember should be reentrant
>   able to call itself from inside itself if you have a note you want to add
>   that is not related to the one you are adding.

This is a limitation of remember itself, not the org remember handler. It would
be useful to be able to have multiple remember sessions running at once.

> + emacs-w3m tight integration with org-mode
>   - might be interesting to use an org-mode file to store bookmarks. this
>     would require changing the way bookmarks are added, to store them in a way
>     similar to org-remember.

I like the idea of using org-mode to store bookmarks. I use org-mode to keep
track of things I want to read online, using a remember template

'("Web reading" ?w "* TOREAD %^L\n%U %k\n%? " "~/Personal/Web.org" "Web sites to read")

which I use to add items to my reading list. I also have the following in my
.emacs:

,----[ link description function ]
| (defun ahkt-link-description (link desc)
|   "Link description generator for orgmode"
|   (cond ((string-match "https?:" link)
| 	   (with-temp-buffer
| 	    (w3m-retrieve link)
| 	    (w3m-region (point-min) (point-max))
| 	    (if (string= "text/html" (w3m-content-type link))
| 	        (replace-regexp-in-string "[ \t]+" " "
| 		 			  (replace-regexp-in-string
| 					   "\\(\n\\|
\\)" ""
| 					   (w3m-current-title))))))
| 	  ((string-match "file:\\([^:]+\\)::\\(.+\\)" link)
| 	   (let ((search (match-string-no-properties 2 link))
| 	       (filename
| 		(car (last
| 		      (split-string (match-string-no-properties 1 link) "/")))))
| 	   (format "%s: %s" filename search)))
| 	((string-match "file:\\([^:]+\\)" link)
| 	 (car (last (split-string (match-string-no-properties 1 link) "/"))))
| 	(t (or desc link))))
| 
| (setq org-make-link-description-function 'ahkt-link-description)
`----

which fills in the descriptions for links with the title from the retrieved web
page. 

>   - Perhaps antenna can also be integrated with org-mode.
>
> + org merge org-annotate-file with remember code
>   to allow annotating anything
>
>   also have a hook for opening files and w3m pages etc. that will print in the
>   minibuffer "this file/page/directory is annotated. press ... to see the
>   annotation".

Have you looked at org-registry.el in contrib? The `org-registry-show' function
will list any org files which have links to the current buffer. 

> + brainstorm: support asking for the template after the note was entered.
> 
>   this might complicate things too much.
> 
>   this is a tricky one to design, but the philosophy is that the time between
>   having an idea and entering it should be minimal. choosing the template type
>   is a cognitive burden before you enter the idea.
> 
>   so dedicate a remember shortcut to the concept of "let me enter this now".
>   c-c c-c, it asks you details like is this a todo item? and which file does
>   it go to?
> 
>   you would have a remember template that allows for other remember templates
>   to be chosen after you enter the note.
> 
>   ideally, it works like this:
> 
>   1.  call org-remember like that
>   2.  enter note
>   3.  c-c c-c
>   4.  choose whether it's a note, journal, or shopping item
>   5.  if it's note or journal, choose whether it's todo
>   6.  if it's note or journal, choose tags (RET for none)
>   7.  show completed buffer
>   8.  y to accept; n to edit

Personally I don't find selecting a template before making an entry a problem,
but I can see what ypu're getting at. I think I can implement something which
will do what you want. Suppose you have a set of templates as below:

("Note" ?n "* %^s %i%? %^g" "Notes.org" top)
("Journal" ?j "* %^s %u %^g\n%i%u%?" "Journal.org" bottom)
("Shopping" ?s "* %i%?" "Shopping.org" bottom)

where %^s is a new expansion which allows entering a todo keyword.

Currently if you call org-remember in the remember buffer, it prompts you for a
template again and applies it, which wipes out everything you have entered. If
this was changed so that the contents of the remember buffer are stored, and
inserted into appropriate places in the new template, it would be possible to
type your idea into the remember buffer, and then apply a template to it,
provided the template contains suitable expansions.  

So the process would be (with org-remember bound to C-c m):

1. C-c m RET to run org-remember and select no template.
2. Enter your idea.
3. C-c m to run org-remember again.
4. Select the appropriate template.
5. Fill in any prompts for todo keywords and tags, or leave blank as appropriate.
6. C-c C-c to file the note.

Would that be suitable?

James

-- 
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|

Re: RFC: Improvements to org-remember

[Samuel Wales]

Hi James,

Parts snipped.

James TD Smith
<ahktenzero@mohorovi.cc> wrote:
> On 2008-11-24 14:41:00(-0700), Samuel Wales wrote:
>> + respect pop-up-windows
>
> Could you explain this in more detail?

That variable controls whether the window gets split.  Some
people prefer buffer-creating commands to act like c-x 4 b
while others prefer that the whole window be used.  Those
who use somewhat large fonts need the latter because
splitting the window makes small windows.  It does not seem
to be possible to change org-remember's behavior with
same-window-buffer-names or -regexps.  That's about all I
know about it.  A dedicated variable would be OK.

>> + if org-remember does not recognize the type, abort completely
>
> Personally I'd prefer it to keep asking for a valid
> template if you enter a character that doesn't correspond

I like your proposal even better.  Ideally ^G and perhaps
esc esc esc will get you back to where you were before.

>> + org-remember-templates takes a character
>>   can it take a function key?
>
> It would be possible to add this. Do you specifically want
> to use the F-keys, or have you just run out of keys for
> templates? I'm planning on adding two-stage selection for
> remember templates, which will make it possible to have
> many more templates.

If possible, it would be nice if it were like emacs key
bindings.  Function keys can be surprisingly useful.

>> + emacs-w3m tight integration with org-mode
>>   - might be interesting to use an org-mode file to store
>>   bookmarks. this
>>     would require changing the way bookmarks are added,
>>     to store them in a way similar to org-remember.
>
> I like the idea of using org-mode to store bookmarks. I
> use org-mode to keep track of things I want to read
> online, using a remember template
>
> '("Web reading" ?w "* TOREAD %^L\n%U %k\n%? "
> "~/Personal/Web.org" "Web sites to read")
>
> which I use to add items to my reading list. I also have
> the following in my .emacs:

> which fills in the descriptions for links with the title
> from the retrieved web page.

IMO it would be good to have this stuff integrated as org
code, and, if the user sets, to replace the relevant keys
(such as v) in w3m.  Sounds like all the code is there.

There are useful possibilities here.  Remember could then
use ido to choose where in the bookmark tree, ideally
creating a parent headline if it does not exist.  Or just go
to a default location to be refiled later.  Or offer places
where you have bookmarked stuff before.

The best part is that you can have all of your bookmarks in
your org outline hierarchy where they belong.  If they are
tagged as bookmarks, then they show up in an agenda command
as a nice sparse tree of bookmarks.  Then you can call w3m,
or whatever else, from there.  No need to keep a separate
bookmark file.

This would be easier to use than the dedicated w3m bookmark
mechanism.

And it would work for files and directories also.  You can
bookmark some code that you are working on, if you want.

Remember seems like the right place to do most of this.

>>   - Perhaps antenna can also be integrated with org-mode.

Extension #1 to the bookmark idea.

Antenna is a way of keeping up with site changes.  In w3m,
it is a separate command from bookmarks, but there is no
reason that both antenna and bookmarks could not use the
same mechanism.  In org, this would be powerful.  For
example, a bookmark could have properties, and among those
properties would be the site's last-modified timestamp or
your last-checked timestamp.  Then without visiting the
pages you can run a command to see which pages have changed.

Where remember comes in is that remember helps you store
those bookmarks.

You might want to limit the scope of your code, but I
thought I'd propose it anyway in case there is interest.

The bookmark idea is much more important than antenna.

>>
>> + org merge org-annotate-file with remember code
>>   to allow annotating anything

> Have you looked at org-registry.el in contrib? The
> `org-registry-show' function will list any org files which
> have links to the current buffer.

Extension #2 to the bookmark idea.

My idea is to always have annotations available for
emacs-w3m, dired, files, like org-annotate-file, just with
more modes.

You can see in the mode line that whatever buffer you are in
has an annotation, and you can make an annotation.  You can
also go to the annotation.

The annotations are stored in an org file anywhere in the
hierarchy.  Thus, if you want, annotations on a doctor's web
site can be stored in the entry for that doctor that is in
your org file.  If you visit that web site from any source,
even Google, the mode line says that it is annotated.  Then
you can pull up that entry with a command.

Likewise with files or dired or whatever.  For example, you
can comment org.el or /etc/passwd without having to modify
them.

Remember code seems a plausible place to arrange for
choosing a location and putting a note into it.  Annotations
are like bookmarks with text that also go the other
direction.  It's natural to combine the idea of a bookmark
and the idea of an annotation.

You might want the mode line to say "there is bookmark to
this (web page, file, etc.)" as one character and "there is
a text note about this" as another character.  Thus, if you
have annotated a file and the file is unmodified, you will
see "-u:--!!" and if you have merely bookmarked the location
without commenting on it, then you will see "-u:--!-".

The bookmark idea is more important than annotations.

>
>> + brainstorm: support asking for the template after the
>> note was entered.

> 1. C-c m RET to run org-remember and select no template.
> 2. Enter your idea.
> 3. C-c m to run org-remember again.
> 4. Select the appropriate template.
> 5. Fill in any prompts for todo keywords and tags, or leave blank as appropriate.
> 6. C-c C-c to file the note.
>
> Would that be suitable?

Looks good.  Thanks for considering it.

-- 
Myalgic encephalomyelitis denialists are knowingly causing further
suffering and death by opposing biomedical research on this serious
infectious disease.  Do you care about the world?
http://www.meactionuk.org.uk/What_Is_ME_What_Is_CFS.htm

Re: RFC: Improvements to org-remember

[Sebastian Rose]

James TD Smith <ahktenzero@mohorovi.cc> writes:
> ** Unit tests
>
>    If we can decide on a unit testing framework, this would be an ideal
>    opportunity to add some test coverage, as I'll need to do extensive testing
>    to ensure I don't break anything with the refactoring or the new features. I
>    have some experience with unit testing using JUnit.


As your the first one to really use unit tests, I suggest you use the
framework you like the most or you trust the most, what ever. In the
worst case, those tests have to be rewritten but this could happen
anyway.


As I read through the 'How you can help' thread [1], I found no basis
for a decision. The only example code on Worg uses ert [2]. Could be a
hint, since one of the active elisp people here used it at least once to
write the example code (Eric?).


Looks like simply choosing one and see how far it goes might be the way
to go here.





Regards,

  Sebastian

---- Footnotes: ---------------

 [1]  http://lists.gnu.org/archive/html/emacs-orgmode/2008-10/msg00451.html
      and previous/next messages.

 [2]  Worg.git/org-tests/ert-publish-test.el


-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover

Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Http:  www.emma-stil.de

Re: RFC: Improvements to org-remember

[Sebastian Rose]

James TD Smith <ahktenzero@mohorovi.cc> writes:
> * New features
> ** Adding non-headline items
>    - Table rows.
>    - Checklist entries
>    - Plain list entries.
> ** Per-template insertion order
> ** Automatic sorting


Yes :-)

+1


Best,

-- 
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.:  +49 (0)511 - 36 58 472
Fax:   +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Http:  www.emma-stil.de

Re: RFC: Improvements to org-remember

[Carsten Dominik]

Hi James, I do like all this.  A few comments:

On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:

> I have a number of improvements to org-remember I am planning to  
> implement. I
> have briefly discussed some of them with Carsten, and I thought I  
> should post a
> detailed description here for discussion as I'm sure you will have  
> ideas,
> suggestions and criticisms of the proposed changes.
>
> * Preliminary work
> ** Refactoring
>
> Carsten suggested that the remember handler could do with some  
> refactoring
> before adding any new features, as it has gotten quite complicated  
> in places.
>
> The longest functions are `org-remember-apply-template' and
> `org-remember-handler', so these are probably the best place to start.
>
> I think it would make sense to move the code to get values for  
> remember
> expansions out of `org-remember-apply-template' into separate  
> functions. These
> could be added to an association list indexed by the expansion  
> character. This
> would also make it easier to add new expansions.

Yes.  However, it is necessary o keep the sequence of handling the  
escapes,
in particular first filling in all non-interactive ones, and only then
doing the interactive ones.

>
>
> ** Unit tests
>
>  If we can decide on a unit testing framework, this would be an ideal
>  opportunity to add some test coverage, as I'll need to do extensive  
> testing
>  to ensure I don't break anything with the refactoring or the new  
> features. I
>  have some experience with unit testing using JUnit.
>
> ** Plists for remember templates

Ah, this will be a big relieve when it is implemented,
should have been like this from the start.

>
>
>  I want to change the format of remember templates to use plists.  
> This is to
>  allow introducing a number of optional parameters to control the  
> new features
>  I want to add. Org uses plists elsewhere, for example in the # 
> +OPTIONS:
>  configuration header, and #+PLOT: lines, so the syntax should  
> already be
>  familiar to org users.
>
>  I also think it would make sense to to move some options which are  
> currently
>  set using escapes outside of the template, specifically "%!" (store
>  template immediately) and "%&" (jump to entry after storing).

Yes, this wil be much better.

>  For backwards compatibility, the current template format would  
> still be
>  supported, but the additional options would not be available.  
> Defaults for the
>  extra parameters would be set so if they are not present the  
> templates would
>  work as they do currently.
>
>  The current options would be represented as below:
>  - :template :: the template itself
>  - :name :: the name of the template
>  - :target :: The :target parameter takes the place of the current  
> file and
> 		headline target specification. The parameter specifies only the
> 		default target; all the other options will remain available via
> 		numeric prefixes to C-c C-c. The available options are:
>    - "<filename>:<heading>" :: a file target. If the heading is  
> omitted,
> 	  			 a top-level heading will be created.


That should be a double colon, for symmetry with org-archive-location,
and to avoid problems with Windows paths.

>
>    - clock :: currently clocked task
>    - current :: add under the the org headline where the point is.
>    - interactive :: select a location interactively using the  
> appropriate
> 		      interface
>    - <function> :: call function to get the target. The function can  
> return
> 		     either a marker, or a file/headline string.
>  - :test :: a function, or list of major modes. The template is only
> 	      available if the function returns true, or the current
> 	      buffer is in one of the appropriate modes
>  - :immediate :: replaces the %! escape; if t, the template is  
> stored as soon
> 		   as all escapes are replaced.
>  - :jumpto :: replaces the %& escape; if t, org jumps to the target  
> location
> 		when the note is stored.
>
> * New features
> ** Adding non-headline items
>
>  For some time I have wanted to be able to use remember to add  
> checklist
>  entries and table rows as well as org headlines. To configure this,  
> a :type
>  parameter will be added to the template, which can be either  
> headline (the
>  default), list, checklist or table.
>
>  - Table rows.

This is an awesome idea, as are plain list items and checkboxes.

For plain list items, I guess the right thing would be to select the
first plain list under the headline, there might be several.
Also, the first table under a headline, in case there are more.



While one could have a property for explicitly selecting a type
like table row or plain-list item or checkbox,  it would also be
possible to derive this from the Remember buffer content automatically.
Which method is better?

>
>
>    Currently if you want to use org to record periodic measurements  
> (for
>    example see the thread about using org to manage fitness  
> training), you
>    have to use properties and column view, which has a number of  
> limitations
>    (speed, calculations). Being able to add table rows via remember  
> would make
>    it much easier to do this.
>
>    The simplest implementation would use a template containing the  
> appropriate
>    number of table columns, for example something like
>
>    "| %U | %^{Value 1} | %^{Value 2} |"
>
>    This would be added to the table at the appropriate position  
> (depending on
>    the :prepend value for the template, and then formatted properly  
> using
>    `org-table-align'.
>
>    The handler would also need to ensure that table formulae get  
> updated
>    (increment row ranges etc) and that values are recalculated (if  
> automatic
>    recalculation is enabled) after the line is added.
>
>    An extension to this would be to include a truncated copy of the  
> table in
>    the remember buffer, with just the headers (and possibly  
> formulae) from the
>    target table, so the user could add multiple lines in the  
> remember buffer
>    and then add them to the table.

I guess you mean here to retrive that truncated copy from the target  
location?

>
>
>  - Checklist entries
>
>    I use checklists a lot for breaking down tasks. Being able to add  
> checklist
>    entries using remember, with the clocked task as a target, would  
> make it
>    possible to do this without having to switch away from the buffer  
> I was
>    working in.
>
>  - Plain list entries.
>
>    It also be useful to be able to add plain list items. This would  
> be like a
>    more featureful version of `org-add-note', with the full range of  
> remember
>    expansions available.


I would like to add here

   - Non-org items, to simply be appended to a file.
     I believe Russel Adams had a request about this, on how to use
     remember to add entries to a ledger file, for which using a
     template would be nice.  The problem is that Org currently requires
     the target file to be in Org-mode - for good reasons, because all
     kinds of Org stuff will be executed when the new entry is added.
     We should allow for the results of a remember template to be added
     to a non-Org file.

   - With these new options, will we need to have optional two-key
     access to templates, or is single key enough?  Maybe only at a  
later
     stage if it is felt that this is really necessary.


> ** Updating completion statistics
>
>  If the target for a remember template contains either of the  
> completion
>  status tracking cookies ([/}, [%]), these should be updated after  
> the entry
>  has been added. This should apply to both entries containing  
> checklists, and
>  entries containing other headlines.
>
>  Also, target matching for templates should ignore the completion  
> status
>  cookies.
>
> ** Per-template insertion order
>
>  It is currently possible to configure a remember template to add  
> entries at
>  the start or end of the target only if the target is a file, or set  
> globally
>  with `org-reverse-note-order'. It would be useful to be able to set  
> this for
>  individual templates.
>
>  An optional :prepend parameter would control where the entry would be
>  inserted. If set to t the new entry would be the first under the  
> heading, and
>  the last the if nil. The default would be whatever `org-reverse- 
> note-order'
>  is set to.
>
> ** Automatic sorting
>
>  It would be useful to be able to automatically sort the target's  
> entries
>  after adding one using remember, for example if you want to keep  
> entries
>  sorted by TODO keyword or in alphabetical order.
>
>  This would require two parameters:
>  - :sort :: if non-nil, sort the contents of the headline the item was
> 	       added under with the specified function. valid options are
>    - todoorder :: sort by TODO state
>    - alpha :: sort alphabetically
>    - numeric :: sort numerically
>    - priority :: sort by priority
>    - time :: sort by time
>    - property :: sort by property
>    - <function> :: use function to determine sort order
>    - Additionally, any of the previous items postfixed with '-rev'  
> will sort
>      in reverse
>  - :sortkey :: for tables, and property searches, this specifies  
> either the
> 		 column or property to sort by.


Another option I would like to see is, how many empty lines should
be inserted before the entry.  Because sometimes it is nice to have
an empty line between entries, and sometimes not.  Default should be
no empty lines.

> * Development plan
>
> It will take me quite a while to implement all this so I plan on  
> doing it in
> stages. After the preliminary work is done, each new feature will be
> implemented in a separate git branch which can be merged into the  
> master
> branch when it's finished and properly tested. I'll post an update  
> to the list
> when each feature is ready for testing.


Yes, this sounds good.  The one thing we need to keep in mind is
that a significant number of people by now really depend on the
templates working, so we cannot break org-remember at any stage for
the normal user.  So my proposal would be to create a new branch in our
main repository, a branch James will be the only one to push to, using
patches and ideas from the rest of us.  If this branch is frequently
rebased to the main Org-mode master, testers could easily and without
loss of other new features switch between the Org master and the
remember branch.

Does that sound good, or would you prefer to use your own repo
for this, James?  I believe the number of testers will be larger
if you use our main repo.  If you give me your user name on
repo.or.cz, I will give you push access.

- Carsten


>
>
> James
>
> --
> |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: RFC: Improvements to org-remember

[Russell Adams]

On Mon, Nov 24, 2008 at 09:58:49AM +0100, Carsten Dominik wrote:
>
> Hi James, I do like all this.  A few comments:
>
> On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:
> ...
>
>> ** Adding non-headline items
>>
>>  For some time I have wanted to be able to use remember to add  
>> checklist
>>  entries and table rows as well as org headlines. To configure this, a 
>> :type
>>  parameter will be added to the template, which can be either headline 
>> (the
>>  default), list, checklist or table.
>>
>>  - Table rows.
>
> This is an awesome idea, as are plain list items and checkboxes.
>
> For plain list items, I guess the right thing would be to select the
> first plain list under the headline, there might be several.
> Also, the first table under a headline, in case there are more.
>

Table rows would rock!

> I would like to add here
>
>   - Non-org items, to simply be appended to a file.
>     I believe Russel Adams had a request about this, on how to use
>     remember to add entries to a ledger file, for which using a
>     template would be nice.  The problem is that Org currently requires
>     the target file to be in Org-mode - for good reasons, because all
>     kinds of Org stuff will be executed when the new entry is added.
>     We should allow for the results of a remember template to be added
>     to a non-Org file.

This style of adding to non-org files essentially makes remember a
template generated data entry front end for arbitrary text.

This would make the ledger file easy. My idea was to use remember to
quickly enter time/expenses/mileage into my ledger.

Thanks for your efforts!


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: RFC: Improvements to org-remember

[James TD Smith]

Hi Carsten,

On 2008-11-24 09:58:49(+0100), Carsten Dominik wrote:
> 
> Hi James, I do like all this.  A few comments:
> 
> On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:
> > I think it would make sense to move the code to get values for remember
> > expansions out of `org-remember-apply-template' into separate functions.
> > These could be added to an association list indexed by the expansion
> > character. This would also make it easier to add new expansions.
> 
> Yes. However, it is necessary to keep the sequence of handling the escapes, in
> particular first filling in all non-interactive ones, and only then doing the
> interactive ones.

I'll probably use two lists, one of interactive escapes and one of
non-interactive escapes.

> > ** Plists for remember templates
> 
> Ah, this will be a big relieve when it is implemented,
> should have been like this from the start.
> 
> > I want to change the format of remember templates to use plists. This is to
> > allow introducing a number of optional parameters to control the new
> > features I want to add. Org uses plists elsewhere, for example in the #
> > +OPTIONS: configuration header, and #+PLOT: lines, so the syntax should
> > already be familiar to org users.
> >
> > I also think it would make sense to to move some options which are currently
> > set using escapes outside of the template, specifically "%!" (store template
> > immediately) and "%&" (jump to entry after storing).
> 
> Yes, this wil be much better.

I was thinking that maybe some other expansions should be moved into the template,
specifically those which don't insert their values where the % expansion is.

For example instead of 
,----
| ("Video" ?v "* TOWATCH %^{Title} %^g%^{Type}p%^{Length}p%^{Year}"
| "~/Personal/Video.org" top)
`----
we could have
,----
| (?v :name "Video" "* TOWATCH %^{Title}" :tags file :properties ("Type" "Length" "Year")
| :target "~/Personal/Video.org" :prepend t)                                                
`----

I think the latter is much better for adding properties, particularly if you
want to have a template with a lot of them.

> > For backwards compatibility, the current template format would still be
> > supported, but the additional options would not be available. Defaults for
> > the extra parameters would be set so if they are not present the templates
> > would work as they do currently.
> >
> >  The current options would be represented as below:
> >  - :template :: the template itself
> >  - :name :: the name of the template
> >  - :target :: The :target parameter takes the place of the current
> >		  file and headline target specification. The parameter
> > 		  specifies only the default target; all the other options will
> > 		  remain available via numeric prefixes to C-c C-c. The
> > 		  available options are:
> >    - "<filename>:<heading>" :: a file target. If the heading is  
> >				   omitted, a top-level heading will be created.
> 
> 
> That should be a double colon, for symmetry with org-archive-location,
> and to avoid problems with Windows paths.

Yes, it was supposed to be a double colon.
 
> > * New features
> > ** Adding non-headline items
> >
> >  For some time I have wanted to be able to use remember to add checklist
> >  entries and table rows as well as org headlines. To configure this, a :type
> >  parameter will be added to the template, which can be either headline (the
> >  default), list, checklist or table.
> >
> >  - Table rows.
> 
> This is an awesome idea, as are plain list items and checkboxes.
> 
> For plain list items, I guess the right thing would be to select the first
> plain list under the headline, there might be several. Also, the first table
> under a headline, in case there are more.
 
The plain list items would be added as direct children of the target headline.
I'm not sure having a plain list item under a headline as a target makes sense,
but it could be implemented by changing the :target specification to allow
specifying a path, so "test.org::Target::list item" would add the new entry as a
child of the first list item. This would also remove the requirement for
remember targets to have headlines which are unique in the file.

For tables it would have to be the first table under the headline, as I don't
think there is a way of identifying a particular table in an org entry. Maybe we
could use #+CAPTION, when it's been added?

> While one could have a property for explicitly selecting a type like table row
> or plain-list item or checkbox, it would also be possible to derive this from
> the Remember buffer content automatically. Which method is better?

I think using the property would be easier to implement, but automatically
figuring out what kind of entry to insert will be needed to handle entries
without templates.

> > An extension to this would be to include a truncated copy of the table in
> > the remember buffer, with just the headers (and possibly formulae) from the
> > target table, so the user could add multiple lines in the remember buffer
> > and then add them to the table.
> 
> I guess you mean here to retrive that truncated copy from the target  
> location?

Yes. 
 
> >  - Checklist entries
> >
> >    I use checklists a lot for breaking down tasks. Being able to add
> >    checklist entries using remember, with the clocked task as a target,
> >    would make it possible to do this without having to switch away from the
> >    buffer I was working in.
> >
> >  - Plain list entries.
> >
> >    It also be useful to be able to add plain list items. This would be like
> >    a more featureful version of `org-add-note', with the full range of
> >    remember expansions available.
> 
> I would like to add here
> 
>    - Non-org items, to simply be appended to a file.
>      I believe Russel Adams had a request about this, on how to use
>      remember to add entries to a ledger file, for which using a
>      template would be nice.  The problem is that Org currently requires
>      the target file to be in Org-mode - for good reasons, because all
>      kinds of Org stuff will be executed when the new entry is added.
>      We should allow for the results of a remember template to be added
>      to a non-Org file.
>
>    - With these new options, will we need to have optional two-key access to
>      templates, or is single key enough? Maybe only at a later stage if it is
>      felt that this is really necessary.

I'd like two-key access for templates anyway; I have a number of similar
templates which are scattered over the available keys and could be grouped
together more logically with two stage selection. 

While we're on the subject, I'm not sure the way org-select-remember-template
currently handles keys which don't correspond to a template is right; I think it
would make more sense to keep asking the user for a template until they select a
valid one, or choose a 'no template' option.

> Another option I would like to see is, how many empty lines should
> be inserted before the entry.  Because sometimes it is nice to have
> an empty line between entries, and sometimes not.  Default should be
> no empty lines.

That should be easy to add. What about entries added before the current contents
of the target headline? The blank lines would need to go after the newly
inserted item to maintain the proper gap between it and the headline below it.

> > * Development plan
> >
> > It will take me quite a while to implement all this so I plan on  
> > doing it in
> > stages. After the preliminary work is done, each new feature will be
> > implemented in a separate git branch which can be merged into the  
> > master
> > branch when it's finished and properly tested. I'll post an update  
> > to the list
> > when each feature is ready for testing.
> 
> Yes, this sounds good.  The one thing we need to keep in mind is
> that a significant number of people by now really depend on the
> templates working, so we cannot break org-remember at any stage for
> the normal user.  So my proposal would be to create a new branch in our
> main repository, a branch James will be the only one to push to, using
> patches and ideas from the rest of us.  If this branch is frequently
> rebased to the main Org-mode master, testers could easily and without
> loss of other new features switch between the Org master and the
> remember branch.
>
> Does that sound good, or would you prefer to use your own repo
> for this, James?  I believe the number of testers will be larger
> if you use our main repo.  If you give me your user name on
> repo.or.cz, I will give you push access.

I think using a branch in the main repo makes sense as I can push to it when I
have things which are ready for testing, and I keep using my own repo to sync
work between my computers without worrying about breaking things for anyone
testing the branch.

I don't currently have an account on repo.or.cz, but I'll sign up and send you
my details. I probably ought to sign up for Worg as well.

James

-- 
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|

Re: RFC: Improvements to org-remember

[Carsten Dominik]

Hi James,

On Nov 25, 2008, at 12:46 PM, James TD Smith wrote:

> Hi Carsten,
>
> On 2008-11-24 09:58:49(+0100), Carsten Dominik wrote:
>>
>> Hi James, I do like all this.  A few comments:
>>
>> On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:
>>> I think it would make sense to move the code to get values for  
>>> remember
>>> expansions out of `org-remember-apply-template' into separate  
>>> functions.
>>> These could be added to an association list indexed by the expansion
>>> character. This would also make it easier to add new expansions.
>>
>> Yes. However, it is necessary to keep the sequence of handling the  
>> escapes, in
>> particular first filling in all non-interactive ones, and only then  
>> doing the
>> interactive ones.
>
> I'll probably use two lists, one of interactive escapes and one of
> non-interactive escapes.

I believe it makes some sense to fill in the interactive parts
in place, while the partially filled template is visible.  The
context may help.

>
>
>>> ** Plists for remember templates
>>
>> Ah, this will be a big relieve when it is implemented,
>> should have been like this from the start.
>>
>>> I want to change the format of remember templates to use plists.  
>>> This is to
>>> allow introducing a number of optional parameters to control the new
>>> features I want to add. Org uses plists elsewhere, for example in  
>>> the #
>>> +OPTIONS: configuration header, and #+PLOT: lines, so the syntax  
>>> should
>>> already be familiar to org users.
>>>
>>> I also think it would make sense to to move some options which are  
>>> currently
>>> set using escapes outside of the template, specifically  
>>> "%!" (store template
>>> immediately) and "%&" (jump to entry after storing).
>>
>> Yes, this wil be much better.
>
> I was thinking that maybe some other expansions should be moved into  
> the template,
> specifically those which don't insert their values where the %  
> expansion is.
>
> For example instead of
> ,----
> | ("Video" ?v "* TOWATCH %^{Title} %^g%^{Type}p%^{Length}p%^{Year}"
> | "~/Personal/Video.org" top)
> `----
> we could have
> ,----
> | (?v :name "Video" "* TOWATCH %^{Title}" :tags file :properties  
> ("Type" "Length" "Year")
> | :target "~/Personal/Video.org" :prepend t)
> `----
>
> I think the latter is much better for adding properties,  
> particularly if you
> want to have a template with a lot of them.


This is an interesting idea.  The TODO state could also be done in  
this way,
maybe offering the fast selection interface for TODO states.

>
>
>>> For backwards compatibility, the current template format would  
>>> still be
>>> supported, but the additional options would not be available.  
>>> Defaults for
>>> the extra parameters would be set so if they are not present the  
>>> templates
>>> would work as they do currently.
>>>
>>> The current options would be represented as below:
>>> - :template :: the template itself
>>> - :name :: the name of the template
>>> - :target :: The :target parameter takes the place of the current
>>> 		  file and headline target specification. The parameter
>>> 		  specifies only the default target; all the other options will
>>> 		  remain available via numeric prefixes to C-c C-c. The
>>> 		  available options are:
>>>  - "<filename>:<heading>" :: a file target. If the heading is
>>> 				   omitted, a top-level heading will be created.
>>
>>
>> That should be a double colon, for symmetry with org-archive- 
>> location,
>> and to avoid problems with Windows paths.
>
> Yes, it was supposed to be a double colon.
>
>>> * New features
>>> ** Adding non-headline items
>>>
>>> For some time I have wanted to be able to use remember to add  
>>> checklist
>>> entries and table rows as well as org headlines. To configure  
>>> this, a :type
>>> parameter will be added to the template, which can be either  
>>> headline (the
>>> default), list, checklist or table.
>>>
>>> - Table rows.
>>
>> This is an awesome idea, as are plain list items and checkboxes.
>>
>> For plain list items, I guess the right thing would be to select  
>> the first
>> plain list under the headline, there might be several. Also, the  
>> first table
>> under a headline, in case there are more.
>
> The plain list items would be added as direct children of the target  
> headline.
> I'm not sure having a plain list item under a headline as a target  
> makes sense,

I agree, let's not go there.

>
> but it could be implemented by changing the :target specification to  
> allow
> specifying a path, so "test.org::Target::list item" would add the  
> new entry as a
> child of the first list item. This would also remove the requirement  
> for
> remember targets to have headlines which are unique in the file.
>
> For tables it would have to be the first table under the headline,  
> as I don't
> think there is a way of identifying a particular table in an org  
> entry. Maybe we
> could use #+CAPTION, when it's been added?

Let's just go for the first table.  I think when one
makes a table the target of a remember process, it deserves
to be the only on under a heading.

>> While one could have a property for explicitly selecting a type  
>> like table row
>> or plain-list item or checkbox, it would also be possible to derive  
>> this from
>> the Remember buffer content automatically. Which method is better?
>
> I think using the property would be easier to implement, but  
> automatically
> figuring out what kind of entry to insert will be needed to handle  
> entries
> without templates.

Will we have entries without templates?

>
>
>>> An extension to this would be to include a truncated copy of the  
>>> table in
>>> the remember buffer, with just the headers (and possibly formulae)  
>>> from the
>>> target table, so the user could add multiple lines in the remember  
>>> buffer
>>> and then add them to the table.
>>
>> I guess you mean here to retrive that truncated copy from the target
>> location?
>
> Yes.
>
>>> - Checklist entries
>>>
>>>  I use checklists a lot for breaking down tasks. Being able to add
>>>  checklist entries using remember, with the clocked task as a  
>>> target,
>>>  would make it possible to do this without having to switch away  
>>> from the
>>>  buffer I was working in.
>>>
>>> - Plain list entries.
>>>
>>>  It also be useful to be able to add plain list items. This would  
>>> be like
>>>  a more featureful version of `org-add-note', with the full range of
>>>  remember expansions available.
>>
>> I would like to add here
>>
>>  - Non-org items, to simply be appended to a file.
>>    I believe Russel Adams had a request about this, on how to use
>>    remember to add entries to a ledger file, for which using a
>>    template would be nice.  The problem is that Org currently  
>> requires
>>    the target file to be in Org-mode - for good reasons, because all
>>    kinds of Org stuff will be executed when the new entry is added.
>>    We should allow for the results of a remember template to be added
>>    to a non-Org file.
>>
>>  - With these new options, will we need to have optional two-key  
>> access to
>>    templates, or is single key enough? Maybe only at a later stage  
>> if it is
>>    felt that this is really necessary.
>
> I'd like two-key access for templates anyway; I have a number of  
> similar
> templates which are scattered over the available keys and could be  
> grouped
> together more logically with two stage selection.


Hmm.  I am not sure if the two-key selection code from the agenda can  
be easily
refactored for this case, so maybe we need to duplicate this  
functionality, or re-write the selector for agenda custom commands.

>
>
> While we're on the subject, I'm not sure the way org-select-remember- 
> template
> currently handles keys which don't correspond to a template is  
> right; I think it
> would make more sense to keep asking the user for a template until  
> they select a
> valid one, or choose a 'no template' option.

I am OK with changing this, it was mere laziness on
my side that it does not work like this right now.

>> Another option I would like to see is, how many empty lines should
>> be inserted before the entry.  Because sometimes it is nice to have
>> an empty line between entries, and sometimes not.  Default should be
>> no empty lines.
>
> That should be easy to add. What about entries added before the  
> current contents
> of the target headline? The blank lines would need to go after the  
> newly
> inserted item to maintain the proper gap between it and the headline  
> below it.

I think it is sufficient to only specify the empty lines
before the heading.  An entry that is inserted as the first child
must then simply be inserted directly after the heading and
possibly timestamps/properties, so that any empty lines
*before* the already present sibling remain.  Please do not
change this - throughout Org, it is the empty space *before*
a headline that counts.

>>> * Development plan
>>>
>>> It will take me quite a while to implement all this so I plan on
>>> doing it in
>>> stages. After the preliminary work is done, each new feature will be
>>> implemented in a separate git branch which can be merged into the
>>> master
>>> branch when it's finished and properly tested. I'll post an update
>>> to the list
>>> when each feature is ready for testing.
>>
>> Yes, this sounds good.  The one thing we need to keep in mind is
>> that a significant number of people by now really depend on the
>> templates working, so we cannot break org-remember at any stage for
>> the normal user.  So my proposal would be to create a new branch in  
>> our
>> main repository, a branch James will be the only one to push to,  
>> using
>> patches and ideas from the rest of us.  If this branch is frequently
>> rebased to the main Org-mode master, testers could easily and without
>> loss of other new features switch between the Org master and the
>> remember branch.
>>
>> Does that sound good, or would you prefer to use your own repo
>> for this, James?  I believe the number of testers will be larger
>> if you use our main repo.  If you give me your user name on
>> repo.or.cz, I will give you push access.
>
> I think using a branch in the main repo makes sense as I can push to  
> it when I
> have things which are ready for testing, and I keep using my own  
> repo to sync
> work between my computers without worrying about breaking things for  
> anyone
> testing the branch.
>
> I don't currently have an account on repo.or.cz, but I'll sign up  
> and send you
> my details. I probably ought to sign up for Worg as well.

Good.  For Worg, you need to send mail to Bastien, not through
emacs-orgmode, but directly, he will this this faster than any mails
going through the list.

One additional proposal:  Would it be useful, during the
development phase, to use a different customization variable,
org-remember-templates-2 or so?  This would allow people to
have a "safe" version of their templates to which they can fall
back if necessary, and to do all kinds of testing in the new
variable.  When things are stabilized in the end, we just
rename the new variable to the old name and be done....


- Carsten

Re: RFC: Improvements to org-remember

[James TD Smith]

Hi Carsten, 

On 2008-11-25 20:27:31(+0100), Carsten Dominik wrote:
> On Nov 25, 2008, at 12:46 PM, James TD Smith wrote:
> > On 2008-11-24 09:58:49(+0100), Carsten Dominik wrote:
> >> On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:
> >>> I think it would make sense to move the code to get values for remember
> >>> expansions out of `org-remember-apply-template' into separate functions.
> >>> These could be added to an association list indexed by the expansion
> >>> character. This would also make it easier to add new expansions.
> >>
> >> Yes. However, it is necessary to keep the sequence of handling the escapes,
> >> in particular first filling in all non-interactive ones, and only then
> >> doing the interactive ones.
> >
> > I'll probably use two lists, one of interactive escapes and one of
> > non-interactive escapes.
> 
> I believe it makes some sense to fill in the interactive parts in place, while
> the partially filled template is visible. The context may help.

I agree. I'm not going to change that.

> >>> ** Plists for remember templates
> >>
> >> Ah, this will be a big relieve when it is implemented, should have been
> >> like this from the start.
> >>
> >>> I want to change the format of remember templates to use plists. This is
> >>> to allow introducing a number of optional parameters to control the new
> >>> features I want to add. Org uses plists elsewhere, for example in the
> >>> #+OPTIONS: configuration header, and #+PLOT: lines, so the syntax should
> >>> already be familiar to org users.
> >>>
> >>> I also think it would make sense to to move some options which are
> >>> currently set using escapes outside of the template, specifically "%!"
> >>> (store template immediately) and "%&" (jump to entry after storing).
> >>
> >> Yes, this wil be much better.
> >
> > I was thinking that maybe some other expansions should be moved into the
> > template, specifically those which don't insert their values where the %
> > expansion is.
> >
> > For example instead of
> > ,----
> > | ("Video" ?v "* TOWATCH %^{Title} %^g%^{Type}p%^{Length}p%^{Year}"
> > | "~/Personal/Video.org" top)
> > `----
> > we could have
> > ,----
> > | (?v :name "Video" "* TOWATCH %^{Title}" :tags file :properties  
> > |  ("Type" "Length" "Year") :target "~/Personal/Video.org" :prepend t)
> > `----
> >
> > I think the latter is much better for adding properties, particularly if you
> > want to have a template with a lot of them.
> 
> This is an interesting idea. The TODO state could also be done in this way,
> maybe offering the fast selection interface for TODO states.

Yes. An expansion for TODO states might be useful as well. 

> >> While one could have a property for explicitly selecting a type like table
> >> row or plain-list item or checkbox, it would also be possible to derive
> >> this from the Remember buffer content automatically. Which method is
> >> better?
> >
> > I think using the property would be easier to implement, but automatically
> > figuring out what kind of entry to insert will be needed to handle entries
> > without templates.
> 
> Will we have entries without templates?

Yes, for two reasons: freeform entry with the possibility of applying a template
later (see my reply to Samuel Wales' suggestions), and so remember can be used
to add non-org items (possibly with other remember handlers).

> > I'd like two-key access for templates anyway; I have a number of similar
> > templates which are scattered over the available keys and could be grouped
> > together more logically with two stage selection.
> 
> Hmm. I am not sure if the two-key selection code from the agenda can be easily
> refactored for this case, so maybe we need to duplicate this functionality, or
> re-write the selector for agenda custom commands.

Is `org-agenda-get-restriction-and-command' the method I should be looking at? 

> >> Another option I would like to see is, how many empty lines should be
> >> inserted before the entry. Because sometimes it is nice to have an empty
> >> line between entries, and sometimes not. Default should be no empty lines.
> >
> > That should be easy to add. What about entries added before the current
> > contents of the target headline? The blank lines would need to go after the
> > newly inserted item to maintain the proper gap between it and the headline
> > below it.
> 
> I think it is sufficient to only specify the empty lines before the heading.
> An entry that is inserted as the first child must then simply be inserted
> directly after the heading and possibly timestamps/properties, so that any
> empty lines *before* the already present sibling remain. Please do not change
> this - throughout Org, it is the empty space *before* a headline that counts.

OK.

> > I think using a branch in the main repo makes sense as I can push to it when
> > I have things which are ready for testing, and I keep using my own repo to
> > sync work between my computers without worrying about breaking things for
> > anyone testing the branch.
> >
> > I don't currently have an account on repo.or.cz, but I'll sign up and send
> > you my details. I probably ought to sign up for Worg as well.
> 
> Good. For Worg, you need to send mail to Bastien, not through emacs-orgmode,
> but directly, he will this this faster than any mails going through the list.
>
> One additional proposal: Would it be useful, during the development phase, to
> use a different customization variable, org-remember-templates-2 or so? This
> would allow people to have a "safe" version of their templates to which they
> can fall back if necessary, and to do all kinds of testing in the new
> variable. When things are stabilized in the end, we just rename the new
> variable to the old name and be done....

I'd prefer to have org-remember-templates-2 as an adjunct to the templates in
org-remember-templates, so in the testing version you would get all the
templates in that plus templates in org-remember-templates-2 (with templates in
org-remember-templates-2 having priority). Templates in the current format
should work in the new version.

James

-- 
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|

Re: RFC: Improvements to org-remember

[Carsten Dominik]

On Nov 30, 2008, at 3:32 AM, James TD Smith wrote:

> Hi Carsten,
>
> On 2008-11-25 20:27:31(+0100), Carsten Dominik wrote:
>> On Nov 25, 2008, at 12:46 PM, James TD Smith wrote:
>>> On 2008-11-24 09:58:49(+0100), Carsten Dominik wrote:
>>>> On Nov 24, 2008, at 12:25 AM, James TD Smith wrote:
>>>>> I think it would make sense to move the code to get values for  
>>>>> remember
>>>>> expansions out of `org-remember-apply-template' into separate  
>>>>> functions.
>>>>> These could be added to an association list indexed by the  
>>>>> expansion
>>>>> character. This would also make it easier to add new expansions.
>>>>
>>>> Yes. However, it is necessary to keep the sequence of handling  
>>>> the escapes,
>>>> in particular first filling in all non-interactive ones, and only  
>>>> then
>>>> doing the interactive ones.
>>>
>>> I'll probably use two lists, one of interactive escapes and one of
>>> non-interactive escapes.
>>
>> I believe it makes some sense to fill in the interactive parts in  
>> place, while
>> the partially filled template is visible. The context may help.
>
> I agree. I'm not going to change that.
>
>>>>> ** Plists for remember templates
>>>>
>>>> Ah, this will be a big relieve when it is implemented, should  
>>>> have been
>>>> like this from the start.
>>>>
>>>>> I want to change the format of remember templates to use plists.  
>>>>> This is
>>>>> to allow introducing a number of optional parameters to control  
>>>>> the new
>>>>> features I want to add. Org uses plists elsewhere, for example  
>>>>> in the
>>>>> #+OPTIONS: configuration header, and #+PLOT: lines, so the  
>>>>> syntax should
>>>>> already be familiar to org users.
>>>>>
>>>>> I also think it would make sense to to move some options which are
>>>>> currently set using escapes outside of the template,  
>>>>> specifically "%!"
>>>>> (store template immediately) and "%&" (jump to entry after  
>>>>> storing).
>>>>
>>>> Yes, this wil be much better.
>>>
>>> I was thinking that maybe some other expansions should be moved  
>>> into the
>>> template, specifically those which don't insert their values where  
>>> the %
>>> expansion is.
>>>
>>> For example instead of
>>> ,----
>>> | ("Video" ?v "* TOWATCH %^{Title} %^g%^{Type}p%^{Length}p%^{Year}"
>>> | "~/Personal/Video.org" top)
>>> `----
>>> we could have
>>> ,----
>>> | (?v :name "Video" "* TOWATCH %^{Title}" :tags file :properties
>>> |  ("Type" "Length" "Year") :target "~/Personal/ 
>>> Video.org" :prepend t)
>>> `----
>>>
>>> I think the latter is much better for adding properties,  
>>> particularly if you
>>> want to have a template with a lot of them.
>>
>> This is an interesting idea. The TODO state could also be done in  
>> this way,
>> maybe offering the fast selection interface for TODO states.
>
> Yes. An expansion for TODO states might be useful as well.
>
>>>> While one could have a property for explicitly selecting a type  
>>>> like table
>>>> row or plain-list item or checkbox, it would also be possible to  
>>>> derive
>>>> this from the Remember buffer content automatically. Which method  
>>>> is
>>>> better?
>>>
>>> I think using the property would be easier to implement, but  
>>> automatically
>>> figuring out what kind of entry to insert will be needed to handle  
>>> entries
>>> without templates.
>>
>> Will we have entries without templates?
>
> Yes, for two reasons: freeform entry with the possibility of  
> applying a template
> later (see my reply to Samuel Wales' suggestions), and so remember  
> can be used
> to add non-org items (possibly with other remember handlers).
>
>>> I'd like two-key access for templates anyway; I have a number of  
>>> similar
>>> templates which are scattered over the available keys and could be  
>>> grouped
>>> together more logically with two stage selection.
>>
>> Hmm. I am not sure if the two-key selection code from the agenda  
>> can be easily
>> refactored for this case, so maybe we need to duplicate this  
>> functionality, or
>> re-write the selector for agenda custom commands.
>
> Is `org-agenda-get-restriction-and-command' the method I should be  
> looking at?


Yes, this is where two-key commands would have to be implemented.

>
>
>>>> Another option I would like to see is, how many empty lines  
>>>> should be
>>>> inserted before the entry. Because sometimes it is nice to have  
>>>> an empty
>>>> line between entries, and sometimes not. Default should be no  
>>>> empty lines.
>>>
>>> That should be easy to add. What about entries added before the  
>>> current
>>> contents of the target headline? The blank lines would need to go  
>>> after the
>>> newly inserted item to maintain the proper gap between it and the  
>>> headline
>>> below it.
>>
>> I think it is sufficient to only specify the empty lines before the  
>> heading.
>> An entry that is inserted as the first child must then simply be  
>> inserted
>> directly after the heading and possibly timestamps/properties, so  
>> that any
>> empty lines *before* the already present sibling remain. Please do  
>> not change
>> this - throughout Org, it is the empty space *before* a headline  
>> that counts.
>
> OK.
>
>>> I think using a branch in the main repo makes sense as I can push  
>>> to it when
>>> I have things which are ready for testing, and I keep using my own  
>>> repo to
>>> sync work between my computers without worrying about breaking  
>>> things for
>>> anyone testing the branch.
>>>
>>> I don't currently have an account on repo.or.cz, but I'll sign up  
>>> and send
>>> you my details. I probably ought to sign up for Worg as well.
>>
>> Good. For Worg, you need to send mail to Bastien, not through emacs- 
>> orgmode,
>> but directly, he will this this faster than any mails going through  
>> the list.
>>
>> One additional proposal: Would it be useful, during the development  
>> phase, to
>> use a different customization variable, org-remember-templates-2 or  
>> so? This
>> would allow people to have a "safe" version of their templates to  
>> which they
>> can fall back if necessary, and to do all kinds of testing in the new
>> variable. When things are stabilized in the end, we just rename the  
>> new
>> variable to the old name and be done....
>
> I'd prefer to have org-remember-templates-2 as an adjunct to the  
> templates in
> org-remember-templates, so in the testing version you would get all  
> the
> templates in that plus templates in org-remember-templates-2 (with  
> templates in
> org-remember-templates-2 having priority). Templates in the current  
> format
> should work in the new version.


OK, sure, this is even better.

- Carsten

>
>
> James
>
> -- 
> |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: RFC: Improvements to org-remember

[Carsten Dominik]

Hi James,

two more feature requests for the new remember templates:


- An :id switch which triggers automatic creation of an ID property
   You can create one by calling org-id-get-create in the entry,
   on the headline or below it.

- A :link switch (or similarly named).  When remember is called from an
   Org-file with this switch, it should create a link to the remember  
entry
   and store that link like org-store-link does.

Thanks

- Carsten

Re: RFC: Improvements to org-remember

[Ben Alexander]

[-- Attachment #1.1: Type: text/plain, Size: 2116 bytes --]


On 2008-Nov-24, at 04:25, emacs-orgmode-request@gnu.org wrote:

> From: James TD Smith <ahktenzero@mohorovi.cc>
> Date: 24 November 2008 01:25:57 EET
> To: emacs-orgmode@gnu.org
> Subject: [Orgmode] RFC: Improvements to org-remember
>
> * New features
> ** Adding non-headline items
>
That's a fantastic idea!
>
> ** Updating completion statistics
>
> ** Automatic sorting
>
Right now, I have a :SORT: property in my property drawer which looks  
like:
   :SORT: C-c S-6 p
This is just a reminder to me for the key chord I need to play to get  
the sort I want.  It's conveniently located near the headline and not  
too hard to open and read when I need to resort manually.

It seems to me that having an hook like 'org-remember-after-filing  
would allow people to choose what kinds of updating they wanted done  
after a remember template was used.   Mixing this with different types  
of templates may take some care: you don't want to run all the hooks  
inside a save-excursion if the point to to allow the hook to move  
point to a special place, but then all hooks would have to be written  
with that in mind. Perhaps the hooks should be run inside a (let )  
with some official bindings for markers for the following:
  - org-remember-marker-to-beginning-of-new-text
  - org-remember-marker-to-end-of-new-text
  - org-remember-marker-to-parent-headline (perhaps most useful for  
non-headline remember templates)
  - org-remember-template-type

But automatic sorting seems useful in many other contexts (like after  
scheduling or rescheduling an item, or changing priority, or editing  
the headline text) so perhaps some wishes/ideas from the list would be  
appropriate.  Could org-mode take ownership of the :SORT: property for  
headlines, and have a org-sort-file-using-property (or a org-sort- 
headline-using-property) which could be added to hook lists where-ever  
the user wanted?

Or is this too specific?  Would it be nice to have plain lists (or  
checkboxed lists) have some kind of sort property too? Where could a  
user store this data so it could be easy to see but also easy to ignore.

[-- Attachment #1.2: Type: text/html, Size: 3717 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: RFC: Improvements to org-remember

[James TD Smith]

Hi Ben, 

On 2008-11-24 11:50:53(+0200), Ben Alexander wrote:
> On 2008-Nov-24, at 04:25, emacs-orgmode-request@gnu.org wrote:
> > ** Automatic sorting
> >
> Right now, I have a :SORT: property in my property drawer which looks  
> like:
>    :SORT: C-c S-6 p
> This is just a reminder to me for the key chord I need to play to get  
> the sort I want.  It's conveniently located near the headline and not  
> too hard to open and read when I need to resort manually.
> 
> It seems to me that having an hook like 'org-remember-after-filing  
> would allow people to choose what kinds of updating they wanted done  
> after a remember template was used.   Mixing this with different types  
> of templates may take some care: you don't want to run all the hooks  
> inside a save-excursion if the point to to allow the hook to move  
> point to a special place, but then all hooks would have to be written  
> with that in mind. Perhaps the hooks should be run inside a (let )  
> with some official bindings for markers for the following:
>   - org-remember-marker-to-beginning-of-new-text
>   - org-remember-marker-to-end-of-new-text
>   - org-remember-marker-to-parent-headline (perhaps most useful for  
> non-headline remember templates)
>   - org-remember-template-type

This is basically how I was thinking of implementing the sort after filing (and
the other post-commit update functions).

> But automatic sorting seems useful in many other contexts (like after  
> scheduling or rescheduling an item, or changing priority, or editing  
> the headline text) so perhaps some wishes/ideas from the list would be  
> appropriate.  Could org-mode take ownership of the :SORT: property for  
> headlines, and have a org-sort-file-using-property (or a org-sort- 
> headline-using-property) which could be added to hook lists where-ever  
> the user wanted?

I think it would be rather difficult to get automatic sorting working for
editing the headline text. Org doesn't have hook lists for priority or
scheduling changes either.

I do like the idea of defining a default sort for a tree. We would need
a :SORT_KEY: property as well, for sorting by property or table column, and a
#+SORT: facility for file level sorting. The property could be used to determine
a default sort for org-sort, with a new option added to select the default
(maybe C-c ^ RET).

> Or is this too specific?  Would it be nice to have plain lists (or  
> checkboxed lists) have some kind of sort property too? Where could a  
> user store this data so it could be easy to see but also easy to ignore.

I'm not sure it's necessary to be able to do this in plain lists. If a plain
list item has enough activity under it to need sorting on a regular basis its
probably worth promoting it to a headline.

--
|-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|