New implementation of the Org remember process ready for comments and testing

[Carsten Dominik <carsten.dominik@gmail.com>]

             New implementation of the Org remember process
             ==============================================

Dear all,

The fact that I have let go of trying to follow up every thread on the
mailing list is really great for me, thanks to John and David for
providing issue and patch tracking - this is a huge simplification.
And thanks to everyone who is contributing with answers and patches!

The best part of this - for me - is that it allows me to step back
and do some more systematic work again.

So I have finally found the time to rewrite the Org+Remember capturing
setup for org-mode.  This stuff is now available in the git repo and
it still needs testing.

Table of Contents
=================
1 Where is the new code?
2 Why a rewrite
3 Why a new name?
4 Acknowledgments
5 Setup
6 Overview over the changes
     6.1 No temporary buffer
     6.2 Different types of capture entries
     6.3 Many different and precise ways to specify the target of a  
template
     6.4 Multiple capture events going on at the same time
     6.5 No prefix args to C-c C-c when finishing up
     6.6 Per template settings
     6.7 Multi-key access
     6.8 The code is now much much cleaner and easier to maintain
7 Request for comments


1 Where is the new code?
~~~~~~~~~~~~~~~~~~~~~~~~~

   If you pull from the git repo, the new code is in
   lisp/org-capture.el.

2 Why a rewrite
~~~~~~~~~~~~~~~~

   Integration between Org and Remember started early on with very
   limited scope.  Since then we have changed and added a lot of stuff
   and accumulated a lot of opaque code.  The code has become nearly
   unmaintainable.  A number of feature requests have been on the table
   as well.

3 Why a new name?
~~~~~~~~~~~~~~~~~~

   I have - at least for now - chosen a new name for the new setup:
  ` org-capture'.  There are two reasons for this:

   1. I know that many people have a working setup for org-remember and
      use this a lot and for critical stuff.  I did not want to break
      any of this code.  While the new implementation works in a very
      similar way, it is still not quite compatible.

   2. The new implementation actually no longer uses remember.el.

   We can discuss if also the new implementation should be called
   `org-remember' eventually, just because everyone knows this process
   under this name.  I have not come to a conclusion about this and
   welcome opinions.

4 Acknowledgments
~~~~~~~~~~~~~~~~~~

   This new implementation draws heavily on ideas by James TD Smith.
   He wrote an [RFC for updates to org-remember] which was discussed on
   the mailing list.  Many of the features he describes have found
   there way into org-capture.  James has also started to implement
   these changes - at the moment I am not using any of his code, but I
   hope that we will be able to start using his template filling code -
   it is quite general and easily extensible, while in the code I am
   using now every %-escape is hard-coded.

   Also Samuel Wales has provided important input, distributed over a
   number of mails over time.  In particular, he had the idea to do the
   remembering directly in the target buffer.  Normally his would have
   caused problems because this would change properties of the
   window and visibility settings of the target buffer.  But using
   indirect buffers works around this issue.

   Of course, John Wiegley's remember.el is the prime source of
   inspiration for the entire capture process in Org.


   [RFC for updates to org-remember]: http://thread.gmane.org/gmane.emacs.orgmode/9579/focus%3D9579

5 Setup
~~~~~~~~

   To use the new setup, do the following:

   1. Run

      M-x org-capture-import-remember-templates RET

      to get a translated version of your remember templates into the
      new variable `org-capture-templates'.  This will "mostly" work,
      but maybe not for all cases.  At least it will give you a good
      place to modify your templates.  The customize interface is the
      best way to work with this complex variable.

   2. Bind the command `org-capture' to a key, similar to what you did
      with org-remember:

      (define-key global-map "\C-cc" 'org-capture)

      You may also use `C-c r' once you have decided to move over to
      the new implementation.

6 Overview over the changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

6.1 No temporary buffer
========================

    We no longer use a temporary buffer for creating the remember note.
    Instead, we make an indirect buffer linked to the target buffer and
    edit the new note already into its intended target file.  This
    means you will never again loose a note by accidentally starting a
    second remember process.  Also, all local settings like
    file-specific TODO keywords etc in the target buffer will be
    available when composing the note.

6.2 Different types of capture entries
=======================================

    The capture process can now add to the target file:

    - a new Org outline node, either at top level or as a child of an
      existing (this is what org-remember already does)
    - plain list items
    - checkboxes
    - a new line in a table (this is cool)
    - a chunk of plain text (and this is general)

    For example, it should be easy now to use `org-capture' in order to
    add stuff to a ledger file (Russell, are you happy now? :)

6.3 Many different and precise ways to specify the target of a template
========================================================================

    You can identify the target location for a note in a number of
    ways:

    - file
    - file + headline
    - file + outline-path
    - file + regular expression
    - file + date tree
    - clock (to file to the running clock)
    - ID
    - user-defined function

    You can also add a template right at point in the buffer where you
    currently are working, bypassing the target setting of the template
    entirely.  This is done by using a zero prefix:  `C-0 C-c c'.

6.4 Multiple capture events going on at the same time
======================================================

    If you need to capture something while already capturing something
    else, you can now do so.  Finishing the second process will
    automatically return you to the window setup of the first one.

    Bernt, this might lead to confusing results if you would try to
    port your immediate clock-in, clock-out code.  It will probably be
    better (if you decide at all to try the new code) to use the
    built-in :clock-in and :clock-resume properties:

    These will even work for a chain: start capture 1, then 2, then 3,
    finalize 3, then 2, then 1, and the previous clock will be
    restarted each time.  Only when you finish the captures in a
    different sequence, then the clock-resume stuff will lead to
    undefined results.

6.5 No prefix args to C-c C-c when finishing up
================================================

    Since the note is already in the target buffer, all exiting does is
    to kill the indirect buffer and to restore the window configuration
    before starting the process.  It does no longer make sense to try
    to change the target location when finishing the capture.
    Therefore, prefix arguments to the `C-c C-c' finishing process are
    no longer interpreted.

    Instead, you can simply press `C-c C-w' to refile a new Org entry.
    Apart from moving the entry, this will also kill the indirect
    buffer in the process and restore the previous window
    configuration.  Note that currently only `C-c C-w' will do this -
    the corresponding speed command or any other keys that lead to
    `org-refile' will not terminate the indirect buffer and restore the
    window configuration.

    To abort a capture you can now simply press `undo' to remove the
    template and then kill the (indirect) buffer.  I would like to make
    `C-c C-k' do these steps automatically, but that does not yet work
    because I have not figured out quite how to do this in a safe way
    (for example, you might have widened the indirect buffer and edited
    stuff in a different place.....).

6.6 Per template settings
==========================

    Many interesting options can be set on a per template way.  In
    fact, global settings like `org-reverse-note-order' is now ignored
    for the capture process.  Some of the stuff that can be set:

    :empty-lines :: Number of empty lines that should surround the
         entry at the target location.  This was a feature request,
         among others by Sebastien Vauban.

    :prepend :: Set this if the note should be added as the first
         rather than the last at the target location

    :clock-in :: Set this to make the clock If the clock should be
         started on the target item while working on the capture.

    :clock-resume :: Set this to resume the interrupted clock when
         finishing a capture process.

    :immediate-finish :: When this is set, the capture process is
         finished immediately after the templates has been filled.

    We can add more stuff here.  James has proposed sorting, this is a
    possibility.  I am not yet sure if this is worth the trouble, and
    also when this should be done (beginning or end of capture
    process).  Input is welcome.

6.7 Multi-key access
=====================

    If you like to use many different templates, you can now set them
    up in a way that multiple keys access a template.  This is very
    similar to the multi-key selection of agenda commands.  So we have
    special entries which "define/explain a prefix key", something like

    '(("t" "templates adding table lines")
      ("ta" "add to table a" table-line (file+headline "~/notes.org"  
"Table A))
      ("tb" "add to table b" table-line (file+headline "~/notes.org"  
"Table B))
      ("tc" "add to table c" table-line (file+headline "~/notes.org"  
"Table C)))

    When starting capture, you can then first press "t" and then see
    the individual options.


6.8 The code is now much much cleaner and easier to maintain
=============================================================

    Just look at it... :)

    The docstrings of `org-capture-templates' and of the command
    `org-capture' contain much of the documentation for the moment,
    until I can write it up for the manual (if we decide that this is
    the way to go.....

7 Request for comments
~~~~~~~~~~~~~~~~~~~~~~~

None of what I describe is set in stone yet - let me know if you have
comments, change requests or other ideas.

My feeling right now is that this should become the default capture
system, and that we will keep the current org-remember in the
distribution for quite some time, for compatibility.


- Carsten