TODO state tracking in org-mode

TODO state tracking in org-mode

[Bernt Hansen]

Here's my attempt at documenting how I use task todo keywords and state
change tracking in org-mode.

* Some Background

I am a consultant who works with multiple clients doing programming and
system maintenance projects.

I have multiple org files which are in my org-agenda-files variable so
any tasks added to these end up in my agenda.  I live in the agenda.
Due to the number of tasks I have to work on the 'hot topics' end up
being scheduled so they are 'in my face' on the daily agenda until they
are completed.  Only when the agenda has no pending work to do for today
do I look for NEXT tasks to work on.

I typically have one org file per client with lots of projects and tasks
in it.  I archive completed projects/tasks monthly to de-clutter my org
files.  I throw lots of details into my org files so they tend to get
rather long.  Active Org files for clients are at least 1,000 lines long
normally; one of them is approaching 10,000 lines.  The archives are
bigger.

* Task States

I have a few different types of tasks in my org files.  They roughly
break down into the following groups with the keywords listed.

 - Notes and documentation
   (these don't get a todo keyword)
 - Purchase Orders
   - OPEN
   - CLOSED
 - Regular Tasks
   - TODO
   - NEXT
   - DONE
   - WAITING
   - CANCELLED
   - SOMEDAY
   - ONGOING

Notes and documentation normally do not get a task todo keyword.  It's
just a heading level under some project which I use to help organize
things.

I use OPEN and CLOSED todo keywords for Purchase orders only -- these
aren't normal tasks where you do things and then they're done.  The
purchase order is OPEN as long as you can bill against it and CLOSED
after that.  This just feels more natural to me than TODO/DONE.

I use ONGOING for tasks that do not have a well defined start and end.
Things like org-mode tuning :), reading email, administrative tasks for
clients etc.  I clock everything (or try to) so when I find I'm working
on something that isn't project specific but is more about organizing
things for a client/project I start clocking the appropriate ONGOING
task until I move on to something else.

All of the remaining task states are for regular tasks.  Tasks normally
go TODO -> NEXT -> DONE.  But things in life rarely go according to plan
so a task will often go to WAITING status, or be postponed indefinitely
(to SOMEDAY), or be CANCELLED without being complete.

* Task State Change Logging

I normally use a Remember template to create new project tasks.  This
creates the new task with a TODO keyword and today's date as the date
the task came into existence.  Subtasks are created when outlining the
project and no dates are normally recorded when they are created.

For a given task I want to record things like the following:

  - Why/What a task is waiting for when it goes to WAITING status
  - When a task is complete
  - When a task is cancelled
  - When a task is reopened (it was complete but now has work to do again)
  - When a task is finished waiting for something
  - When a purchase order is opened or closed

* Setup

I use the following setup in my .emacs for org-mode tasks.  Since I have
multiple org files I don't want to duplicate the settings for every
file.  I use an identical setup for every org file I use so I define
these globally.  I hate maintaining duplicated definitions.

,----[ .emacs settings ]
| (setq org-log-done t
|       org-use-fast-todo-selection t)
| 
| (setq org-todo-keyword-faces
|       '(("TODO"  . (:foreground "red" :weight bold))
| 	("NEXT"  . (:foreground "red" :weight bold))
| 	("DONE"  . (:foreground "forest green" :weight bold))
| 	("WAITING"  . (:foreground "orange" :weight bold))
| 	("CANCELLED"  . (:foreground "forest green" :weight bold))
| 	("SOMEDAY"  . (:foreground "orange" :weight bold))
| 	("OPEN"  . (:foreground "red" :weight bold))
| 	("CLOSED"  . (:foreground "forest green" :weight bold))
| 	("ONGOING"  . (:foreground "orange" :weight bold))))
| 
| (setq org-todo-keywords
|       '((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
| 	(sequence "WAITING(w@/!)" "|" "CANCELLED(c!/!)")
| 	(sequence "SOMEDAY(s!/!)" "|")
| 	(sequence "OPEN(O!)" "|" "CLOSED(C!)")
| 	(sequence "ONGOING(o)" "|")))
`----

Defining org-log-done to true tells org-mode to record a CLOSED: tag and
date/time stamp when a task is completed (either DONE, CANCELLED, or
CLOSED).  Moving the task back to an active todo state removes this tag.

org-todo-keyword-faces defines what colours to use for the various todo
keywords.  It's not strictly necessary but it's nice :)

org-todo-keywords defines the todo keywords and when extra information
should be recorded.  I only use sequence todo keywords. Keywords to the
left of the '|' are todo states and keywords to the right of the '|' are
done (completed) states in the sequence.  S-left and S-right cycles
through the states in the sequence.

  - TODO(t) sets the key 't' to be the fast todo selection key for
    this todo state.  Nothing special is recorded.
  - WAITING(w@/!) defines the WAITING keyword with 'w' as a fast todo
    selection key.  The '@' tells org-mode to record a note and a
    date/time stamp when entering this todo state.  The '/!' tells
    org-mode to also record a date/time stamp when leaving this state.
  - OPEN(O!) sets the key 'O' to be the fast todo selection key for
    this todo state and the '!' tells org-mode to record a date/time
    stamp when entering this state.
  - DONE(d!/!) sets the key 'd' to be the fast todo selection key for
    this state.  The '!' before the '/' tells org-mode to record a
    date/time stamp when entering this todo state and the '!' after
    the '/' tells org-mode to record a date/time stamp when leaving
    this state.

Here's the result of this logging on a sample task.

** DONE New Sample Task
      CLOSED: [2008-03-21 Fri 18:48]
      - State "DONE"       [2008-03-21 Fri 18:48]
      - State "NEXT"       [2008-03-21 Fri 18:47]
      - State "DONE"       [2008-03-21 Fri 18:45]
      - State "ONGOING"    [2008-03-21 Fri 18:44]
      - State "WAITING"    [2008-03-21 Fri 18:34] \\
	For Someone(tm) to finish Something(tm)
      [2008-03-21 Fri]
      
      1. This task was created on Friday Mar 21, 2008
      2. It moved to WAITING status at 18:34 with a note that it's
         waiting for Someone(tm)
      3. It left WAITING status and went to ONGOING at 18:47
	 ONGOING doesn't normally record a timestamp but since
	 we're leaving the WAITING state a timestamp is recorded
      4. Then it was DONE (we thought)
      5. But it later moved to NEXT because we marked it DONE too early
      6. And then it was finally completely DONE

** CLOSED Sample Purchase Order
	  CLOSED: [2008-03-21 Fri 19:03]
	  - State "CLOSED"     [2008-03-21 Fri 19:03]
	  - State "OPEN"       [2008-03-21 Fri 19:03]

This setup works very well for me.  I'm sure it can still be improved
but I thought I'd share it with the rest of you in case you find it
useful.

Regards,
Bernt

Re: TODO state tracking in org-mode

[Manish]

On Sat, Mar 22, 2008 at 5:57 AM, Bernt Hansen <bernt@norang.ca> wrote:
> Here's my attempt at documenting how I use task todo keywords and state
>  change tracking in org-mode.
>
>  * Some Background
>
>  I am a consultant who works with multiple clients doing programming and
>  system maintenance projects.
>
>  I have multiple org files which are in my org-agenda-files variable so
>  any tasks added to these end up in my agenda.  I live in the agenda.
>  Due to the number of tasks I have to work on the 'hot topics' end up
>  being scheduled so they are 'in my face' on the daily agenda until they
>  are completed.  Only when the agenda has no pending work to do for today
>  do I look for NEXT tasks to work on.
>
>  I typically have one org file per client with lots of projects and tasks
>  in it.  I archive completed projects/tasks monthly to de-clutter my org
>  files.  I throw lots of details into my org files so they tend to get
>  rather long.  Active Org files for clients are at least 1,000 lines long
>  normally; one of them is approaching 10,000 lines.  The archives are
>  bigger.
>
>  * Task States
>
>  I have a few different types of tasks in my org files.  They roughly
>  break down into the following groups with the keywords listed.
>
>   - Notes and documentation
>    (these don't get a todo keyword)
>   - Purchase Orders
>    - OPEN
>    - CLOSED
>   - Regular Tasks
>    - TODO
>    - NEXT
>    - DONE
>    - WAITING
>    - CANCELLED
>    - SOMEDAY
>    - ONGOING
>
>  Notes and documentation normally do not get a task todo keyword.  It's
>  just a heading level under some project which I use to help organize
>  things.
>
>  I use OPEN and CLOSED todo keywords for Purchase orders only -- these
>  aren't normal tasks where you do things and then they're done.  The
>  purchase order is OPEN as long as you can bill against it and CLOSED
>  after that.  This just feels more natural to me than TODO/DONE.
>
>  I use ONGOING for tasks that do not have a well defined start and end.
>  Things like org-mode tuning :), reading email, administrative tasks for
>  clients etc.  I clock everything (or try to) so when I find I'm working
>  on something that isn't project specific but is more about organizing
>  things for a client/project I start clocking the appropriate ONGOING
>  task until I move on to something else.
>
>  All of the remaining task states are for regular tasks.  Tasks normally
>  go TODO -> NEXT -> DONE.  But things in life rarely go according to plan
>  so a task will often go to WAITING status, or be postponed indefinitely
>  (to SOMEDAY), or be CANCELLED without being complete.
>
>  * Task State Change Logging
>
>  I normally use a Remember template to create new project tasks.  This
>  creates the new task with a TODO keyword and today's date as the date
>  the task came into existence.  Subtasks are created when outlining the
>  project and no dates are normally recorded when they are created.
>
>  For a given task I want to record things like the following:
>
>   - Why/What a task is waiting for when it goes to WAITING status
>   - When a task is complete
>   - When a task is cancelled
>   - When a task is reopened (it was complete but now has work to do again)
>   - When a task is finished waiting for something
>   - When a purchase order is opened or closed
>
>  * Setup
>
>  I use the following setup in my .emacs for org-mode tasks.  Since I have
>  multiple org files I don't want to duplicate the settings for every
>  file.  I use an identical setup for every org file I use so I define
>  these globally.  I hate maintaining duplicated definitions.
>
>  ,----[ .emacs settings ]
>  | (setq org-log-done t
>  |       org-use-fast-todo-selection t)
>  |
>  | (setq org-todo-keyword-faces
>  |       '(("TODO"  . (:foreground "red" :weight bold))
>  |       ("NEXT"  . (:foreground "red" :weight bold))
>  |       ("DONE"  . (:foreground "forest green" :weight bold))
>  |       ("WAITING"  . (:foreground "orange" :weight bold))
>  |       ("CANCELLED"  . (:foreground "forest green" :weight bold))
>  |       ("SOMEDAY"  . (:foreground "orange" :weight bold))
>  |       ("OPEN"  . (:foreground "red" :weight bold))
>  |       ("CLOSED"  . (:foreground "forest green" :weight bold))
>  |       ("ONGOING"  . (:foreground "orange" :weight bold))))
>  |
>  | (setq org-todo-keywords
>  |       '((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
>  |       (sequence "WAITING(w@/!)" "|" "CANCELLED(c!/!)")
>  |       (sequence "SOMEDAY(s!/!)" "|")
>  |       (sequence "OPEN(O!)" "|" "CLOSED(C!)")
>  |       (sequence "ONGOING(o)" "|")))
>  `----
>
>  Defining org-log-done to true tells org-mode to record a CLOSED: tag and
>  date/time stamp when a task is completed (either DONE, CANCELLED, or
>  CLOSED).  Moving the task back to an active todo state removes this tag.
>
>  org-todo-keyword-faces defines what colours to use for the various todo
>  keywords.  It's not strictly necessary but it's nice :)
>
>  org-todo-keywords defines the todo keywords and when extra information
>  should be recorded.  I only use sequence todo keywords. Keywords to the
>  left of the '|' are todo states and keywords to the right of the '|' are
>  done (completed) states in the sequence.  S-left and S-right cycles
>  through the states in the sequence.
>
>   - TODO(t) sets the key 't' to be the fast todo selection key for
>     this todo state.  Nothing special is recorded.
>   - WAITING(w@/!) defines the WAITING keyword with 'w' as a fast todo
>     selection key.  The '@' tells org-mode to record a note and a
>     date/time stamp when entering this todo state.  The '/!' tells
>     org-mode to also record a date/time stamp when leaving this state.
>   - OPEN(O!) sets the key 'O' to be the fast todo selection key for
>     this todo state and the '!' tells org-mode to record a date/time
>     stamp when entering this state.
>   - DONE(d!/!) sets the key 'd' to be the fast todo selection key for
>     this state.  The '!' before the '/' tells org-mode to record a
>     date/time stamp when entering this todo state and the '!' after
>     the '/' tells org-mode to record a date/time stamp when leaving
>     this state.
>
>  Here's the result of this logging on a sample task.
>
>  ** DONE New Sample Task
>       CLOSED: [2008-03-21 Fri 18:48]
>       - State "DONE"       [2008-03-21 Fri 18:48]
>       - State "NEXT"       [2008-03-21 Fri 18:47]
>       - State "DONE"       [2008-03-21 Fri 18:45]
>       - State "ONGOING"    [2008-03-21 Fri 18:44]
>       - State "WAITING"    [2008-03-21 Fri 18:34] \\
>         For Someone(tm) to finish Something(tm)
>       [2008-03-21 Fri]
>
>       1. This task was created on Friday Mar 21, 2008
>       2. It moved to WAITING status at 18:34 with a note that it's
>          waiting for Someone(tm)
>       3. It left WAITING status and went to ONGOING at 18:47
>          ONGOING doesn't normally record a timestamp but since
>          we're leaving the WAITING state a timestamp is recorded
>       4. Then it was DONE (we thought)
>       5. But it later moved to NEXT because we marked it DONE too early
>       6. And then it was finally completely DONE
>
>  ** CLOSED Sample Purchase Order
>           CLOSED: [2008-03-21 Fri 19:03]
>           - State "CLOSED"     [2008-03-21 Fri 19:03]
>           - State "OPEN"       [2008-03-21 Fri 19:03]
>
>  This setup works very well for me.  I'm sure it can still be improved
>  but I thought I'd share it with the rest of you in case you find it
>  useful.
>

Great!  I picked up a few ideas.  Thanks for sharing.

It would be even better if you could also share your remember
templates setup and OS and version information.

-- Manish

Re: TODO state tracking in org-mode

[Bernt Hansen]

Manish <mailtomanish.sharma@gmail.com> writes:

> It would be even better if you could also share your remember
> templates setup and OS and version information.

Here's a bunch of org-related settings.

OS version is Linux (Debian unstable = sid)

GNU Emacs 22.1.1 (i486-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of 2007-12-15 on noname, modified by Debian
Org-mode version 5.23a++

My remember templates are pretty straight forward and taken from a
previous post on the list.  Here's a whole pile of settings from my
.emacs file...

(add-hook 'remember-mode-hook 'my-start-clock-if-needed)
(defun my-start-clock-if-needed ()
  (save-excursion
    (goto-char (point-min))
    (when (re-search-forward ":CLOCK-IN:" nil t)
       (replace-match "")
       (org-clock-in))))

(defun my-org-todo ()
  (interactive)
  (org-narrow-to-subtree)
  (org-show-todo-tree nil)
  (widen))

;; A bunch of key bindings for convenience
;;
(global-set-key (kbd "C-<f5>") 'compile)
(global-set-key (kbd "<f5>") 'my-org-todo)
(global-set-key (kbd "<f6>") 'org-toggle-archive-tag)
(global-set-key (kbd "<f7>") 'set-truncate-lines)
(global-set-key (kbd "C-<f8>") (lambda () (interactive) (find-file "~/org/todo.org")))
(global-set-key (kbd "<f8>") 'org-cycle-agenda-files)
(global-set-key (kbd "S-<f8>") (lambda () (interactive) (switch-to-buffer "*terminal*")))
(global-set-key (kbd "<f9> b") 'bbdb)
(global-set-key (kbd "<f9> c") 'calendar)
(global-set-key (kbd "<f9> f") 'boxquote-insert-file)
(global-set-key (kbd "<f9> g") 'gnus)
(global-set-key (kbd "<f9> M-g") 'gnus-unplugged)
(global-set-key (kbd "<f9> o") 'org-occur)
(global-set-key (kbd "<f9> p") 'my-phone-call)
(global-set-key (kbd "<f9> r") 'boxquote-region)
(global-set-key (kbd "<f9> u") (lambda () (interactive) (untabify (point-min) (point-max))))
(global-set-key (kbd "<f9> v") 'visible-mode)
(global-set-key (kbd "C-<f9>") 'previous-buffer)
(global-set-key (kbd "C-M-<f9>") 'org-narrow-to-subtree)
(global-set-key (kbd "C-M-<f10>") 'widen)
(global-set-key (kbd "C-<f10>") 'next-buffer)
(global-set-key (kbd "<f12>") 'org-agenda)
(global-set-key (kbd "C-<f12>") 'org-clock-goto)
(global-set-key (kbd "C-M-r") 'org-remember)

(require 'remember)
;
(add-hook 'remember-mode-hook 'org-remember-apply-template)
;
(setq my-phone-remember-templates
      (quote (("client1" ?c "* Phone Call - %:name - %:company           :phone:\n  :CLOCK-IN:%?" "~/org/client1.org" "Tasks")
	      ("client2" ?d "* Phone Call - %:name - %:company           :phone:\n  :CLOCK-IN:%?" "~/org/client2.org" "Tasks")
	      ("norang" ?n "* Phone Call - %:name - %:company           :phone:\n  :CLOCK-IN:%?" "~/org/norang.org" "Tasks")
	      ("phone call" ?p "* Phone Call - %? -            :phone:\n  :CLOCK-IN:" "~/org/todo.org" "Tasks"))))

(custom-set-variables
  ;; custom-set-variables was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 '(org-agenda-custom-commands (quote (("W" "Waiting Tasks" tags "/!WAITING" nil) ("n" "Next Tasks" tags "/!NEXT" nil) ("N" "Next Tasks - No Specified Duration" tags "Effort={^$}/!NEXT" nil) ("h" "Next Home Tasks" tags "@home/!NEXT" nil) ("w" "Next Work Tasks" tags "@work/!NEXT" nil) ("E" "Errands To Run" tags "@errand/!-DONE-CANCELLED" nil) ("S" "Stuff To Do Someday" tags "/!SOMEDAY" nil) ("x" "New Tasks" tags-todo "newtask" nil) ("o" "Ongoing Tasks" tags "/!ONGOING" nil))))
 '(org-agenda-files (quote ("~/org/todo.org" "~/org/client1.org" "~/org/client2.org" "~/org/norang.org" "~/org/petproject.org" "~/org/specproj.org")))
 '(org-agenda-show-all-dates t)
 '(org-agenda-skip-deadline-if-done t)
 '(org-agenda-skip-scheduled-if-done t)
 '(org-agenda-start-on-weekday nil)
 '(org-deadline-warning-days 14)
 '(org-fast-tag-selection-single-key (quote expert))
 '(org-remember-store-without-prompt t)
 '(org-remember-templates (quote (("todo" 116 "* TODO %?
  %u" "~/org/todo.org" "Tasks") ("client1" 99 "* TODO %?
  %u" "~/org/client1.org" "Tasks") ("client2" 100 "* TODO %?
  %u" "~/org/client2.org" "Tasks") ("norang" 110 "* TODO %?
  %u" "~/org/norang.org" "Tasks") ("petproject" 98 "* TODO %?
  %u" "~/org/petproject.org" "Tasks"))))
 '(org-reverse-note-order t)
 '(org-stuck-projects (quote ("LEVEL=2-notaproject-phone/!-DONE-CANCELLED-ONGOING-SOMEDAY-WAITING" ("NEXT") nil "")))
 '(outline-regexp "\\(?:\\*+\\|=+ \\)")
 '(remember-annotation-functions (quote (org-remember-annotation)))
 '(remember-handler-functions (quote (org-remember-handler)))
 '(safe-local-variable-values (quote ((org-drawers quote ("PROPERTIES" "HIDE"))))))
(custom-set-faces
  ;; custom-set-faces was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 )

-Bernt

Re: Re: TODO state tracking in org-mode

[Manish]

  ,----
  | On Sun, Mar 23, 2008 at 7:16 AM, Bernt Hansen wrote:
  | > Manish <mailtomanish.sharma@gmail.com> writes:
  | >
  | >  > It would be even better if you could also share your remember
  | >  > templates setup and OS and version information.
  | >
  | >  Here's a bunch of org-related settings.
  | >
  | >  OS version is Linux (Debian unstable = sid)
  `----

I envy you so much. :)

  ,----
  | >  My remember templates are pretty straight forward and taken from a
  | >  previous post on the list.  Here's a whole pile of settings from my
  | >  .emacs file...
  `----

Ah. I recall now.  Do you think
"(add-hook 'remember-mode-hook 'my-start-clock-if-needed)"
should be replaced with
"(add-hook'remember-mode-hook 'my-start-clock-if-needed 'append)" as per
Nick's comments here:
http://article.gmane.org/gmane.emacs.orgmode/5655/match=my+start+clock+if+needed+append
Or may be it's not required anymore?

Thanks again for writing this up.  Posts like your's are a
motivation to get better organized and use org-mode more effectively.

-- Manish

Re: TODO state tracking in org-mode

[Bernt Hansen]

Manish <mailtomanish.sharma@gmail.com> writes:

> Ah. I recall now.  Do you think
> "(add-hook 'remember-mode-hook 'my-start-clock-if-needed)"
> should be replaced with
> "(add-hook'remember-mode-hook 'my-start-clock-if-needed 'append)" as per
> Nick's comments here:
> http://article.gmane.org/gmane.emacs.orgmode/5655/match=my+start+clock+if+needed+append
> Or may be it's not required anymore?

Possibly.  I haven't looked into that because what I have set up now
seems to work for me.  I've collected my settings over a long period of
time and I've had wrong settings before - so it wouldn't surprise me if
this should be changed.

Thanks,
Bernt

Re: Re: TODO state tracking in org-mode

[Carsten Dominik]

On Mar 23, 2008, at 12:28 PM, Bernt Hansen wrote:

> Manish <mailtomanish.sharma@gmail.com> writes:
>
>> Ah. I recall now.  Do you think
>> "(add-hook 'remember-mode-hook 'my-start-clock-if-needed)"
>> should be replaced with
>> "(add-hook'remember-mode-hook 'my-start-clock-if-needed 'append)"  
>> as per
>> Nick's comments here:
>> http://article.gmane.org/gmane.emacs.orgmode/5655/match=my+start+clock+if+needed+append
>> Or may be it's not required anymore?
>
> Possibly.  I haven't looked into that because what I have set up now
> seems to work for me.  I've collected my settings over a long period  
> of
> time and I've had wrong settings before - so it wouldn't surprise me  
> if
> this should be changed.

With the append option is actually the safer way of doing things.

- Carsten