processing pending emails as part of your GTD system

processing pending emails as part of your GTD system

[Pete Phillips]

This post is slighly off-topic, as it is not directly about org-mode.

As many of you on the list will know, I'm a big fan of David Allen's GTD
system, and over the years I have tweaked my use of MH and MH-E to help
support this use.

One thing I have done is design a method so that I can easily put emails
into a set of 'pending' mail folders, and then get cron to process these
and dump the emails back into my +inbox at appropriate dates.

Having mentioned this on the MH-E list recently, someone asked if I
could share the scripts with them, so I have written this up here:

  http://ccgi.philfam.co.uk/wordpress/2008/04/20/24

I suspect you could extend this to any email system where you have
command line access to your email system. 

Hope you find this useful.
Pete

--
Pete Phillips, Acting Director,     |   http://www.smtl.co.uk/
Surgical Materials Testing Lab,     |   http://www.worldwidewounds.com/
Princess of Wales Hospital, S Wales |   http://www.dressings.org/
Tel/Fax: +44 1656-752820/30         |   pete@smtl.co.uk

Re: processing pending emails as part of your GTD system

[Christopher League]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]

On Apr 21, 2008, at 2:55 AM, Pete Phillips wrote:
> One thing I have done is design a method so that I can easily put  
> emails
> into a set of 'pending' mail folders, and then get cron to process  
> these
> and dump the emails back into my +inbox at appropriate dates.

Interesting, I have a similar script that also requires cron/shell  
access to mail server, but works with mbox format instead of MH, and  
is compatible with, e.g., dovecot IMAP server.  Mine is slightly less  
fancy, in that I file messages to absolute folders named "days/25" or  
"months/07-July" (rather than "nextweek") to have them dumped back  
into my spool at the indicated time.  See the 'email-tickler-update'  
script attached.


More relevant to org-mode, I have an emacs-lisp/apple-script combo for  
pasting links to Apple Mail messages into org files:

(defun cal-grab-mail-links ()
   (interactive)
   (call-process "/usr/bin/osascript" nil t nil
                 "/home/league/Library/Scripts/Applications/Mail/Copy  
Message for OrgMode.scpt")
   (yank))
(define-key org-mode-map "\C-cm" 'cal-grab-mail-links)

;;;;; and the .scpt component:

-- Replace all occurences of one string for another in a text
-- The trick here is to change the internal delimiter,
-- spliting and joining the text
--
on replaceString(theText, oldString, newString)
	set AppleScript's text item delimiters to oldString
	set tempList to every text item of theText
	set AppleScript's text item delimiters to newString
	set theText to the tempList as string
	set AppleScript's text item delimiters to ""
	return theText
end replaceString

tell application "Mail"
	set _sel to get selection
	set _links to {}
	repeat with _msg in _sel
		set _subj to _msg's subject
		set _subj to my replaceString(_subj, "[", "(")
		set _subj to my replaceString(_subj, "]", ")")
		set _messageURL to "[[message://%3c" & _msg's message id & "%3e][" &  
_subj & "]]"
		set end of _links to _messageURL
	end repeat
	set AppleScript's text item delimiters to return
	set the clipboard to (_links as string)
end tell


[-- Attachment #2: email-tickler-update.txt --]
[-- Type: text/plain, Size: 2160 bytes --]

#!/usr/bin/env zsh
## email-tickler-update -- manage a set of 43 mailboxes as a 'tickler' file
## Written by and for Christopher League <league@contrapunctus.net>
## but released to the public domain.

## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

                           ### Settings ###

mail_spool=/var/mail
mail_dir=mail

today=$(date +%d)                            # 03, 14, 31, ...
month=$(date +%m-%b)                         # 02-Feb, 11-Nov, ...

day_mbox=${mail_dir}/days/${today}
month_mbox=${mail_dir}/months/${month}

users=($*)

                       ### Helper functions ###

## Maybe run a command; maybe print it instead.  If this script is run
## with DRYRUN set (to anything except empty string), it will avoid
## taking any real actions, and just print the commands.
run() {
    if [[ -z $DRYRUN ]]; then   # do it for real
        $*
    else                        # output only
        print '%' $*
    fi
}

## Run a command, but exit with message on error.
guard() {
    run $*
    if [[ $? != 0 ]]; then
        print "FATAL($?): $*"
        exit $?
    fi
}

readable() {
    if [[ ! -r $1 ]]; then
        print unreadable: $1
        exit 1
    fi
}

                       ### Primary actions ###

## Append given file to mail spool, then empty it out.
move_to_spool() {
    cat $1 >>${user_mail_spool}
    print -n >$1
}

## This wraps the above with the proper locking protocol, and does it
## only if given mailbox is non-empty.
protected_move() {
    guard lockfile $1.lock
    if [[ -s $1 ]]; then
        guard lockfile ${user_mail_spool}.lock
        guard move_to_spool $1
        guard rm -f ${user_mail_spool}.lock
    fi
    guard rm -f $1.lock
}

                          ### Main loop ###

for u in $users; do
    user_mail_spool=${mail_spool}/$u
    readable ${user_mail_spool}
    readable ~$u/${day_mbox}
    readable ~$u/${month_mbox}
    protected_move ~$u/${day_mbox}
    if [[ $today == 01 ]]; then
        protected_move ~$u/${month_mbox}
    fi
done

exit 0

[-- Attachment #3: Type: text/plain, Size: 1 bytes --]



[-- Attachment #4: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: processing pending emails as part of your GTD system

[Erik Hetzner]

[-- Attachment #1.1.1: Type: text/plain, Size: 1310 bytes --]

At Mon, 21 Apr 2008 07:55:13 +0100,
Pete Phillips <pete@smtl.co.uk> wrote:
> This post is slighly off-topic, as it is not directly about org-mode.
> 
> As many of you on the list will know, I'm a big fan of David Allen's GTD
> system, and over the years I have tweaked my use of MH and MH-E to help
> support this use.
> 
> One thing I have done is design a method so that I can easily put emails
> into a set of 'pending' mail folders, and then get cron to process these
> and dump the emails back into my +inbox at appropriate dates.
>
> […]

This is a neat trick. I’m giving it a try to help me keep my inbox as
empty as possible.

However, I use IMAP & Wanderlust. Attached is a quick Ruby script that
should provide similar functionality for any IMAP system, and some
cron entries to call it, for anybody who wants it. This works for me,
but I have commented out the lines in the script which actually delete
mail from one mailbox; I don’t want to be responsible for losing mail.

The mailboxes used by the cron entries are (a little different from
yours):

pending/tomorrow
pending/tonight
pending/nextweek
pending/d01 .. pending/day31
pending/january .. pending/december
pending/monday .. pending/sunday
today

Hope this helps somebody.

best,
Erik Hetzner


[-- Attachment #1.1.2: crontab-example --]
[-- Type: application/octet-stream, Size: 1194 bytes --]

IMAP_PASSWORD=...
IMAP_USER=...
IMAP_HOST=...
IMAP_INBOX=INBOX

#############################################################
#  Mail Section
#############################################################
### Each month, refile all the pending stuff back to my inbox.
### 00:01 on the first day of each month
1 0 1 * * $HOME/bin/imap-dump-folder.rb pending/`date +\%B | tr A-Z a-z` $IMAP_INBOX
### Each monday at 00:02 refile all the next-week stuff
2 0 * * 1 $HOME/bin/imap-dump-folder.rb pending/nextweek $IMAP_INBOX
### Each day also has its own pending - Saturday, Sunday etc
3 0 * * * $HOME/bin/imap-dump-folder.rb pending/`date +\%A | tr A-Z a-z` $IMAP_INBOX
### Each date of the month (1-31) also has its own pending - d01, d11, d31 etc
4 0 * * * $HOME/bin/imap-dump-folder.rb pending/d`date +\%d` $IMAP_INBOX
### Each night at 17:30 - refile all pending/tonight to INBOX
30 17 * * * $HOME/bin/imap-dump-folder.rb pending/tonight $IMAP_INBOX
### Each night at 12:30 - refile all today stuff back to INBOX
30 00 * * * $HOME/bin/imap-dump-folder.rb today $IMAP_INBOX
# Each day refile the tomorrow pending back to INBOX
5 0 * * * $HOME/bin/imap-dump-folder.rb pending/tomorrow $IMAP_INBOX


[-- Attachment #1.1.3: imap-dump-folder.rb --]
[-- Type: application/octet-stream, Size: 1632 bytes --]

#!/usr/bin/env ruby
require 'net/imap'

#defaults
host = ENV['IMAP_HOST'] || ''
port = (ENV['IMAP_PORT'] || '143').to_i
use_ssl = !(ENV['IMAP_SSL'].nil?)
user = ENV['IMAP_USER'] || ''
password = ENV['IMAP_PASSWORD'] || ''
source_folder = ""
target_folder = ""
help_me = false

def imap_dump_folder(host, port, use_ssl, user, password, source_folder, target_folder)
  imap = Net::IMAP.new(host, port, use_ssl)
  imap.login(user, password)
  imap.select(source_folder)
  (1..imap.responses["EXISTS"][-1]).each do |mid|
    imap.copy(mid, target_folder)
    #imap.store(mid, "+FLAGS", [:Deleted])
  end
  #imap.expunge()
end

def usage(do_exit = true)
  STDERR.print "usage: #{$0} [options] source target
  --host=string        host server name
  --port=string        host server port
  --ssl                use ssl
  --user=string        username
  --password=string    password
  --help               show this message.
"
  exit if do_exit
end

ARGV.each do |option|
  case (option)
  when /^--host=(.+)/
    host = $1
  when /^--port=([0-9]+)/
    port = $1.to_i
  when /^--user=(.+)/
    user = $1
  when /^--password=(.+)/
    password = $1
  when /^--ssl/
    use_ssl = true
  when /^--help/
    help_me = true
  when /^(.+)/
    if (source_folder == '') then
      source_folder = $1
    elsif (target_folder == '') then
      target_folder = $1
    else
      usage()
    end
  else
    usage()
  end
end

if (host == "" or user == "" or password == "" or source_folder == "" or target_folder == "" or help_me == true) then
  usage(true)
end

imap_dump_folder(host, port, use_ssl, user, password, source_folder, target_folder)

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: processing pending emails as part of your GTD system

[Adam Spiers]

Pete Phillips (pete@smtl.co.uk) wrote:
> This post is slighly off-topic, as it is not directly about org-mode.
> 
> As many of you on the list will know, I'm a big fan of David Allen's GTD
> system, and over the years I have tweaked my use of MH and MH-E to help
> support this use.
> 
> One thing I have done is design a method so that I can easily put emails
> into a set of 'pending' mail folders, and then get cron to process these
> and dump the emails back into my +inbox at appropriate dates.

That's interesting.  I'm also a fan of GTD and therefore also love
being able to get mail out of the inbox into the archives, so that the
inbox only contains unprocessed mail.  However, as I mentioned a while
back on this list, I take a completely different approach to
implementing the "tickler" mechanism for allowing an archived mail to
regain visibility on a particular date:

  http://article.gmane.org/gmane.emacs.orgmode/2513/
  http://article.gmane.org/gmane.emacs.orgmode/5595/

etc.

Re: processing pending emails as part of your GTD system

[Carsten Dominik]

On Apr 30, 2008, at 11:25 PM, Adam Spiers wrote:

> Pete Phillips (pete@smtl.co.uk) wrote:
>> This post is slighly off-topic, as it is not directly about org-mode.
>>
>> As many of you on the list will know, I'm a big fan of David  
>> Allen's GTD
>> system, and over the years I have tweaked my use of MH and MH-E to  
>> help
>> support this use.
>>
>> One thing I have done is design a method so that I can easily put  
>> emails
>> into a set of 'pending' mail folders, and then get cron to process  
>> these
>> and dump the emails back into my +inbox at appropriate dates.
>
> That's interesting.  I'm also a fan of GTD and therefore also love
> being able to get mail out of the inbox into the archives, so that the
> inbox only contains unprocessed mail.  However, as I mentioned a while
> back on this list, I take a completely different approach to
> implementing the "tickler" mechanism for allowing an archived mail to
> regain visibility on a particular date:
>
>  http://article.gmane.org/gmane.emacs.orgmode/2513/
>  http://article.gmane.org/gmane.emacs.orgmode/5595/

Hi Adam,

I remember these very interesting posts, but I don't see the relation to
a tickler file.  How do you get back to a certain email on a specific  
date?
By scheduling a todo containing the link?

- Carsten

>
>
> etc.
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: processing pending emails as part of your GTD system

[Adam Spiers]

On Thu, May 01, 2008 at 08:32:59AM +0200, Carsten Dominik wrote:
> On Apr 30, 2008, at 11:25 PM, Adam Spiers wrote:
> >That's interesting.  I'm also a fan of GTD and therefore also love
> >being able to get mail out of the inbox into the archives, so that the
> >inbox only contains unprocessed mail.  However, as I mentioned a while
> >back on this list, I take a completely different approach to
> >implementing the "tickler" mechanism for allowing an archived mail to
> >regain visibility on a particular date:
> >
> > http://article.gmane.org/gmane.emacs.orgmode/2513/
> > http://article.gmane.org/gmane.emacs.orgmode/5595/
> 
> Hi Adam,
> 
> I remember these very interesting posts, but I don't see the relation to
> a tickler file.  How do you get back to a certain email on a specific  
> date?
> By scheduling a todo containing the link?

Yes, or by setting a deadline.  Then the org agenda "tickles" you at the
right time, and gives you the link to follow to the original material.

Re: processing pending emails as part of your GTD system

[bva]

On 2008-04-23 Wed, at 14:05, emacs-orgmode-request@gnu.org wrote:

>
> Message: 3
> Date: Tue, 22 Apr 2008 12:55:55 -0400
> From: Christopher League <league@contrapunctus.net>
> Subject: Re: [Orgmode] processing pending emails as part of your GTD
> 	system
> To: org-mode mailing list <emacs-orgmode@gnu.org>
> Message-ID: <3D9B5F2E-A15C-45C6-AF49-8E0E62248936@contrapunctus.net>
> Content-Type: text/plain; charset="us-ascii"
>
>
> More relevant to org-mode, I have an emacs-lisp/apple-script combo for
> pasting links to Apple Mail messages into org files:
>
Wow.  Thank you very much for this.

I've tried to do something like this, but I'm a read-only coder.

I've been using Apple's Mail program since I got my first Mac 5 years  
ago, and only recently (since 2007-08) started using GTD and org- 
mode.  I've been very frustrated by some of the time-consuming choices  
I've made to keep Mail and 'projects.org' linked.

How did you know about this ``message://%3c%3e'' schema syntax!?  Is  
there anything else I can do to Mail from within Emacs (or from within  
Quicksilver {see footnote}).  Where can I read more! Is it safe to  
upgrade Mail (i.e. if I upgrade to OS X 10.666 will this script  
break?) It seems that the message is found even if I move it to  
another folder within Mail after generating the link; anything I need  
to be warned about?  I often have two message viewers open (one with  
mailboxes showing and one without, for easier filing {see footnote})  
-- how does your script (or Mail) choose which one is 'selected'?

That's a lot questions ... feel free to redirect me to a resource to  
read first instead of taking your time repeating well-known details of  
Mail's internals.  It's just that the Dictionary of Mail (in Script  
Editor) isn't very comprehensible to me.  I don't speak AppleScript.

Two issues with the the emacs-lisp and applescript.

1) Why doesn't the applescript work for the messages in the first  
folder I tried - my GTD folder?  Messages from the org-mode digest are  
automatically filed to "On My Mac" -> "Reference" -> "GHI" -> "GTD" to  
help keep my inbox clean.  But any message I try to run your script on  
gives me blank "id" properties.

Example: [[message://%3c%3e][Emacs-orgmode Digest, Vol 26, Issue 54]]

when I click on this link in my org-mode file, Mail generates an error  
dialog with the message "No associated application could be found."

Every other message in any other folder I've tried works fine. Strange.

2) The emacs-lisp code calls (yank) which doesn't grab text from the  
Apple Clipboard for me.  I am using "GNU Emacs 22.1.50.1 (powerpc- 
apple-darwin7.9.0, Carbon Version 1.6.0) of 2007-10-02 on  
applecore.inf.ed.ac.uk - Aquamacs Distribution 1.2a" and org-version  
5.23a.  Instead, I modified your code to use (cua-paste).

Now I've just got to figure out where to put this code!  Thank you  
ever so much!

-Ben

{footnote}
I've tried to select multiple messages in Mail and then use  
Quicksilver to re-file them all in one go.  I tested this (I thought)  
then spent hours creating a whole 'Reference' folder hierarchy.  But  
it just doesn't work! If I could find a way for the currently selected  
messages to be passed to Quicksilver AND also use quicksilver to  
select the folders (the search capabilities are great!) maybe I could  
cobble something together.

Right now, I've decided that spending time to automate this isn't  
worth the time saved.

{end footnote}

> (defun cal-grab-mail-links ()
>   (interactive)
>   (call-process "/usr/bin/osascript" nil t nil
>                 "/home/league/Library/Scripts/Applications/Mail/Copy
> Message for OrgMode.scpt")
>   (yank))
> (define-key org-mode-map "\C-cm" 'cal-grab-mail-links)
>
> ;;;;; and the .scpt component:
>
> -- Replace all occurences of one string for another in a text
> -- The trick here is to change the internal delimiter,
> -- spliting and joining the text
> --
> on replaceString(theText, oldString, newString)
> 	set AppleScript's text item delimiters to oldString
> 	set tempList to every text item of theText
> 	set AppleScript's text item delimiters to newString
> 	set theText to the tempList as string
> 	set AppleScript's text item delimiters to ""
> 	return theText
> end replaceString
>
> tell application "Mail"
> 	set _sel to get selection
> 	set _links to {}
> 	repeat with _msg in _sel
> 		set _subj to _msg's subject
> 		set _subj to my replaceString(_subj, "[", "(")
> 		set _subj to my replaceString(_subj, "]", ")")
> 		set _messageURL to "[[message://%3c" & _msg's message id & "%3e][" &
> _subj & "]]"
> 		set end of _links to _messageURL
> 	end repeat
> 	set AppleScript's text item delimiters to return
> 	set the clipboard to (_links as string)
> end tell
>

Re: processing pending emails as part of your GTD system

[Christopher League]

On Apr 23, 2008, at 11:34 AM, bva@alexanderonline.org wrote:
> How did you know about this ``message://%3c%3e'' schema syntax!?  Is  
> there anything else I can do to Mail from within Emacs (or from  
> within Quicksilver {see footnote}).  Where can I read more! Is it  
> safe to upgrade Mail (i.e. if I upgrade to OS X 10.666 will this  
> script break?)

Things like this change all the time between OS revisions, so no  
guarantee.  I think the message URLs are new in Leopard (10.5).  You  
can find some details here:

http://daringfireball.net/2007/12/message_urls_leopard_mail

The macosxhints.com site is good too for tricks like that.  I think I  
just modified someone else's "Copy Message URL" script to put it in  
the right format for org-mode.  I'm no apple-script expert either.

> It seems that the message is found even if I move it to another  
> folder within Mail after generating the link; anything I need to be  
> warned about?

Yes, Mail in Leopard indexes all the messages and (usually) follows  
them through refiles.  Once in a while a linked message can't be  
found, but if I revisit the folder or "synchronize with server", then  
it works again.

> I often have two message viewers open (one with mailboxes showing  
> and one without, for easier filing {see footnote}) -- how does your  
> script (or Mail) choose which one is 'selected'?

No idea, likely the last Mail window with focus?

> 1) Why doesn't the applescript work for the messages in the first  
> folder I tried - my GTD folder?  Messages from the org-mode digest  
> are automatically filed to "On My Mac" -> "Reference" -> "GHI" ->  
> "GTD" to help keep my inbox clean.  But any message I try to run  
> your script on gives me blank "id" properties.
>
> Example: [[message://%3c%3e][Emacs-orgmode Digest, Vol 26, Issue 54]]

Hm, that %3c%3e means that there was no message-ID field for that  
message.  If you look at "Long Headers" and there IS a "Message-ID:"  
field (there is on org-mode messages for me), then maybe it has to do  
with Mail's index after auto-filing.  I do my auto-filing on the  
server with procmail, so I never use Mail's facility for that.

> 2) The emacs-lisp code calls (yank) which doesn't grab text from the  
> Apple Clipboard for me.  I am using "GNU Emacs 22.1.50.1 (powerpc- 
> apple-darwin7.9.0, Carbon Version 1.6.0) of 2007-10-02 on  
> applecore.inf.ed.ac.uk - Aquamacs Distribution 1.2a" and org-version  
> 5.23a.  Instead, I modified your code to use (cua-paste).

Okay, simple enough.  Mine is GNU Emacs 22.2.1 (i386-apple-darwin9,  
Carbon Version 1.6.0) [compiled from emacs22-carbon in Fink].

I use Quicksilver, but don't have much in the way of Mail  
integration.  Let me know if you find something.


Chris