Re: Scaling org-mode

[Matt Lundin <mdl@imapmail.org>]

d@teklibre.org (Dave Täht) writes:

> I have really been enjoying importing my life into org-mode, which I've
> been doing for about two months now. 
>
> But.
>
> It currently visits about 100 files and 10k of text to construct the
> agenda. It's starting to get kind of slow and interrupt my workflow,
> particularly the background process that scans them. 
>
> While the system is effectively frozen, my message buffer fills up with
> messages about setting the flyspell dictionary to en, etc. This is quite
> annoying with text to speech turned on. I ended up just having appts
> spoken.

My guess is that this is a flyspell problem. I only get messages about
flyspell if I don't have aspell or the proper dictionary installed. Do
you get these messages when you open buffers in other text modes? What
exactly are the messages? What is your flyspell config?

> Solution #1) cut the number of files down - is a good one. I probably
> can cut those files easily in half right now. The problem is that I have
> about 600 more files to import (scenes from a book), and I really like
> the idea of being able to know what my characters are doing in 2023, and
> separate files was kind of useful at one point.

One recommendation would be to leave files that are purely notes out of
your agenda. I noticed from your blog posts that you are adding
timestamps to events from your novel. Are these active or inactive
timestamps? My guess is that parsing all that data is slowing org-mode
down. You can always create a timeline from within a particular file
without including it in your normal agenda files. You can also create
custom agenda views to work on different sets of org files.

http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php

While org-mode is exceptionally robust as a plain text tool, if you need
to manage an ever-growing collection of thousands and thousands of
pieces of data, you'll probably be better served by 1) using a database
or 2) dividing your org-mode files into separate "collections" (i.e.,
reserving your regular agenda for only a subset of org files).

I keep my org files lean by regularly archiving subtrees. I can always
search the archives when I need to.

> so thought 2) would be to have it only attempt to construct background
> agendas when the system is otherwise idle for a few minutes. I don't
> know how to do that, I figure wrapping this bit with something that
> could detect idleness instead of just running arbitrarily would be good.
>
>   (run-at-time nil 3600 'org-agenda-to-appt)
>
> don't know how to detect idleness.

http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html

Best,
Matt