Re: Having (too) many files in org-agenda-files

[Matt Lundin <mdl@imapmail.org>]

Hi Sebastian, 

I wrote this response yesterday, but it may have gotten lost in gmane's
reverse obfuscation system for email addresses. :)

(OT: Is there a "correct" way to respond to your messages in gnus? I
subscribe to org-mode via gmane/nntp and in your emails the org-mode ML
address is obfuscated and redirected:
emacs-orgmode-mXXj517/zsQ@public.gmane.org.)

-----

Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org>
writes:

> Of course, I have many, many files in Org mode. All files I write (or
> touch) in fact.
>
> Of course, I would like to search through my files at some point in time.
> I even would like to search through your files at some point in time, I mean
> through =org-mode/contrib/babel= and =Worg= for example.
>

The drawback of org-mode's plain text format is that org-mode needs to
load, parse, and fontify, and preserve in memory all files it queries
(i.e., agenda files). Too many agenda files can thus cause a pretty big
performance hit (both when loading agenda files and constructing agenda
views). To avoid this, I would recommend including only important files
in org-agenda-files. Possible solutions for searching other files
include:

1. loading the other directories when needed (e.g., through a function
   that changes the value of org-agenda-files)
2. using the variable org-agenda-text-search-extra-files (see below)
3. searching other directories with external tools (grep, perl script,
   etc.)

> The problem is the load-time of my Emacs, now:
>
> Emacs Init startup time: 221 seconds.

Do you invoke an agenda command in your emacs, such as
org-agenda-to-appt? That would cause org-mode to load all org files.

> coming from 20 seconds before the heavy use of Org...
>
> with tens of times such lines in my *Messages* buffer:
>
> OVERVIEW
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps.............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps..............................................)
> Checking for library `filladapt'... Found
> +-> Requiring `outline-mode-easy-bindings' (already loaded)
> +-> Requiring `ispell' (already loaded)
> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
> Starting new Ispell process [en_US] ...
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found

It seems that filladapt is causing some of the verbosity (and possible
slowness) there. When loading agenda files, my *Messages* buffer only
shows:

OVERVIEW [29 times]

> You'll tell me: not a problem, you do that only once a day, and you
> use Emacs client/server for the rest of the time. True. A bit, because
> I sometimes have to restart Emacs for testing a fresh one (not
> impacted by defvars already defined, or deffaces, etc.).
>
> Having to wait almost 4 minutes is a real pain. So, here my
> comments/questions:
>
> - Isn't it possible to delay the fontification/ispell/etc. to when we really
>   display (i.e., pop up) the buffer?  I guess this must be a major component
>   of the time this takes.

I imagine the fontification is necessary for the agenda to function
properly. Thus, org files are parsed en masse when the agenda is called
for the first time. (Hence my question about whether you were calling
the agenda from ~/.emacs.)

>
> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you want
>   have scanned for the agenda construction, and an extra list such as
>   =org-search-files= for files not containing any dates?  Then, some time
>   would have to be taken when =C-c a s=, but not before. And not if you don't
>   search for anything in your Org files during that Emacs session...
>

This already exists. See org-agenda-text-search-extra-files. 

Best,
Matt