Re: Re: Having (too) many files in org-agenda-files

From: Carsten Dominik <carsten.dominik@gmail.com>
To: "Sébastien Vauban" <wxhgmqzgwmuf@spammotel.com>
Cc: emacs-orgmode@gnu.org
Subject: Re: Re: Having (too) many files in org-agenda-files
Date: Thu, 30 Sep 2010 13:24:16 +0200	[thread overview]
Message-ID: <493C67F4-E312-43F3-B5D2-62F740EADD72@gmail.com> (raw)
In-Reply-To: <87fwwsfazb.fsf@mundaneum.com>

On Sep 30, 2010, at 12:20 AM, Sébastien Vauban wrote:

> Hi Carsten,
>
> Carsten Dominik wrote:
>> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>>> Carsten Dominik wrote:
>>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>>> Of course, I have many, many files in Org mode. All files I  
>>>>> write (or
>>>>> touch) in fact.
>>>>>
>>>>> Of course, I would like to search through my files at some point  
>>>>> in time.
>>>>> The problem is the load-time of my Emacs, now 221 seconds,  
>>>>> coming from 20
>>>>> seconds before the heavy use of Org...
>>>>
>>>> 4 minutes of startup time is entirely unacceptable. And I think  
>>>> you need
>>>> to identify what is causing this.
>>>>
>>>> fontification at display time is standard, I believe.
>>>
>>> It seems not, from what I see in the Messages buffer. I really  
>>> don't have
>>> the impression of having fiddled with that, really.
>
>>>> You must be doing something strange [...] maybe forcing global
>>>> fontification for each file or so.
>>>
>>> To repeat myself, no to that question. But ispell and flyspell are  
>>> called
>>> for Org files. Maybe there are interactions?
>>>
>>> [Using org-agenda-text-search-extra-files] reduces my load time  
>>> from 221
>>> seconds down to 92 seconds. Already a huge diff!
>>
>> Still bad though. I am wondering what is causing the fontification  
>> message.
>> I do not get this, so it must be something in your setup. You  
>> should try to
>> find out when this is happening and why.
>
> You've seen, from my reply to Matt, why my *Messages* buffer is more  
> verbose
> than yours. It's on purpose, just to be able to figure out more  
> easily what's
> going on. Though, I don't have yet...
>
>
>> Also, you might consider to remove (org-agenda-list) from .emacs. I  
>> think it
>> is pretty much always a bad idea to put a command like this into your
>> startup. Just make it a habit to call it early after starting Emacs.
>
> Honestly, that would not change that much. Having to wait 92 seconds  
> at
> startup, or a long time just a few minutes later has the same impact  
> for me.
> And: don't try to make me stop using Org ;-))

:-)

The reason why I am suggesting to remove this command is more that it  
make
emacs more usable for command line usage etc.

Furthermore, you are adding quite a bit to font-lock, but I don't see  
why it should be so slow.

My Emacs starts up in 4 seconds at most.  When I create an agenda, I  
am talking 1 second to maybe 2.
I have about 10 files contributing to the agenda.

- Carsten

>
>
>> It also seems to me you some of the extra packages might be activated
>> several times in each file. One possible reason could be that you  
>> have put
>> the code to turn them on into several hooks like text-mode-hook,
>> outline-mode-hook, and org-mode-hook. Turning on org-mode will  
>> first run
>> text-mode-hook, then outline-mode-hook, then org-mode-hook .....  
>> Maybe you
>> are also calling font-lock-fontify-buffer explicitly in one of your  
>> hooks?
>
> The only fontification customs that I do are here:
>
> --8<---------------cut here---------------start------------->8---
> ;; special words
> (setq keywords-critical-pattern
>      "\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\| 
> [Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff] 
> [Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
> (make-face 'keywords-critical)
> (GNUEmacs (set-face-attribute 'keywords-critical nil :foreground  
> "red" :background "yellow" :weight 'bold))
>
> (setq keywords-org-critical-pattern
>      "\\(BUGS\\|FIXME\\|XXX\\|[^*] TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm] 
> [Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa] 
> [Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>                                  ; smaller subset of keywords for  
> ensuring no conflict with Org mode TODO keywords
>
> ;; FIXME Highlighting all special keywords but "TODO" in Org mode is  
> already a
> ;; good step. Though, a nicer integration would be that "TODO"  
> strings in the
> ;; headings are not touched by this code, and that only "TODO"  
> strings in the
> ;; text body would be. Don't know (yet) how to do that...
> (make-face 'keywords-org-critical)
> (GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground  
> "red" :background "yellow" :weight 'bold))
>
> (setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
> (make-face 'keywords-normal)
> (GNUEmacs (set-face-attribute 'keywords-normal nil :foreground  
> "magenta2" :background "yellow"))
>
> ;; set up highlighting of special words for proper selected major  
> modes only
> (dolist (mode '(fundamental-mode
>                svn-log-view-mode
>                text-mode))  ; no interference with Org mode (which  
> derives from text-mode)
>  (font-lock-add-keywords mode
>                          `((,keywords-critical-pattern 1 'keywords- 
> critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; set up highlighting of special words for Org mode only
> (dolist (mode '(org-mode))
>  (font-lock-add-keywords mode
>                          `((,keywords-org-critical-pattern 1  
> 'keywords-org-critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; add fontification patterns (even in comments) to a selected major  
> mode
> ;; *and* all major modes derived from it
> (defun fontify-keywords ()
>  (interactive)
>  (font-lock-add-keywords nil
>                          `((,keywords-critical-pattern 1 'keywords- 
> critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; set up highlighting of special words for selected major modes  
> *and* all
> ;; major modes derived from them
> (dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl- 
> mode-hook css-mode-hook
>                emacs-lisp-mode-hook
>                html-mode-hook
>                java-mode-hook
>                latex-mode-hook lisp-mode-hook
>                makefile-mode-hook message-mode-hook
>                php-mode-hook python-mode-hook
>                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
>  (add-hook hook 'fontify-keywords))
> --8<---------------cut here---------------end--------------->8---
>
> The goal is to highlight words such as FIXME, WARNING, XXX, TODO in  
> as many
> files as possible.
>
> For Org, a small custom (of the custom) is done in order not to  
> catch all TODO
> words: the TODO words in headlines (i.e., preceded by `* ') are *not*
> highlighted.
>
> They are well highlighted when found in other portions of (normal)  
> text.
>
> Other custom:
>
> --8<---------------cut here---------------start------------->8---
> ;; highlight columns 78 to 80 in some modes
> (when (try-require 'column-marker)
>  (dolist (hook '(emacs-lisp-mode-hook
>                  cperl-mode-hook
>                  shell-mode-hook
>                  text-mode-hook
>                  change-log-mode-hook
>                  makefile-mode-hook
>                  message-mode-hook
>                  texinfo-mode-hook))
>    (add-hook hook (lambda ()
>                     (interactive)
>                     (column-marker-1 78)
>                     (column-marker-2 79)
>                     (column-marker-3 80))))
> --8<---------------cut here---------------end--------------->8---
>
> for getting markers in columns 78 to 80, showing me I'm good for a  
> `M-q'...
>
> I've done multiple tests, now:
>
> - with none of the 2 above sets of customs
> - with the 1st custom disabled, and the column-marker one enabled
> - with both enabled
>
> Times observed (in seconds):
>
> |      | no highlight | column marker only | all highlights |
> |------+--------------+--------------------+----------------|
> |      |           63 |                 82 |             91 |
> |      |           65 |                 84 |             96 |
> |      |           76 |                100 |            130 |
> |------+--------------+--------------------+----------------|
> | mean |           68 |                 89 |            106 |
>
> (note that times are really not that stable, while I always launched  
> Emacs in
> the same conditions, without doing anything else during that time!)
>
> So, the above 2 sets of customs have a real impact on the load time.
>
> But I'm still unsure how I have to change all of the above snippets  
> of code,
> in order to keep the same functionality, but without the time delays  
> when
> Emacs starts up...
>
>
>> Hope some of this helps. If not, let me see you full configuration  
>> - maybe I
>> can spot something else.
>
> My .emacs file is there:
>
> http://www.mygooglest.com/sva/emacs-init-sva.el
>
> Though it's well documented, it's not a tiny one... But very well  
> structured,
> so quite easy to find stuff related to each other.
>
> I'm always interested by any comment...
>
> Best regards,
>  Seb
>
> -- 
> Sébastien Vauban
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten