Having (too) many files in org-agenda-files

Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi,

Of course, I have many, many files in Org mode. All files I write (or touch)
in fact.

Of course, I would like to search through my files at some point in time.
I even would like to search through your files at some point in time, I mean
through =org-mode/contrib/babel= and =Worg= for example.

Hence, I put them in =org-agenda-files=:

--8<---------------cut here---------------start------------->8---
;; set which files to search for TODO entries and scheduled items
;; (avoiding hidden files)
(setq org-agenda-files
      (append (directory-files org-directory t "^[^\\.].*\\.org$")
              (if (file-exists-p "~/Projects/")
                  (directory-files "~/Projects/" t "^[^\\.].*\\.org$")
                nil)
              (if (file-exists-p "~/Public/")
                  (directory-files "~/Public/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Public/www.mygooglest.com/source/sva/")
                  (directory-files "~/Public/www.mygooglest.com/source/sva/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Examples/Org-scraps/")
                  (directory-files "~/Examples/Org-scraps/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Downloads/emacs/site-lisp/org-mode/contrib/babel/")
                  (directory-files "~/Downloads/emacs/site-lisp/org-mode/contrib/babel/" t "^[^\\.].*\\.org$")
                nil)
              (if (file-exists-p "~/Downloads/emacs/site-lisp/Worg/")
                  (directory-files "~/Downloads/emacs/site-lisp/Worg/" t "^[^\\.].*\\.org$")
                nil)
              ))
--8<---------------cut here---------------end--------------->8---

The problem is the load-time of my Emacs, now:

--8<---------------cut here---------------start------------->8---
Emacs Init startup time: 221 seconds.
--8<---------------cut here---------------end--------------->8---

coming from 20 seconds before the heavy use of Org...

with tens of times such lines in my *Messages* buffer:

--8<---------------cut here---------------start------------->8---
OVERVIEW
Checking for library `filladapt'...
+-> Requiring `filladapt' (already loaded)
Checking for library `filladapt'... Found
Fontifying Axa.org... (regexps............................................)
Checking for library `filladapt'... Found
Fontifying Axa.org... (regexps.............................................)
Checking for library `filladapt'... Found
Fontifying Axa.org... (regexps..............................................)
Checking for library `filladapt'... Found
+-> Requiring `outline-mode-easy-bindings' (already loaded)
+-> Requiring `ispell' (already loaded)
Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
Starting new Ispell process [en_US] ...
Checking for library `filladapt'...
+-> Requiring `filladapt' (already loaded)
Checking for library `filladapt'... Found
--8<---------------cut here---------------end--------------->8---

You'll tell me: not a problem, you do that only once a day, and you use Emacs
client/server for the rest of the time. True. A bit, because I sometimes have
to restart Emacs for testing a fresh one (not impacted by defvars already
defined, or deffaces, etc.).

Having to wait almost 4 minutes is a real pain. So, here my
comments/questions:

- Isn't it possible to delay the fontification/ispell/etc. to when we really
  display (i.e., pop up) the buffer?  I guess this must be a major component
  of the time this takes.

- Couldn't we have 2 vars: =org-agenda-files= for the files you know you want
  have scanned for the agenda construction, and an extra list such as
  =org-search-files= for files not containing any dates?  Then, some time
  would have to be taken when =C-c a s=, but not before. And not if you don't
  search for anything in your Org files during that Emacs session...

Any other idea?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Having (too) many files in org-agenda-files

[Matt Lundin]

Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org>
writes:

> Of course, I have many, many files in Org mode. All files I write (or
> touch) in fact.
>
> Of course, I would like to search through my files at some point in time.
> I even would like to search through your files at some point in time, I mean
> through =org-mode/contrib/babel= and =Worg= for example.
>

The drawback of org-mode's plain text format is that org-mode needs to
load, parse, and fontify, and preserve in memory all files it queries
(i.e., agenda files). Too many agenda files can thus cause a pretty big
performance hit (both when loading agenda files and constructing agenda
views). To avoid this, I would recommend including only important files
in org-agenda-files. Possible solutions for searching other files
include:

1. loading the other directories when needed (e.g., through a function
   that changes the value of org-agenda-files)
2. using the variable org-agenda-text-search-extra-files (see below)
3. searching other directories with external tools (grep, perl script,
   etc.)

> The problem is the load-time of my Emacs, now:
>
> Emacs Init startup time: 221 seconds.

Do you invoke an agenda command in your emacs, such as
org-agenda-to-appt? That would cause org-mode to load all org files.

> coming from 20 seconds before the heavy use of Org...
>
> with tens of times such lines in my *Messages* buffer:
>
> OVERVIEW
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps.............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps..............................................)
> Checking for library `filladapt'... Found
> +-> Requiring `outline-mode-easy-bindings' (already loaded)
> +-> Requiring `ispell' (already loaded)
> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
> Starting new Ispell process [en_US] ...
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found

It seems that filladapt is causing some of the verbosity (and possible
slowness) there. When loading agenda files, my *Messages* buffer only
shows:

OVERVIEW [29 times]

> You'll tell me: not a problem, you do that only once a day, and you
> use Emacs client/server for the rest of the time. True. A bit, because
> I sometimes have to restart Emacs for testing a fresh one (not
> impacted by defvars already defined, or deffaces, etc.).
>
> Having to wait almost 4 minutes is a real pain. So, here my
> comments/questions:
>
> - Isn't it possible to delay the fontification/ispell/etc. to when we really
>   display (i.e., pop up) the buffer?  I guess this must be a major component
>   of the time this takes.

I imagine the fontification is necessary for the agenda to function
properly. Thus, org files are parsed en masse when the agenda is called
for the first time. (Hence my question about whether you were calling
the agenda from ~/.emacs.)

>
> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you want
>   have scanned for the agenda construction, and an extra list such as
>   =org-search-files= for files not containing any dates?  Then, some time
>   would have to be taken when =C-c a s=, but not before. And not if you don't
>   search for anything in your Org files during that Emacs session...
>

This already exists. See org-agenda-text-search-extra-files. 

Best,
Matt

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Matt,

Matt Lundin wrote:
> Sébastien Vauban writes:

First thing: no prob' for my name. The reply is important, my name much
less... isn't it, Matt (or Mattew? ;-))


>> Of course, I have many, many files in Org mode. All files I write (or
>> touch) in fact.
>>
>> Of course, I would like to search through my files at some point in time. I
>> even would like to search through your files at some point in time, I mean
>> through =org-mode/contrib/babel= and =Worg= for example.
>
> The drawback of org-mode's plain text format is that org-mode needs to load,
> parse, and fontify, and preserve in memory all files it queries (i.e.,
> agenda files). Too many agenda files can thus cause a pretty big performance
> hit (both when loading agenda files and constructing agenda views). To avoid
> this, I would recommend including only important files in org-agenda-files.
> Possible solutions for searching other files include:
>
> 1. loading the other directories when needed (e.g., through a function that
>    changes the value of org-agenda-files)

Good idea.


> 2. using the variable org-agenda-text-search-extra-files (see below)

In fact, more or less the same as the above... and already foreseen. Nice.


> 3. searching other directories with external tools (grep, perl script, etc.)

Not really interested by this. I love some much the powerful and very quick
search capabilities offered by Org.


>> The problem is the load-time of my Emacs, now:
>>
>> Emacs Init startup time: 221 seconds.
>
> Do you invoke an agenda command in your emacs, such as org-agenda-to-appt?
> That would cause org-mode to load all org files.

As seen in my reply to Carsten today, yes, I have an org-agenda-list. I know
that it is looking in every pointed file. The problem is that it really takes
much time... fontifying and so.


>> coming from 20 seconds before the heavy use of Org...
>>
>> with tens of times such lines in my *Messages* buffer:
>>
>> OVERVIEW
>> Checking for library `filladapt'...
>> +-> Requiring `filladapt' (already loaded)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps............................................)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps.............................................)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps..............................................)
>> Checking for library `filladapt'... Found
>> +-> Requiring `outline-mode-easy-bindings' (already loaded)
>> +-> Requiring `ispell' (already loaded)
>> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
>> Starting new Ispell process [en_US] ...
>> Checking for library `filladapt'...
>> +-> Requiring `filladapt' (already loaded)
>> Checking for library `filladapt'... Found
>
> It seems that filladapt is causing some of the verbosity (and possible
> slowness) there. When loading agenda files, my *Messages* buffer only shows:
>
> OVERVIEW [29 times]

OK, I have to tell that, in order to debug some problems, I've changed some
basic functions in order to get more verbosity in the Messages buffer. Here
they are:

--8<---------------cut here---------------start------------->8---
      ;; REPLACES ORIGINAL in `C source code' (dumped)
      ;; redefine require to leave a trace of packages being loaded
      (if (not (fboundp 'orig-require))
          (fset 'orig-require (symbol-function 'require))
        (message "The code to redefine `require' should not be loaded twice"))

      (defvar my-require-depth 0)

      (defun require (feature &optional filename noerror)
        "Leave a trace of packages being loaded."
        (cond ((member feature features)
               (message "%sRequiring `%s' (already loaded)"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature))
              (t
               (message "%sRequiring `%s'"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature)
               (let ((my-require-depth (+ 1 my-require-depth)))
                 (orig-require feature filename noerror))
               (message "%sRequiring `%s'...done"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature))))

      (defvar missing-packages-list nil
        "List of packages that `try-require' can't find.")

      ;; attempt to load a feature/library, failing silently
      (defun try-require (feature)
        "Attempt to load a library or module. Return true if the
library given as argument is successfully loaded. If not, instead
of an error, just add the package to a list of missing packages."
        (condition-case err
            ;; protected form
            (progn
              (message "Checking for library `%s'..." feature)
              (if (stringp feature)
                  (load-library feature)
                (require feature))
              (message "Checking for library `%s'... Found" feature))
          ;; error handler
          (file-error  ; condition
           (progn
             (message "Checking for library `%s'... Missing" feature)
             (add-to-list 'missing-packages-list feature 'append))
           nil)))
--8<---------------cut here---------------end--------------->8---


>> You'll tell me: not a problem, you do that only once a day, and you use
>> Emacs client/server for the rest of the time. True. A bit, because I
>> sometimes have to restart Emacs for testing a fresh one (not impacted by
>> defvars already defined, or deffaces, etc.).
>>
>> Having to wait almost 4 minutes is a real pain. So, here my
>> comments/questions:
>>
>> - Isn't it possible to delay the fontification/ispell/etc. to when we
>>   really display (i.e., pop up) the buffer? I guess this must be a major
>>   component of the time this takes.
>
> I imagine the fontification is necessary for the agenda to function
> properly.

This assumption should be checked. Does parsing really mean fontifying?  From
what I understood about jit-lock-mode, I'd say no... but...


> Thus, org files are parsed en masse when the agenda is called for the first
> time. (Hence my question about whether you were calling the agenda from
> ~/.emacs.)
>
>> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you
>>   want have scanned for the agenda construction, and an extra list such as
>>   =org-search-files= for files not containing any dates? Then, some time
>>   would have to be taken when =C-c a s=, but not before. And not if you
>>   don't search for anything in your Org files during that Emacs session...
>
> This already exists. See org-agenda-text-search-extra-files.

Thanks for this, did not know about that (up to today)...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Having (too) many files in org-agenda-files

[Shelagh Manton]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com>
writes:

> Hi,
>
> Of course, I have many, many files in Org mode. All files I write (or touch)
> in fact.
>
> Of course, I would like to search through my files at some point in time.
> I even would like to search through your files at some point in time, I mean
> through =org-mode/contrib/babel= and =Worg= for example.
[snip]
>
> You'll tell me: not a problem, you do that only once a day, and you use Emacs
> client/server for the rest of the time. True. A bit, because I sometimes have
> to restart Emacs for testing a fresh one (not impacted by defvars already
> defined, or deffaces, etc.).
>
> Having to wait almost 4 minutes is a real pain. So, here my
> comments/questions:
>
> - Isn't it possible to delay the fontification/ispell/etc. to when we really
>   display (i.e., pop up) the buffer?  I guess this must be a major component
>   of the time this takes.
>
> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you want
>   have scanned for the agenda construction, and an extra list such as
>   =org-search-files= for files not containing any dates?  Then, some time
>   would have to be taken when =C-c a s=, but not before. And not if you don't
>   search for anything in your Org files during that Emacs session...
>
> Any other idea?

I use the jit-lock package that comes with emacs. "Stealth fontifying"

It makes font-lock only happen after the buffer is loaded and when you
are not doing anything else. You can access the variables through
customize-group RET jit-lock. It may be helpful here.

Shelagh
>
> Best regards,
>   Seb

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Shelagh,

Shelagh Manton wrote:
> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
> I use the jit-lock package that comes with emacs. "Stealth fontifying"
>
> It makes font-lock only happen after the buffer is loaded and when you are
> not doing anything else.

I just checked. The following is default:

--8<---------------cut here---------------start------------->8---
    (setq font-lock-support-mode 'jit-lock-mode)
--8<---------------cut here---------------end--------------->8---

hence, I also have it.


> You can access the variables through customize-group RET jit-lock. It may be
> helpful here.

As you can see, I did not change anything:

--8<---------------cut here---------------start------------->8---
Editing a setting changes only the text in this buffer.
To apply your changes, use the Save or Set buttons.
Saving a change normally works by editing your init file.
For details, see Saving Customizations in the Emacs manual.


 Operate on all settings in this buffer that are not marked HIDDEN:
 Set for current session Save for future sessions
 Undo edits Reset to saved Erase customizations   Exit

Parent groups: Font Lock
Parent group documentation: Elisp Manual and Emacs Manual.


/- Jit Lock group: Font Lock support mode to fontify just-in-time.-------\
      State: visible group members are all at standard values.

Jit Lock Chunk Size: Hide Value 500
   State: STANDARD.
   Jit-lock fontifies chunks of at most this many characters at a time. More

Jit Lock Stealth Time: Hide Value Value Menu never
   State: STANDARD.
   Time in seconds to wait before beginning stealth fontification. More

Jit Lock Stealth Nice: Hide Value Value Menu seconds: 0.5
   State: STANDARD.
   Time in seconds to pause between chunks of stealth fontification. More

Jit Lock Stealth Load: Hide Value Value Menu load: 200
   State: STANDARD.
   Load in percentage above which stealth fontification is suspended. More

Jit Lock Stealth Verbose: Hide Value Toggle  off (nil)
   State: STANDARD.
   If non-nil, means stealth fontification should show status messages.

Jit Lock Contextually: Hide Value Value Menu syntax-driven
   State: STANDARD.
   If non-nil, means fontification should be syntactically true. More

Jit Lock Context Time: Hide Value 0.5
   State: STANDARD.
   Idle time after which text is contextually refontified, if applicable.

Jit Lock Defer Time: Hide Value Value Menu never
   State: STANDARD.
   Idle time after which deferred fontification should take place. More

\- Jit Lock group end --------------------------------------------------/
--8<---------------cut here---------------end--------------->8---

Though, I find some values quite weird:

- jit-lock-stealth-time = nil (never)
- jit-lock-stealth-load = 200 (%!?)
- jit-lock-defer-time = nil (never)

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Having (too) many files in org-agenda-files

[Carsten Dominik]

Hi Sebastian,

4 minutes of startup time is entirely unacceptable.  And I think
you need to identify what is causing this.

First of all, unless you are having to totally old Version of Emacs,
fontification at display time is standard, I believe.

You must be doing something strange, like forcing Emacs to load
all those files in org-agenda-files for example, and maybe forcing  
global
fontification for each file or so.

About a separate variable for search file, that does of cause exist:
org-agenda-text-search-extra-files

- Carsten


On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:

> Hi,
>
> Of course, I have many, many files in Org mode. All files I write  
> (or touch)
> in fact.
>
> Of course, I would like to search through my files at some point in  
> time.
> I even would like to search through your files at some point in  
> time, I mean
> through =org-mode/contrib/babel= and =Worg= for example.
>
> Hence, I put them in =org-agenda-files=:
>
> --8<---------------cut here---------------start------------->8---
> ;; set which files to search for TODO entries and scheduled items
> ;; (avoiding hidden files)
> (setq org-agenda-files
>      (append (directory-files org-directory t "^[^\\.].*\\.org$")
>              (if (file-exists-p "~/Projects/")
>                  (directory-files "~/Projects/" t "^[^\\.].*\\.org$")
>                nil)
>              (if (file-exists-p "~/Public/")
>                  (directory-files "~/Public/" t "^[^\\.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Public/www.mygooglest.com/source/sva/ 
> ")
>                  (directory-files "~/Public/www.mygooglest.com/source/sva/ 
> " t "^[^\\.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Examples/Org-scraps/")
>                  (directory-files "~/Examples/Org-scraps/" t "^[^\ 
> \.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Downloads/emacs/site-lisp/org- 
> mode/contrib/babel/")
>                  (directory-files "~/Downloads/emacs/site-lisp/org- 
> mode/contrib/babel/" t "^[^\\.].*\\.org$")
>                nil)
>              (if (file-exists-p "~/Downloads/emacs/site-lisp/Worg/")
>                  (directory-files "~/Downloads/emacs/site-lisp/ 
> Worg/" t "^[^\\.].*\\.org$")
>                nil)
>              ))
> --8<---------------cut here---------------end--------------->8---
>
> The problem is the load-time of my Emacs, now:
>
> --8<---------------cut here---------------start------------->8---
> Emacs Init startup time: 221 seconds.
> --8<---------------cut here---------------end--------------->8---
>
> coming from 20 seconds before the heavy use of Org...
>
> with tens of times such lines in my *Messages* buffer:
>
> --8<---------------cut here---------------start------------->8---
> OVERVIEW
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found
> Fontifying Axa.org...  
> (regexps............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org...  
> (regexps.............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org...  
> (regexps..............................................)
> Checking for library `filladapt'... Found
> +-> Requiring `outline-mode-easy-bindings' (already loaded)
> +-> Requiring `ispell' (already loaded)
> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
> Starting new Ispell process [en_US] ...
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found
> --8<---------------cut here---------------end--------------->8---
>
> You'll tell me: not a problem, you do that only once a day, and you  
> use Emacs
> client/server for the rest of the time. True. A bit, because I  
> sometimes have
> to restart Emacs for testing a fresh one (not impacted by defvars  
> already
> defined, or deffaces, etc.).
>
> Having to wait almost 4 minutes is a real pain. So, here my
> comments/questions:
>
> - Isn't it possible to delay the fontification/ispell/etc. to when  
> we really
>  display (i.e., pop up) the buffer?  I guess this must be a major  
> component
>  of the time this takes.
>
> - Couldn't we have 2 vars: =org-agenda-files= for the files you know  
> you want
>  have scanned for the agenda construction, and an extra list such as
>  =org-search-files= for files not containing any dates?  Then, some  
> time
>  would have to be taken when =C-c a s=, but not before. And not if  
> you don't
>  search for anything in your Org files during that Emacs session...
>
> Any other idea?
>
> Best regards,
>  Seb
>
> -- 
> Sébastien Vauban
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>> Of course, I have many, many files in Org mode. All files I write (or touch)
>> in fact.
>>
>> Of course, I would like to search through my files at some point in time. I
>> even would like to search through your files at some point in time, I mean
>> through =org-mode/contrib/babel= and =Worg= for example. Hence, I put them
>> in =org-agenda-files=.
>>
>> The problem is the load-time of my Emacs, now 221 seconds, coming from 20
>> seconds before the heavy use of Org...
>>
>> You'll tell me: not a problem, you do that only once a day, and you use
>> Emacs client/server for the rest of the time. True. A bit, because I
>> sometimes have to restart Emacs for testing a fresh one (not impacted by
>> defvars already defined, or deffaces, etc.).
>>
>> Having to wait almost 4 minutes is a real pain. So, here my
>> comments/questions:
>>
>> - Isn't it possible to delay the fontification/ispell/etc. to when we
>>   really display (i.e., pop up) the buffer? I guess this must be a major
>>   component of the time this takes.
>>
>> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you
>>   want have scanned for the agenda construction, and an extra list such as
>>   =org-search-files= for files not containing any dates? Then, some time
>>   would have to be taken when =C-c a s=, but not before. And not if you
>>   don't search for anything in your Org files during that Emacs session...
>
> 4 minutes of startup time is entirely unacceptable.  And I think
> you need to identify what is causing this.
>
> First of all, unless you are having to totally old Version of Emacs,

Nope. GNU Emacs 23.1.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0)
of 2010-03-29 on rothera, modified by Debian


> fontification at display time is standard, I believe.

It seems not, from what I see in the Messages buffer. I really don't have the
impression of having fiddled with that, really.


> You must be doing something strange, like forcing Emacs to load
> all those files in org-agenda-files for example,

That, yes!  I should have explicitly written it, but -- yes -- I do have:

--8<---------------cut here---------------start------------->8---
(org-agenda-list)
--8<---------------cut here---------------end--------------->8---

in my .emacs.


> and maybe forcing global fontification for each file or so.

To repeat myself, no to that question. But ispell and flyspell are called for
Org files. Maybe there are interactions?


> About a separate variable for search file, that does of cause exist:
> org-agenda-text-search-extra-files

Of course!

Doing this:

--8<---------------cut here---------------start------------->8---
(setq org-agenda-files
      (append (directory-files org-directory t "^[^\\.].*\\.org$")
              (if (file-exists-p "~/Projects/")
                  (directory-files "~/Projects/" t "^[^\\.].*\\.org$")
                nil)))

(setq org-agenda-text-search-extra-files
      (append
              (if (file-exists-p "~/Public/")
                  (directory-files "~/Public/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Public/www.mygooglest.com/source/sva/")
                  (directory-files "~/Public/www.mygooglest.com/source/sva/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Examples/Org-scraps/")
                  (directory-files "~/Examples/Org-scraps/" t "^[^\\.].*\\.txt$")
                nil)
              (if (file-exists-p "~/Downloads/emacs/site-lisp/org-mode/contrib/babel/")
                  (directory-files "~/Downloads/emacs/site-lisp/org-mode/contrib/babel/" t "^[^\\.].*\\.org$")
                nil)
              (if (file-exists-p "~/Downloads/emacs/site-lisp/Worg/")
                  (directory-files "~/Downloads/emacs/site-lisp/Worg/" t "^[^\\.].*\\.org$")
                nil)
              ))
--8<---------------cut here---------------end--------------->8---

reduces my load time from 221 seconds down to 92 seconds. Already a huge diff!

Thanks...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Carsten Dominik]

On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:

> Hi Carsten,
>
> Carsten Dominik wrote:
>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>> Of course, I have many, many files in Org mode. All files I write  
>>> (or touch)
>>> in fact.
>>>
>>> Of course, I would like to search through my files at some point  
>>> in time. I
>>> even would like to search through your files at some point in  
>>> time, I mean
>>> through =org-mode/contrib/babel= and =Worg= for example. Hence, I  
>>> put them
>>> in =org-agenda-files=.
>>>
>>> The problem is the load-time of my Emacs, now 221 seconds, coming  
>>> from 20
>>> seconds before the heavy use of Org...
>>>
>>> You'll tell me: not a problem, you do that only once a day, and  
>>> you use
>>> Emacs client/server for the rest of the time. True. A bit, because I
>>> sometimes have to restart Emacs for testing a fresh one (not  
>>> impacted by
>>> defvars already defined, or deffaces, etc.).
>>>
>>> Having to wait almost 4 minutes is a real pain. So, here my
>>> comments/questions:
>>>
>>> - Isn't it possible to delay the fontification/ispell/etc. to when  
>>> we
>>>  really display (i.e., pop up) the buffer? I guess this must be a  
>>> major
>>>  component of the time this takes.
>>>
>>> - Couldn't we have 2 vars: =org-agenda-files= for the files you  
>>> know you
>>>  want have scanned for the agenda construction, and an extra list  
>>> such as
>>>  =org-search-files= for files not containing any dates? Then, some  
>>> time
>>>  would have to be taken when =C-c a s=, but not before. And not if  
>>> you
>>>  don't search for anything in your Org files during that Emacs  
>>> session...
>>
>> 4 minutes of startup time is entirely unacceptable.  And I think
>> you need to identify what is causing this.
>>
>> First of all, unless you are having to totally old Version of Emacs,
>
> Nope. GNU Emacs 23.1.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0)
> of 2010-03-29 on rothera, modified by Debian
>
>
>> fontification at display time is standard, I believe.
>
> It seems not, from what I see in the Messages buffer. I really don't  
> have the
> impression of having fiddled with that, really.
>
>
>> You must be doing something strange, like forcing Emacs to load
>> all those files in org-agenda-files for example,
>
> That, yes!  I should have explicitly written it, but -- yes -- I do  
> have:
>
> --8<---------------cut here---------------start------------->8---
> (org-agenda-list)
> --8<---------------cut here---------------end--------------->8---
>
> in my .emacs.
>
>
>> and maybe forcing global fontification for each file or so.
>
> To repeat myself, no to that question. But ispell and flyspell are  
> called for
> Org files. Maybe there are interactions?
>
>
>> About a separate variable for search file, that does of cause exist:
>> org-agenda-text-search-extra-files
>
> Of course!
>
> Doing this:
>
> --8<---------------cut here---------------start------------->8---
> (setq org-agenda-files
>      (append (directory-files org-directory t "^[^\\.].*\\.org$")
>              (if (file-exists-p "~/Projects/")
>                  (directory-files "~/Projects/" t "^[^\\.].*\\.org$")
>                nil)))
>
> (setq org-agenda-text-search-extra-files
>      (append
>              (if (file-exists-p "~/Public/")
>                  (directory-files "~/Public/" t "^[^\\.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Public/www.mygooglest.com/source/sva/ 
> ")
>                  (directory-files "~/Public/www.mygooglest.com/source/sva/ 
> " t "^[^\\.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Examples/Org-scraps/")
>                  (directory-files "~/Examples/Org-scraps/" t "^[^\ 
> \.].*\\.txt$")
>                nil)
>              (if (file-exists-p "~/Downloads/emacs/site-lisp/org- 
> mode/contrib/babel/")
>                  (directory-files "~/Downloads/emacs/site-lisp/org- 
> mode/contrib/babel/" t "^[^\\.].*\\.org$")
>                nil)
>              (if (file-exists-p "~/Downloads/emacs/site-lisp/Worg/")
>                  (directory-files "~/Downloads/emacs/site-lisp/ 
> Worg/" t "^[^\\.].*\\.org$")
>                nil)
>              ))
> --8<---------------cut here---------------end--------------->8---
>
> reduces my load time from 221 seconds down to 92 seconds. Already a  
> huge diff!

Still bad though.  I am wondering what is causing the fontification  
message.
I do not get this, so it must be something in your setup.  You should
try to find out when this is happening and why.

Also, you might consider to remove (org-agenda-list) from .emacs.
I think it is pretty much always a bad idea to put a command like this  
into
your startup.  Just make it a habit to call it early after starting  
Emacs.

It also seems to me you some of the extra packages might be
activated several times in each file.  One possible reason could
be that you have put the code to turn them on into several hooks
like text-mode-hook, outline-mode-hook, and org-mode-hook.
Turning on org-mode will first run text-mode-hook, then
outline-mode-hook, then org-mode-hook .....
Maybe you are also calling font-lock-fontify-buffer explicitly
in one of your hooks?

Hope some of this helps.  If not, let me see you full
configuration - maybe I can spot something else.

- Carsten

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>> Carsten Dominik wrote:
>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>> Of course, I have many, many files in Org mode. All files I write (or
>>>> touch) in fact.
>>>>
>>>> Of course, I would like to search through my files at some point in time.
>>>> The problem is the load-time of my Emacs, now 221 seconds, coming from 20
>>>> seconds before the heavy use of Org...
>>>
>>> 4 minutes of startup time is entirely unacceptable. And I think you need
>>> to identify what is causing this.
>>>
>>> fontification at display time is standard, I believe.
>>
>> It seems not, from what I see in the Messages buffer. I really don't have
>> the impression of having fiddled with that, really.

>>> You must be doing something strange [...] maybe forcing global
>>> fontification for each file or so.
>>
>> To repeat myself, no to that question. But ispell and flyspell are called
>> for Org files. Maybe there are interactions?
>>
>> [Using org-agenda-text-search-extra-files] reduces my load time from 221
>> seconds down to 92 seconds. Already a huge diff!
>
> Still bad though. I am wondering what is causing the fontification message.
> I do not get this, so it must be something in your setup. You should try to
> find out when this is happening and why.

You've seen, from my reply to Matt, why my *Messages* buffer is more verbose
than yours. It's on purpose, just to be able to figure out more easily what's
going on. Though, I don't have yet...


> Also, you might consider to remove (org-agenda-list) from .emacs. I think it
> is pretty much always a bad idea to put a command like this into your
> startup. Just make it a habit to call it early after starting Emacs.

Honestly, that would not change that much. Having to wait 92 seconds at
startup, or a long time just a few minutes later has the same impact for me.
And: don't try to make me stop using Org ;-))


> It also seems to me you some of the extra packages might be activated
> several times in each file. One possible reason could be that you have put
> the code to turn them on into several hooks like text-mode-hook,
> outline-mode-hook, and org-mode-hook. Turning on org-mode will first run
> text-mode-hook, then outline-mode-hook, then org-mode-hook ..... Maybe you
> are also calling font-lock-fontify-buffer explicitly in one of your hooks?

The only fontification customs that I do are here:

--8<---------------cut here---------------start------------->8---
;; special words
(setq keywords-critical-pattern
      "\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
(make-face 'keywords-critical)
(GNUEmacs (set-face-attribute 'keywords-critical nil :foreground "red" :background "yellow" :weight 'bold))

(setq keywords-org-critical-pattern
      "\\(BUGS\\|FIXME\\|XXX\\|[^*] TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
                                  ; smaller subset of keywords for ensuring no conflict with Org mode TODO keywords

;; FIXME Highlighting all special keywords but "TODO" in Org mode is already a
;; good step. Though, a nicer integration would be that "TODO" strings in the
;; headings are not touched by this code, and that only "TODO" strings in the
;; text body would be. Don't know (yet) how to do that...
(make-face 'keywords-org-critical)
(GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground "red" :background "yellow" :weight 'bold))

(setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
(make-face 'keywords-normal)
(GNUEmacs (set-face-attribute 'keywords-normal nil :foreground "magenta2" :background "yellow"))

;; set up highlighting of special words for proper selected major modes only
(dolist (mode '(fundamental-mode
                svn-log-view-mode
                text-mode))  ; no interference with Org mode (which derives from text-mode)
  (font-lock-add-keywords mode
                          `((,keywords-critical-pattern 1 'keywords-critical prepend)
                            (,keywords-normal-pattern 1 'keywords-normal prepend))))

;; set up highlighting of special words for Org mode only
(dolist (mode '(org-mode))
  (font-lock-add-keywords mode
                          `((,keywords-org-critical-pattern 1 'keywords-org-critical prepend)
                            (,keywords-normal-pattern 1 'keywords-normal prepend))))

;; add fontification patterns (even in comments) to a selected major mode
;; *and* all major modes derived from it
(defun fontify-keywords ()
  (interactive)
  (font-lock-add-keywords nil
                          `((,keywords-critical-pattern 1 'keywords-critical prepend)
                            (,keywords-normal-pattern 1 'keywords-normal prepend))))

;; set up highlighting of special words for selected major modes *and* all
;; major modes derived from them
(dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl-mode-hook css-mode-hook
                emacs-lisp-mode-hook
                html-mode-hook
                java-mode-hook
                latex-mode-hook lisp-mode-hook
                makefile-mode-hook message-mode-hook
                php-mode-hook python-mode-hook
                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
  (add-hook hook 'fontify-keywords))
--8<---------------cut here---------------end--------------->8---

The goal is to highlight words such as FIXME, WARNING, XXX, TODO in as many
files as possible.

For Org, a small custom (of the custom) is done in order not to catch all TODO
words: the TODO words in headlines (i.e., preceded by `* ') are *not*
highlighted.

They are well highlighted when found in other portions of (normal) text.

Other custom:

--8<---------------cut here---------------start------------->8---
;; highlight columns 78 to 80 in some modes
(when (try-require 'column-marker)
  (dolist (hook '(emacs-lisp-mode-hook
                  cperl-mode-hook
                  shell-mode-hook
                  text-mode-hook
                  change-log-mode-hook
                  makefile-mode-hook
                  message-mode-hook
                  texinfo-mode-hook))
    (add-hook hook (lambda ()
                     (interactive)
                     (column-marker-1 78)
                     (column-marker-2 79)
                     (column-marker-3 80))))
--8<---------------cut here---------------end--------------->8---

for getting markers in columns 78 to 80, showing me I'm good for a `M-q'...

I've done multiple tests, now:

- with none of the 2 above sets of customs
- with the 1st custom disabled, and the column-marker one enabled
- with both enabled

Times observed (in seconds):

|      | no highlight | column marker only | all highlights |
|------+--------------+--------------------+----------------|
|      |           63 |                 82 |             91 |
|      |           65 |                 84 |             96 |
|      |           76 |                100 |            130 |
|------+--------------+--------------------+----------------|
| mean |           68 |                 89 |            106 |

(note that times are really not that stable, while I always launched Emacs in
the same conditions, without doing anything else during that time!)

So, the above 2 sets of customs have a real impact on the load time.

But I'm still unsure how I have to change all of the above snippets of code,
in order to keep the same functionality, but without the time delays when
Emacs starts up...


> Hope some of this helps. If not, let me see you full configuration - maybe I
> can spot something else.

My .emacs file is there:

http://www.mygooglest.com/sva/emacs-init-sva.el

Though it's well documented, it's not a tiny one... But very well structured,
so quite easy to find stuff related to each other.

I'm always interested by any comment...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Carsten Dominik]

On Sep 30, 2010, at 12:20 AM, Sébastien Vauban wrote:

> Hi Carsten,
>
> Carsten Dominik wrote:
>> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>>> Carsten Dominik wrote:
>>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>>> Of course, I have many, many files in Org mode. All files I  
>>>>> write (or
>>>>> touch) in fact.
>>>>>
>>>>> Of course, I would like to search through my files at some point  
>>>>> in time.
>>>>> The problem is the load-time of my Emacs, now 221 seconds,  
>>>>> coming from 20
>>>>> seconds before the heavy use of Org...
>>>>
>>>> 4 minutes of startup time is entirely unacceptable. And I think  
>>>> you need
>>>> to identify what is causing this.
>>>>
>>>> fontification at display time is standard, I believe.
>>>
>>> It seems not, from what I see in the Messages buffer. I really  
>>> don't have
>>> the impression of having fiddled with that, really.
>
>>>> You must be doing something strange [...] maybe forcing global
>>>> fontification for each file or so.
>>>
>>> To repeat myself, no to that question. But ispell and flyspell are  
>>> called
>>> for Org files. Maybe there are interactions?
>>>
>>> [Using org-agenda-text-search-extra-files] reduces my load time  
>>> from 221
>>> seconds down to 92 seconds. Already a huge diff!
>>
>> Still bad though. I am wondering what is causing the fontification  
>> message.
>> I do not get this, so it must be something in your setup. You  
>> should try to
>> find out when this is happening and why.
>
> You've seen, from my reply to Matt, why my *Messages* buffer is more  
> verbose
> than yours. It's on purpose, just to be able to figure out more  
> easily what's
> going on. Though, I don't have yet...
>
>
>> Also, you might consider to remove (org-agenda-list) from .emacs. I  
>> think it
>> is pretty much always a bad idea to put a command like this into your
>> startup. Just make it a habit to call it early after starting Emacs.
>
> Honestly, that would not change that much. Having to wait 92 seconds  
> at
> startup, or a long time just a few minutes later has the same impact  
> for me.
> And: don't try to make me stop using Org ;-))

:-)

The reason why I am suggesting to remove this command is more that it  
make
emacs more usable for command line usage etc.

Furthermore, you are adding quite a bit to font-lock, but I don't see  
why it should be so slow.

My Emacs starts up in 4 seconds at most.  When I create an agenda, I  
am talking 1 second to maybe 2.
I have about 10 files contributing to the agenda.

- Carsten


>
>
>> It also seems to me you some of the extra packages might be activated
>> several times in each file. One possible reason could be that you  
>> have put
>> the code to turn them on into several hooks like text-mode-hook,
>> outline-mode-hook, and org-mode-hook. Turning on org-mode will  
>> first run
>> text-mode-hook, then outline-mode-hook, then org-mode-hook .....  
>> Maybe you
>> are also calling font-lock-fontify-buffer explicitly in one of your  
>> hooks?
>
> The only fontification customs that I do are here:
>
> --8<---------------cut here---------------start------------->8---
> ;; special words
> (setq keywords-critical-pattern
>      "\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\| 
> [Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff] 
> [Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
> (make-face 'keywords-critical)
> (GNUEmacs (set-face-attribute 'keywords-critical nil :foreground  
> "red" :background "yellow" :weight 'bold))
>
> (setq keywords-org-critical-pattern
>      "\\(BUGS\\|FIXME\\|XXX\\|[^*] TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm] 
> [Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa] 
> [Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>                                  ; smaller subset of keywords for  
> ensuring no conflict with Org mode TODO keywords
>
> ;; FIXME Highlighting all special keywords but "TODO" in Org mode is  
> already a
> ;; good step. Though, a nicer integration would be that "TODO"  
> strings in the
> ;; headings are not touched by this code, and that only "TODO"  
> strings in the
> ;; text body would be. Don't know (yet) how to do that...
> (make-face 'keywords-org-critical)
> (GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground  
> "red" :background "yellow" :weight 'bold))
>
> (setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
> (make-face 'keywords-normal)
> (GNUEmacs (set-face-attribute 'keywords-normal nil :foreground  
> "magenta2" :background "yellow"))
>
> ;; set up highlighting of special words for proper selected major  
> modes only
> (dolist (mode '(fundamental-mode
>                svn-log-view-mode
>                text-mode))  ; no interference with Org mode (which  
> derives from text-mode)
>  (font-lock-add-keywords mode
>                          `((,keywords-critical-pattern 1 'keywords- 
> critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; set up highlighting of special words for Org mode only
> (dolist (mode '(org-mode))
>  (font-lock-add-keywords mode
>                          `((,keywords-org-critical-pattern 1  
> 'keywords-org-critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; add fontification patterns (even in comments) to a selected major  
> mode
> ;; *and* all major modes derived from it
> (defun fontify-keywords ()
>  (interactive)
>  (font-lock-add-keywords nil
>                          `((,keywords-critical-pattern 1 'keywords- 
> critical prepend)
>                            (,keywords-normal-pattern 1 'keywords- 
> normal prepend))))
>
> ;; set up highlighting of special words for selected major modes  
> *and* all
> ;; major modes derived from them
> (dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl- 
> mode-hook css-mode-hook
>                emacs-lisp-mode-hook
>                html-mode-hook
>                java-mode-hook
>                latex-mode-hook lisp-mode-hook
>                makefile-mode-hook message-mode-hook
>                php-mode-hook python-mode-hook
>                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
>  (add-hook hook 'fontify-keywords))
> --8<---------------cut here---------------end--------------->8---
>
> The goal is to highlight words such as FIXME, WARNING, XXX, TODO in  
> as many
> files as possible.
>
> For Org, a small custom (of the custom) is done in order not to  
> catch all TODO
> words: the TODO words in headlines (i.e., preceded by `* ') are *not*
> highlighted.
>
> They are well highlighted when found in other portions of (normal)  
> text.
>
> Other custom:
>
> --8<---------------cut here---------------start------------->8---
> ;; highlight columns 78 to 80 in some modes
> (when (try-require 'column-marker)
>  (dolist (hook '(emacs-lisp-mode-hook
>                  cperl-mode-hook
>                  shell-mode-hook
>                  text-mode-hook
>                  change-log-mode-hook
>                  makefile-mode-hook
>                  message-mode-hook
>                  texinfo-mode-hook))
>    (add-hook hook (lambda ()
>                     (interactive)
>                     (column-marker-1 78)
>                     (column-marker-2 79)
>                     (column-marker-3 80))))
> --8<---------------cut here---------------end--------------->8---
>
> for getting markers in columns 78 to 80, showing me I'm good for a  
> `M-q'...
>
> I've done multiple tests, now:
>
> - with none of the 2 above sets of customs
> - with the 1st custom disabled, and the column-marker one enabled
> - with both enabled
>
> Times observed (in seconds):
>
> |      | no highlight | column marker only | all highlights |
> |------+--------------+--------------------+----------------|
> |      |           63 |                 82 |             91 |
> |      |           65 |                 84 |             96 |
> |      |           76 |                100 |            130 |
> |------+--------------+--------------------+----------------|
> | mean |           68 |                 89 |            106 |
>
> (note that times are really not that stable, while I always launched  
> Emacs in
> the same conditions, without doing anything else during that time!)
>
> So, the above 2 sets of customs have a real impact on the load time.
>
> But I'm still unsure how I have to change all of the above snippets  
> of code,
> in order to keep the same functionality, but without the time delays  
> when
> Emacs starts up...
>
>
>> Hope some of this helps. If not, let me see you full configuration  
>> - maybe I
>> can spot something else.
>
> My .emacs file is there:
>
> http://www.mygooglest.com/sva/emacs-init-sva.el
>
> Though it's well documented, it's not a tiny one... But very well  
> structured,
> so quite easy to find stuff related to each other.
>
> I'm always interested by any comment...
>
> Best regards,
>  Seb
>
> -- 
> Sébastien Vauban
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi,

Carsten Dominik wrote:
> On Sep 30, 2010, at 12:20 AM, Sébastien Vauban wrote:
>> Carsten Dominik wrote:
>>> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>>>> Carsten Dominik wrote:
>>>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>>>> Of course, I have many, many files in Org mode. All files I write (or
>>>>>> touch) in fact.
>>>>>>
>>>>>> Of course, I would like to search through my files at some point in
>>>>>> time. The problem is the load-time of my Emacs, now 221 seconds, coming
>>>>>> from 20 seconds before the heavy use of Org...
>>>>>
>>>>> 4 minutes of startup time is entirely unacceptable. And I think you need
>>>>> to identify what is causing this.
>>>>>
>>>>> fontification at display time is standard, I believe.
>>>>
>>>> It seems not, from what I see in the Messages buffer. I really don't have
>>>> the impression of having fiddled with that, really.
>>
>>>>> You must be doing something strange [...] maybe forcing global
>>>>> fontification for each file or so.
>>>>
>>>> To repeat myself, no to that question. But ispell and flyspell are called
>>>> for Org files. Maybe there are interactions?
>>>>
>>>> [Using org-agenda-text-search-extra-files] reduces my load time from 221
>>>> seconds down to 92 seconds. Already a huge diff!
>>>
>>> Still bad though. I am wondering what is causing the fontification
>>> message. I do not get this, so it must be something in your setup. You
>>> should try to find out when this is happening and why.
>>
>> You've seen, from my reply to Matt, why my *Messages* buffer is more
>> verbose than yours. It's on purpose, just to be able to figure out more
>> easily what's going on. Though, I don't have yet...
>>
>>> Also, you might consider to remove (org-agenda-list) from .emacs. I think
>>> it is pretty much always a bad idea to put a command like this into your
>>> startup. Just make it a habit to call it early after starting Emacs.
>>
>> Honestly, that would not change that much. Having to wait 92 seconds at
>> startup, or a long time just a few minutes later has the same impact for
>> me. And: don't try to make me stop using Org ;-))
>
> :-)
>
> The reason why I am suggesting to remove this command is more that it make
> emacs more usable for command line usage etc.

Not convinced yet ;-)

For command-line Emacs, either I use emacsclient or emacs without my config
file, so that's not the problem there.

And the fact is I need my agenda quite early in the process (after firing up
Emacs).


> Furthermore, you are adding quite a bit to font-lock, but I don't see why it
> should be so slow.

Does someone know if the way I use column-marker and the highlighting some
keywords is wrong?


> My Emacs starts up in 4 seconds at most. When I create an agenda, I am
> talking 1 second to maybe 2. I have about 10 files contributing to the
> agenda.

I would love such figures.


>>> It also seems to me you some of the extra packages might be activated
>>> several times in each file. One possible reason could be that you have put
>>> the code to turn them on into several hooks like text-mode-hook,
>>> outline-mode-hook, and org-mode-hook. Turning on org-mode will first run
>>> text-mode-hook, then outline-mode-hook, then org-mode-hook ..... Maybe you
>>> are also calling font-lock-fontify-buffer explicitly in one of your hooks?
>>
>> The only fontification customs that I do are here:
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; special words
>> (setq keywords-critical-pattern
>>      "\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\|
>> [Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff]
>> [Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>> (make-face 'keywords-critical)
>> (GNUEmacs (set-face-attribute 'keywords-critical nil :foreground "red"
>> :background "yellow" :weight 'bold))
>>
>> (setq keywords-org-critical-pattern
>>      "\\(BUGS\\|FIXME\\|XXX\\|[^*] TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm]
>> [Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa]
>> [Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>>                                  ; smaller subset of keywords for ensuring
>> no conflict with Org mode TODO keywords
>>
>> ;; FIXME Highlighting all special keywords but "TODO" in Org mode is already
>> a
>> ;; good step. Though, a nicer integration would be that "TODO" strings in
>> the
>> ;; headings are not touched by this code, and that only "TODO" strings in
>> the
>> ;; text body would be. Don't know (yet) how to do that...
>> (make-face 'keywords-org-critical)
>> (GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground "red"
>> :background "yellow" :weight 'bold))
>>
>> (setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
>> (make-face 'keywords-normal)
>> (GNUEmacs (set-face-attribute 'keywords-normal nil :foreground "magenta2"
>> :background "yellow"))
>>
>> ;; set up highlighting of special words for proper selected major modes only
>> (dolist (mode '(fundamental-mode
>>                svn-log-view-mode
>>                text-mode))  ; no interference with Org mode (which derives
>> from text-mode)
>>  (font-lock-add-keywords mode
>>                          `((,keywords-critical-pattern 1 'keywords-
>> critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; set up highlighting of special words for Org mode only
>> (dolist (mode '(org-mode))
>>  (font-lock-add-keywords mode
>>                          `((,keywords-org-critical-pattern 1
>> keywords-org-critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; add fontification patterns (even in comments) to a selected major mode
>> ;; *and* all major modes derived from it
>> (defun fontify-keywords ()
>>  (interactive)
>>  (font-lock-add-keywords nil
>>                          `((,keywords-critical-pattern 1 'keywords-
>> critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; set up highlighting of special words for selected major modes *and* all
>> ;; major modes derived from them
>> (dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl-
>> mode-hook css-mode-hook
>>                emacs-lisp-mode-hook
>>                html-mode-hook
>>                java-mode-hook
>>                latex-mode-hook lisp-mode-hook
>>                makefile-mode-hook message-mode-hook
>>                php-mode-hook python-mode-hook
>>                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
>>  (add-hook hook 'fontify-keywords))
>> --8<---------------cut here---------------end--------------->8---
>>
>> The goal is to highlight words such as FIXME, WARNING, XXX, TODO in as many
>> files as possible.
>>
>> For Org, a small custom (of the custom) is done in order not to catch all
>> TODO words: the TODO words in headlines (i.e., preceded by `* ') are *not*
>> highlighted.
>>
>> They are well highlighted when found in other portions of (normal) text.
>>
>> Other custom:
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; highlight columns 78 to 80 in some modes
>> (when (try-require 'column-marker)
>>  (dolist (hook '(emacs-lisp-mode-hook
>>                  cperl-mode-hook
>>                  shell-mode-hook
>>                  text-mode-hook
>>                  change-log-mode-hook
>>                  makefile-mode-hook
>>                  message-mode-hook
>>                  texinfo-mode-hook))
>>    (add-hook hook (lambda ()
>>                     (interactive)
>>                     (column-marker-1 78)
>>                     (column-marker-2 79)
>>                     (column-marker-3 80))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> for getting markers in columns 78 to 80, showing me I'm good for a `M-q'...
>>
>> I've done multiple tests, now:
>>
>> - with none of the 2 above sets of customs
>> - with the 1st custom disabled, and the column-marker one enabled
>> - with both enabled
>>
>> Times observed (in seconds):
>>
>> |      | no highlight | column marker only | all highlights |
>> |------+--------------+--------------------+----------------|
>> |      |           63 |                 82 |             91 |
>> |      |           65 |                 84 |             96 |
>> |      |           76 |                100 |            130 |
>> |------+--------------+--------------------+----------------|
>> | mean |           68 |                 89 |            106 |
>>
>> (note that times are really not that stable, while I always launched Emacs
>> in the same conditions, without doing anything else during that time!)
>>
>> So, the above 2 sets of customs have a real impact on the load time.
>>
>> But I'm still unsure how I have to change all of the above snippets of
>> code, in order to keep the same functionality, but without the time delays
>> when Emacs starts up...
>>
>>
>>> Hope some of this helps. If not, let me see you full configuration - maybe
>>> I can spot something else.
>>
>> My .emacs file is there:
>>
>> http://www.mygooglest.com/sva/emacs-init-sva.el
>>
>> Though it's well documented, it's not a tiny one... But very well
>> structured, so quite easy to find stuff related to each other.
>>
>> I'm always interested by any comment...

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Nick Dokos]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:

> >>> Also, you might consider to remove (org-agenda-list) from .emacs. I think
> >>> it is pretty much always a bad idea to put a command like this into your
> >>> startup. Just make it a habit to call it early after starting Emacs.
> >>
> >> Honestly, that would not change that much. Having to wait 92 seconds at
> >> startup, or a long time just a few minutes later has the same impact for
> >> me. And: don't try to make me stop using Org ;-))
> >
> > :-)
> >
> > The reason why I am suggesting to remove this command is more that it make
> > emacs more usable for command line usage etc.
> 
> Not convinced yet ;-)
> 
> For command-line Emacs, either I use emacsclient or emacs without my config
> file, so that's not the problem there.
> 
> And the fact is I need my agenda quite early in the process (after firing up
> Emacs).
> 

Maybe you should try adding  a call to the profiler to your .emacs
and try to measure e.g. how much the font locking costs. Off the top of
my head and entirely untested, adding

,----
| (require 'elp)
| (elp-instrument-package "font-lock")
| 
| ;; rest of your .emacs
| 
| (elp-results)
`----

might be enough.

HTH,
Nick

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Nick,

Nick Dokos wrote:
> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> wrote:
>>>>> Also, you might consider to remove (org-agenda-list) from .emacs. I
>>>>> think it is pretty much always a bad idea to put a command like this
>>>>> into your startup. Just make it a habit to call it early after starting
>>>>> Emacs.
>>>>
>>>> Honestly, that would not change that much. Having to wait 92 seconds at
>>>> startup, or a long time just a few minutes later has the same impact for
>>>> me.
>
> Maybe you should try adding  a call to the profiler to your .emacs
> and try to measure e.g. how much the font locking costs. Off the top of
> my head and entirely untested, adding
>
> ,----
> | (require 'elp)
> | (elp-instrument-package "font-lock")
> | 
> | ;; rest of your .emacs
> | 
> | (elp-results)
> `----
>
> might be enough.

Here are the results.

--8<---------------cut here---------------start------------->8---
font-lock-fontify-buffer                                   96          15.383627999  0.1602461249
font-lock-default-fontify-buffer                           96          15.378296000  0.1601905833
font-lock-fontify-region                                   96          15.284044000  0.1592087916
font-lock-default-fontify-region                           96          15.282674000  0.1591945208
font-lock-fontify-keywords-region                          96          15.184516     0.1581720416
font-lock-prepend-text-property                            8608        0.2712220000  3.150...e-05
font-lock-unfontify-region                                 96          0.0859729999  0.0008955520
font-lock-add-keywords                                     166         0.0800819999  0.0004824216
font-lock-compile-keywords                                 138         0.0671930000  0.0004869057
font-lock-default-unfontify-region                         96          0.0378660000  0.0003944375
font-lock-mode                                             142         0.0344680000  0.0002427323
font-lock-default-function                                 142         0.028445      0.0002003169
font-lock-mode-internal                                    73          0.0272529999  0.0003733287
font-lock-turn-on-thing-lock                               73          0.0238939999  0.0003273150
font-lock-compile-keyword                                  5505        0.0225559999  4.097...e-06
font-lock-set-defaults                                     525         0.0174539999  3.324...e-05
font-lock-remove-keywords                                  130         0.0018569999  1.428...e-05
font-lock-change-mode                                      33          0.0016060000  4.866...e-05
font-lock-eval-keywords                                    81          0.0012500000  1.543...e-05
font-lock-value-in-major-mode                              187         0.0003879999  2.074...e-06
font-lock-extend-region-wholelines                         96          0.000309      3.21875e-06
font-lock-extend-region-multiline                          96          0.0002549999  2.656...e-06
font-lock-extend-jit-lock-region-after-change              28          0.0002529999  9.035...e-06
font-lock-after-fontify-buffer                             96          0.0002499999  2.604...e-06
font-lock-choose-keywords                                  41          8.899...e-05  2.170...e-06
font-lock-update-removed-keyword-alist                     36          6.8e-05       1.888...e-06
--8<---------------cut here---------------end--------------->8---

with 94 files in my =org-agenda-files=. Yes, 94, not 96?

You can effectively see that =font-lock-*= is taking me 15s during my Emacs
startup.

Though, I don't know what's the next action to be taken from this observation.
Can you guide me?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Carsten Dominik]

On Sep 30, 2010, at 7:08 PM, Sébastien Vauban wrote:

> Hi Nick,
>
> Nick Dokos wrote:
>> Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:
>>>>>> Also, you might consider to remove (org-agenda-list)  
>>>>>> from .emacs. I
>>>>>> think it is pretty much always a bad idea to put a command like  
>>>>>> this
>>>>>> into your startup. Just make it a habit to call it early after  
>>>>>> starting
>>>>>> Emacs.
>>>>>
>>>>> Honestly, that would not change that much. Having to wait 92  
>>>>> seconds at
>>>>> startup, or a long time just a few minutes later has the same  
>>>>> impact for
>>>>> me.
>>
>> Maybe you should try adding  a call to the profiler to your .emacs
>> and try to measure e.g. how much the font locking costs. Off the  
>> top of
>> my head and entirely untested, adding
>>
>> ,----
>> | (require 'elp)
>> | (elp-instrument-package "font-lock")
>> |
>> | ;; rest of your .emacs
>> |
>> | (elp-results)
>> `----
>>
>> might be enough.
>
> Here are the results.
>
> --8<---------------cut here---------------start------------->8---
> font-lock-fontify-buffer                                    
> 96          15.383627999  0.1602461249

Well, the point I am trying to make already for a number of messages  
is this:
font-lock-fontify-buffer should not be called at all - it is not called
in my setup.

But, apparently, fontlock is bad, but not the worst issue you are  
encountering.

> font-lock-default-fontify-buffer                            
> 96          15.378296000  0.1601905833
> font-lock-fontify-region                                    
> 96          15.284044000  0.1592087916
> font-lock-default-fontify-region                            
> 96          15.282674000  0.1591945208
> font-lock-fontify-keywords-region                           
> 96          15.184516     0.1581720416
> font-lock-prepend-text-property                             
> 8608        0.2712220000  3.150...e-05
> font-lock-unfontify-region                                  
> 96          0.0859729999  0.0008955520
> font-lock-add-keywords                                      
> 166         0.0800819999  0.0004824216
> font-lock-compile-keywords                                  
> 138         0.0671930000  0.0004869057
> font-lock-default-unfontify-region                          
> 96          0.0378660000  0.0003944375
> font-lock-mode                                              
> 142         0.0344680000  0.0002427323
> font-lock-default-function                                  
> 142         0.028445      0.0002003169
> font-lock-mode-internal                                     
> 73          0.0272529999  0.0003733287
> font-lock-turn-on-thing-lock                                
> 73          0.0238939999  0.0003273150
> font-lock-compile-keyword                                   
> 5505        0.0225559999  4.097...e-06
> font-lock-set-defaults                                      
> 525         0.0174539999  3.324...e-05
> font-lock-remove-keywords                                   
> 130         0.0018569999  1.428...e-05
> font-lock-change-mode                                       
> 33          0.0016060000  4.866...e-05
> font-lock-eval-keywords                                     
> 81          0.0012500000  1.543...e-05
> font-lock-value-in-major-mode                               
> 187         0.0003879999  2.074...e-06
> font-lock-extend-region-wholelines                          
> 96          0.000309      3.21875e-06
> font-lock-extend-region-multiline                           
> 96          0.0002549999  2.656...e-06
> font-lock-extend-jit-lock-region-after-change               
> 28          0.0002529999  9.035...e-06
> font-lock-after-fontify-buffer                              
> 96          0.0002499999  2.604...e-06
> font-lock-choose-keywords                                   
> 41          8.899...e-05  2.170...e-06
> font-lock-update-removed-keyword-alist                      
> 36          6.8e-05       1.888...e-06
> --8<---------------cut here---------------end--------------->8---
>
> with 94 files in my =org-agenda-files=. Yes, 94, not 96?
>
> You can effectively see that =font-lock-*= is taking me 15s during  
> my Emacs
> startup.
>
> Though, I don't know what's the next action to be taken from this  
> observation.
> Can you guide me?

You should instrument other packages like org and outline to see where  
the rest of the time is eaten.

- Carsten

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Carsten and Nick,

Carsten Dominik wrote:
> On Sep 30, 2010, at 7:08 PM, Sébastien Vauban wrote:
>> Nick Dokos wrote:
>>> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> wrote:
>>>>>> Having to wait 92 seconds at startup, or a long time just a few minutes
>>>>>> later has the same impact for me.
>>>
>>> Maybe you should try adding a call to the profiler to your .emacs and try
>>> to measure e.g. how much the font locking costs. Off the top of my head
>>> and entirely untested, adding (require 'elp) might be enough.
>>
>> Here are the results.
>>
>> font-lock-fontify-buffer               96   15.383627999  0.1602461249
>
> Well, the point I am trying to make already for a number of messages is
> this: font-lock-fontify-buffer should not be called at all - it is not
> called in my setup.
>
> But, apparently, fontlock is bad, but not the worst issue you are
> encountering.
>
>> font-lock-default-fontify-buffer       96   15.378296000  0.1601905833
>> font-lock-fontify-region               96   15.284044000  0.1592087916
>> font-lock-default-fontify-region       96   15.282674000  0.1591945208
>> font-lock-fontify-keywords-region      96   15.184516     0.1581720416
>> font-lock-prepend-text-property        8608   0.2712220000  3.150...e-05
>> font-lock-unfontify-region             96   0.0859729999  0.0008955520
>> font-lock-add-keywords                 166   0.0800819999  0.0004824216
>> font-lock-compile-keywords             138   0.0671930000  0.0004869057
>> font-lock-default-unfontify-region     96   0.0378660000  0.0003944375
>> font-lock-mode                         142   0.0344680000  0.0002427323

For whatever reason, I've noticed that there are:

- 96 calls for 32 files in my org-agenda-files
- 99 calls for 33 files in my org-agenda-files

hence, 3 font-lock-fontify-buffer calls per Org file[1].

Anyway, I now do have found the culprit: the =column-marker= package (which
highlights my columns 78, 79 and 80 of text), loaded in my .emacs file by this
snippet):

#+begin_src emacs-lisp
  ;; highlight columns 78 to 80 in some modes
  (when (require 'column-marker)

    (dolist (hook '(emacs-lisp-mode-hook
                    cperl-mode-hook
                    shell-mode-hook
                    text-mode-hook
                    change-log-mode-hook
                    makefile-mode-hook
                    message-mode-hook
                    texinfo-mode-hook))
      (add-hook hook (lambda ()
                       (interactive)
                       (column-marker-1 78)
                       (column-marker-2 79)
                       (column-marker-3 80)))))
#+end_src

It accounts for a bit more than 10 s delay in the total time of my .emacs file
-- which, currently, still calls (org-agenda-list):

- with it, total time is 70s - 73s
- without it, total time is 80s - 84s

The difference is really huge...

Though, such a capability being really nice, do you know some replacement for
this?

Best regards,
  Seb

Footnotes:
[1] The offending package highlights 3 columns in the buffer: maybe that's the
    factor 3 for every Org file...

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Eric S Fraga]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:

[...]

> Anyway, I now do have found the culprit: the =column-marker= package (which
> highlights my columns 78, 79 and 80 of text), loaded in my .emacs file by this
> snippet):

[...]

> Though, such a capability being really nice, do you know some replacement for
> this?

Maybe check out /develock-mode/ in Emacs 23.x.  

Aside: Personally, I don't like it and I wish I could figure out how to
turn it off so if anybody can help with this, I would greatly appreciate
it!  It appears to be turned on by default in some modes (not org,
luckily) but I have no found any hook which turns it on...
-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 23.2.1
: using Org-mode version 7.3 (release_7.3.162.g8e74)

Re: Re: Having (too) many files in org-agenda-files

[Eric S Fraga]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
>> Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:
>> 
>> [...]
>> 
>> > Anyway, I now do have found the culprit: the =column-marker= package (which
>> > highlights my columns 78, 79 and 80 of text), loaded in my .emacs file by this
>> > snippet):
>> 
>> [...]
>> 
>> > Though, such a capability being really nice, do you know some replacement for
>> > this?
>> 
>> Maybe check out /develock-mode/ in Emacs 23.x.  
>> 
>
> The description and comments on emacswiki.org are quite colorful (in
> a sick sort of way :-) ):
>
> "I tried it in Lisp mode and it filled my buffer with so much ketchup
> and mustard I nearly fell off my chair."

Love it!  That's exactly how I feel about it.

The only thing I can figure is that whoever wrote this mode must be
colour blind... :(

>> Aside: Personally, I don't like it and I wish I could figure out how to
>> turn it off so if anybody can help with this, I would greatly appreciate
>> it!  It appears to be turned on by default in some modes (not org,
>> luckily) but I have no found any hook which turns it on...
>
> M-x locate-library <RET> develock <RET>
>
> M-x shell <RET>
> rm /path/from/locate-library/above/develock.*
>
> restart emacs and fix the errors
>
> :-)

Okay, a little drastic but I may yet reach this point...

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 23.2.1
: using Org-mode version 7.3 (release_7.3.163.g76877)

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Eric,

Eric S Fraga wrote:
> Nick Dokos <nicholas.dokos-VXdhtT5mjnY@public.gmane.org> writes:
>> Eric S Fraga <e.fraga-hclig2XLE9Zaa/9Udqfwiw@public.gmane.org> wrote:
>>> Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>>> 
>>>> Anyway, I now do have found the culprit: the =column-marker= package (which
>>>> highlights my columns 78, 79 and 80 of text), loaded in my .emacs file by this
>>>> snippet):
>>>>
>>>> Though, such a capability being really nice, do you know some replacement for
>>>> this?
>>> 
>>> Maybe check out /develock-mode/ in Emacs 23.x.  
>>
>> M-x locate-library <RET> develock <RET>

Don't have it:

    No library develock in search path

on:

    GNU Emacs 23.1.50.1 (i386-mingw-nt5.1.2600) of 2009-10-14 on
    LENNART-69DE564 (patched)

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[David Maus]

[-- Attachment #1.1: Type: text/plain, Size: 1057 bytes --]

At Thu, 02 Dec 2010 10:25:54 +0100,
Sébastien Vauban wrote:
> 
> Hi Eric,
> 
> Eric S Fraga wrote:
> > Nick Dokos <nicholas.dokos@hp.com> writes:
> >> Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> >>> Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> writes:
> >>> 
> >>>> Anyway, I now do have found the culprit: the =column-marker= package (which
> >>>> highlights my columns 78, 79 and 80 of text), loaded in my .emacs file by this
> >>>> snippet):
> >>>>
> >>>> Though, such a capability being really nice, do you know some replacement for
> >>>> this?
> >>> 
> >>> Maybe check out /develock-mode/ in Emacs 23.x.  
> >>
> >> M-x locate-library <RET> develock <RET>
> 
> Don't have it:
> 
>     No library develock in search path
> 
> on:
> 
>     GNU Emacs 23.1.50.1 (i386-mingw-nt5.1.2600) of 2009-10-14 on
>     LENNART-69DE564 (patched)

Maybe this is obvious, but: http://www.emacswiki.org/emacs/DevelockMode

Best,
  -- David
-- 
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #1.2: Type: application/pgp-signature, Size: 230 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Jeff Horn]

> Though, such a capability being really nice, do you know some replacement for
> this?

Pardon the noise if this has an obvious answer... but why not just use
auto-fill with a column set to 80 instead of the highlighting? Is it
because sometimes you want to overflow past the 80th column?

Jeff

-- 
Jeffrey Horn
Graduate Lecturer and PhD Student in Economics
George Mason University

(704) 271-4797
jhorn@gmu.edu
jrhorn424@gmail.com

http://www.failuretorefrain.com/jeff/

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Jeff and Eric,

Jeff Horn wrote:
>> Though, such a capability being really nice, do you know some replacement for
>> this?
>
> Pardon the noise if this has an obvious answer... but why not just use
> auto-fill with a column set to 80 instead of the highlighting? Is it
> because sometimes you want to overflow past the 80th column?

The auto-fill is not sufficient, for many reasons:

- other people do work in your text

- or, even alone: you have a correctly filled paragraph, and you come and add
  a few words in the middle: everything is moved to the right, but *not
  auto-filled* (unless your cursor position is above 80 at that point)!

Eric, I'll have a look at develock. Thanks.

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Daniel Clemente]

  Do you have your .org files inside a version control system like git or bzr?
  In that case, it's vc-mode who is delaying too much.
  See the thread http://www.mail-archive.com/emacs-orgmode@gnu.org/msg25902.html for a dirty workaround (disabling vc-mode).

  In my system I have now 200 agenda files (120k lines, 5'6 Mb) and I must wait 1 minute on a fast computer (1'45" on a slower one) to run org-agenda-list.
  I timed it closing all *.org and running:   time emacsclient -n --eval '(org-agenda-list)' 

  The problem is in vc-mode, not org. It is too slow. And what is exactly doing? I attach it at the end of this mail. Sometimes it's ls-files, other diff-index.

-- Daniel

#+begin_example
Debugger entered--Lisp error: (quit)
  call-process("git" nil (t nil) nil "ls-files" "-c" "-z" "--" "wiki/Imagen.org")
  apply(call-process "git" nil (t nil) nil ("ls-files" "-c" "-z" "--" "wiki/Imagen.org"))
  process-file("git" nil (t nil) nil "ls-files" "-c" "-z" "--" "wiki/Imagen.org")
  apply(process-file "git" nil (t nil) nil "ls-files" ("-c" "-z" "--" "wiki/Imagen.org"))
  vc-git--call((t nil) "ls-files" "-c" "-z" "--" "wiki/Imagen.org")
  apply(vc-git--call (t nil) "ls-files" ("-c" "-z" "--" "wiki/Imagen.org"))
  vc-git--out-ok("ls-files" "-c" "-z" "--" "wiki/Imagen.org")
  byte-code(…)
  vc-git-registered("/home/dc/.mirp/wiki/Imagen.org")
  apply(vc-git-registered "/home/dc/.mirp/wiki/Imagen.org")
  vc-call-backend(Git registered "/home/dc/.mirp/wiki/Imagen.org")
  #[…]
  mapc(#[(b) "…" [b file vc-call-backend registered vc-file-setprop vc-backend throw found t] 4] (RCS CVS SVN SCCS Bzr Git Hg Mtn Arch))
  byte-code("…")
  vc-registered("/home/dc/.mirp/wiki/Imagen.org")
  vc-backend("/home/dc/.mirp/wiki/Imagen.org")
  #[…]
  vc-find-file-hook()
  run-hooks(find-file-hook)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer Imagen.org> "~/wiki/Imagen.org" nil nil "~/wiki/Imagen.org" (5929158 19))
  find-file-noselect("/home/dc/.mirp/wiki/Imagen.org" nil nil nil)
  find-file("/home/dc/.mirp/wiki/Imagen.org")

#+end_example




El Thu, 30 Sep 2010 19:08:02 +0200 Sébastien Vauban va escriure:
> 
> Hi Nick,
> 
> Nick Dokos wrote:
> > Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:
> >>>>> Also, you might consider to remove (org-agenda-list) from .emacs. I
> >>>>> think it is pretty much always a bad idea to put a command like this
> >>>>> into your startup. Just make it a habit to call it early after starting
> >>>>> Emacs.
> >>>>
> >>>> Honestly, that would not change that much. Having to wait 92 seconds at
> >>>> startup, or a long time just a few minutes later has the same impact for
> >>>> me.
> >
> > Maybe you should try adding  a call to the profiler to your .emacs
> > and try to measure e.g. how much the font locking costs. Off the top of
> > my head and entirely untested, adding
> >
> > ,----
> > | (require 'elp)
> > | (elp-instrument-package "font-lock")
> > | 
> > | ;; rest of your .emacs
> > | 
> > | (elp-results)
> > `----
> >
> > might be enough.
> 
> Here are the results.
> 
> --8<---------------cut here---------------start------------->8---
> font-lock-fontify-buffer                                   96          15.383627999  0.1602461249
> font-lock-default-fontify-buffer                           96          15.378296000  0.1601905833
> font-lock-fontify-region                                   96          15.284044000  0.1592087916
> font-lock-default-fontify-region                           96          15.282674000  0.1591945208
> font-lock-fontify-keywords-region                          96          15.184516     0.1581720416
> font-lock-prepend-text-property                            8608        0.2712220000  3.150...e-05
> font-lock-unfontify-region                                 96          0.0859729999  0.0008955520
> font-lock-add-keywords                                     166         0.0800819999  0.0004824216
> font-lock-compile-keywords                                 138         0.0671930000  0.0004869057
> font-lock-default-unfontify-region                         96          0.0378660000  0.0003944375
> font-lock-mode                                             142         0.0344680000  0.0002427323
> font-lock-default-function                                 142         0.028445      0.0002003169
> font-lock-mode-internal                                    73          0.0272529999  0.0003733287
> font-lock-turn-on-thing-lock                               73          0.0238939999  0.0003273150
> font-lock-compile-keyword                                  5505        0.0225559999  4.097...e-06
> font-lock-set-defaults                                     525         0.0174539999  3.324...e-05
> font-lock-remove-keywords                                  130         0.0018569999  1.428...e-05
> font-lock-change-mode                                      33          0.0016060000  4.866...e-05
> font-lock-eval-keywords                                    81          0.0012500000  1.543...e-05
> font-lock-value-in-major-mode                              187         0.0003879999  2.074...e-06
> font-lock-extend-region-wholelines                         96          0.000309      3.21875e-06
> font-lock-extend-region-multiline                          96          0.0002549999  2.656...e-06
> font-lock-extend-jit-lock-region-after-change              28          0.0002529999  9.035...e-06
> font-lock-after-fontify-buffer                             96          0.0002499999  2.604...e-06
> font-lock-choose-keywords                                  41          8.899...e-05  2.170...e-06
> font-lock-update-removed-keyword-alist                     36          6.8e-05       1.888...e-06
> --8<---------------cut here---------------end--------------->8---
> 
> with 94 files in my =org-agenda-files=. Yes, 94, not 96?
> 
> You can effectively see that =font-lock-*= is taking me 15s during my Emacs
> startup.
> 
> Though, I don't know what's the next action to be taken from this observation.
> Can you guide me?
> 
> Best regards,
>   Seb
> 
> -- 
> Sébastien Vauban
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>
>> I would like to search through my files at some point in time. I even would
>> like to search through your files at some point in time, I mean through
>> =org-mode/contrib/babel= and =Worg= for example.
>>
>> Hence, I put them in =org-agenda-files= [...].
>
> About a separate variable for search file, that does of cause exist:
> org-agenda-text-search-extra-files

Funny, though, that these files are already scanned when:

- clocking in a task of some org-agenda-files, or

  I don't see the reason for this.

- calling the org-dispatcher (=C-c a=)

  I understand that such scan should occur as soon as I press =C-c a s=, but
  why already when pressing =C-c a=?

Best regards,
  Seb

-- 
Sébastien Vauban


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Having (too) many files in org-agenda-files

[Carsten Dominik]

On Sep 30, 2010, at 3:37 PM, Sébastien Vauban wrote:

> Hi Carsten,
>
> Carsten Dominik wrote:
>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>
>>> I would like to search through my files at some point in time. I  
>>> even would
>>> like to search through your files at some point in time, I mean  
>>> through
>>> =org-mode/contrib/babel= and =Worg= for example.
>>>
>>> Hence, I put them in =org-agenda-files= [...].
>>
>> About a separate variable for search file, that does of cause exist:
>> org-agenda-text-search-extra-files
>
> Funny, though, that these files are already scanned when:
>
> - clocking in a task of some org-agenda-files, or
>
>  I don't see the reason for this.
>
> - calling the org-dispatcher (=C-c a=)
>
>  I understand that such scan should occur as soon as I press =C-c a  
> s=, but
>  why already when pressing =C-c a=?

No, this is wrong.  Please check the content of org-agenda-files.

- Carsten

>
> Best regards,
>  Seb
>
> -- 
> Sébastien Vauban
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: Having (too) many files in org-agenda-files

[Matt Lundin]

Hi Sebastian, 

I wrote this response yesterday, but it may have gotten lost in gmane's
reverse obfuscation system for email addresses. :)

(OT: Is there a "correct" way to respond to your messages in gnus? I
subscribe to org-mode via gmane/nntp and in your emails the org-mode ML
address is obfuscated and redirected:
emacs-orgmode-mXXj517/zsQ@public.gmane.org.)

-----

Sébastien Vauban <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org>
writes:

> Of course, I have many, many files in Org mode. All files I write (or
> touch) in fact.
>
> Of course, I would like to search through my files at some point in time.
> I even would like to search through your files at some point in time, I mean
> through =org-mode/contrib/babel= and =Worg= for example.
>

The drawback of org-mode's plain text format is that org-mode needs to
load, parse, and fontify, and preserve in memory all files it queries
(i.e., agenda files). Too many agenda files can thus cause a pretty big
performance hit (both when loading agenda files and constructing agenda
views). To avoid this, I would recommend including only important files
in org-agenda-files. Possible solutions for searching other files
include:

1. loading the other directories when needed (e.g., through a function
   that changes the value of org-agenda-files)
2. using the variable org-agenda-text-search-extra-files (see below)
3. searching other directories with external tools (grep, perl script,
   etc.)

> The problem is the load-time of my Emacs, now:
>
> Emacs Init startup time: 221 seconds.

Do you invoke an agenda command in your emacs, such as
org-agenda-to-appt? That would cause org-mode to load all org files.

> coming from 20 seconds before the heavy use of Org...
>
> with tens of times such lines in my *Messages* buffer:
>
> OVERVIEW
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps.............................................)
> Checking for library `filladapt'... Found
> Fontifying Axa.org... (regexps..............................................)
> Checking for library `filladapt'... Found
> +-> Requiring `outline-mode-easy-bindings' (already loaded)
> +-> Requiring `ispell' (already loaded)
> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
> Starting new Ispell process [en_US] ...
> Checking for library `filladapt'...
> +-> Requiring `filladapt' (already loaded)
> Checking for library `filladapt'... Found

It seems that filladapt is causing some of the verbosity (and possible
slowness) there. When loading agenda files, my *Messages* buffer only
shows:

OVERVIEW [29 times]

> You'll tell me: not a problem, you do that only once a day, and you
> use Emacs client/server for the rest of the time. True. A bit, because
> I sometimes have to restart Emacs for testing a fresh one (not
> impacted by defvars already defined, or deffaces, etc.).
>
> Having to wait almost 4 minutes is a real pain. So, here my
> comments/questions:
>
> - Isn't it possible to delay the fontification/ispell/etc. to when we really
>   display (i.e., pop up) the buffer?  I guess this must be a major component
>   of the time this takes.

I imagine the fontification is necessary for the agenda to function
properly. Thus, org files are parsed en masse when the agenda is called
for the first time. (Hence my question about whether you were calling
the agenda from ~/.emacs.)

>
> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you want
>   have scanned for the agenda construction, and an extra list such as
>   =org-search-files= for files not containing any dates?  Then, some time
>   would have to be taken when =C-c a s=, but not before. And not if you don't
>   search for anything in your Org files during that Emacs session...
>

This already exists. See org-agenda-text-search-extra-files. 

Best,
Matt

Re: Having (too) many files in org-agenda-files

[Matthew Lundin]

Sébastien,

Matt Lundin <mdl@imapmail.org> writes:

> Hi Sebastian, 

My apologies for misspelling your name!

- Matt