Scaling org-mode

Scaling org-mode

[Dave Täht]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I have really been enjoying importing my life into org-mode, which I've
been doing for about two months now. 

But.

It currently visits about 100 files and 10k of text to construct the
agenda. It's starting to get kind of slow and interrupt my workflow,
particularly the background process that scans them. 

While the system is effectively frozen, my message buffer fills up with
messages about setting the flyspell dictionary to en, etc. This is quite
annoying with text to speech turned on. I ended up just having appts
spoken.

;; quick hack for saying announcements, need more thought turned into it.

(when window-system
  (defun dtaht/say-stuff (id msg &optional delay vattrib hattrib font) 
    "Speak a message msg. Currently requires say.el"
    (unless vattrib (setq vattrib "top"))
    (unless hattrib (setq hattrib "right"))
    (unless delay (setq delay 5000))
    (unless font (setq font "Arial 12"))
    (save-window-excursion
      (say (dtaht/ssml-escape msg)
	))))

;; my personal fav, run every 15 minutes

(defun nag-timer () "Nag me when there isn't a clock running"  
  (interactive)
  (unless (marker-buffer org-clock-marker)
    (say "Are you mating now?")))

It gets a bit chunky even when all the org files are in memory (and I
have gobs of memory).

Solution #1) cut the number of files down - is a good one. I probably
can cut those files easily in half right now. The problem is that I have
about 600 more files to import (scenes from a book), and I really like
the idea of being able to know what my characters are doing in 2023, and
separate files was kind of useful at one point.

That's a couple hundred k of text (what I have in there now is mostly
"normal" items for managing my personal and professional life)

Solution #2) Make org-mode faster. I am compiling and installing
org-mode from git at the moment. So it's compiled. I have a lot of hooks
installed for text buffers - At minimum, auto-capitalize, flyspell, yas,
wrap, and abbrev. Most (all?) of those are compiled, too, but aren't
needed to be run or initialized when merely being visited.

so thought 1) would be to come up with some sort of text mode hook that
only hooks in when a human is viewing or editing the buffer, not when it
is happening programmatically.

so thought 2) would be to have it only attempt to construct background
agendas when the system is otherwise idle for a few minutes. I don't
know how to do that, I figure wrapping this bit with something that
could detect idleness instead of just running arbitrarily would be good.

  (run-at-time nil 3600 'org-agenda-to-appt)

don't know how to detect idleness.

And thought 0) would be to understand emacs well enough to profile
what's really going on as maybe "starting a new flyspell process" is not
really the problem but a symptom, but I don't (yet). (suggestions
desired. I have read up a bit on emacs profiling)

I am deeply grateful for the existence of org-mode and the fine work
going into it. I wrote a nice blog entry about it recently. 

http://the-edge.blogspot.com/2009/08/going-retro-re-adopting-emacs.html

TIA.

- -- 
Dave Taht
http://the-edge.blogspot.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>

iEYEARECAAYFAkqshuUACgkQpdejJcOV4uRkTwCgyz5IMWJTzCHa8CmUAgU/fyIa
cvwAoJKQ5Gr+1vlSAbEfbKob76xJIvB8
=/Xdv
-----END PGP SIGNATURE-----

Re: Scaling org-mode

[Matt Lundin]

d@teklibre.org (Dave Täht) writes:

> I have really been enjoying importing my life into org-mode, which I've
> been doing for about two months now. 
>
> But.
>
> It currently visits about 100 files and 10k of text to construct the
> agenda. It's starting to get kind of slow and interrupt my workflow,
> particularly the background process that scans them. 
>
> While the system is effectively frozen, my message buffer fills up with
> messages about setting the flyspell dictionary to en, etc. This is quite
> annoying with text to speech turned on. I ended up just having appts
> spoken.

My guess is that this is a flyspell problem. I only get messages about
flyspell if I don't have aspell or the proper dictionary installed. Do
you get these messages when you open buffers in other text modes? What
exactly are the messages? What is your flyspell config?

> Solution #1) cut the number of files down - is a good one. I probably
> can cut those files easily in half right now. The problem is that I have
> about 600 more files to import (scenes from a book), and I really like
> the idea of being able to know what my characters are doing in 2023, and
> separate files was kind of useful at one point.

One recommendation would be to leave files that are purely notes out of
your agenda. I noticed from your blog posts that you are adding
timestamps to events from your novel. Are these active or inactive
timestamps? My guess is that parsing all that data is slowing org-mode
down. You can always create a timeline from within a particular file
without including it in your normal agenda files. You can also create
custom agenda views to work on different sets of org files.

http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php

While org-mode is exceptionally robust as a plain text tool, if you need
to manage an ever-growing collection of thousands and thousands of
pieces of data, you'll probably be better served by 1) using a database
or 2) dividing your org-mode files into separate "collections" (i.e.,
reserving your regular agenda for only a subset of org files).

I keep my org files lean by regularly archiving subtrees. I can always
search the archives when I need to.

> so thought 2) would be to have it only attempt to construct background
> agendas when the system is otherwise idle for a few minutes. I don't
> know how to do that, I figure wrapping this bit with something that
> could detect idleness instead of just running arbitrarily would be good.
>
>   (run-at-time nil 3600 'org-agenda-to-appt)
>
> don't know how to detect idleness.

http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html

Best,
Matt

Re: Scaling org-mode

[Nick Dokos]

Dave Täht <d@teklibre.org> wrote:

> ...
> so thought 2) would be to have it only attempt to construct background
> agendas when the system is otherwise idle for a few minutes. I don't
> know how to do that, I figure wrapping this bit with something that
> could detect idleness instead of just running arbitrarily would be good.
> 
>   (run-at-time nil 3600 'org-agenda-to-appt)
> 
> don't know how to detect idleness.
> 

[Apologies if this is off-topic - I haven't read your message carefully,
as I was catching up on email between jobs.]

Not sure whether it'll work any better, but there is run-with-idle-timer.
C-h f run-with-idle-timer <RET> says:

,----
| run-with-idle-timer is an interactive compiled Lisp function in
| `timer.el'.
| 
| (run-with-idle-timer SECS REPEAT FUNCTION &rest ARGS)
| 
| Perform an action the next time Emacs is idle for SECS seconds.
| The action is to call FUNCTION with arguments ARGS.
| SECS may be an integer, a floating point number, or the internal
| time format (HIGH LOW USECS) returned by, e.g., `current-idle-time'.
| If Emacs is currently idle, and has been idle for N seconds (N < SECS),
| then it will call FUNCTION in SECS - N seconds from now.
| 
| If REPEAT is non-nil, do the action each time Emacs has been idle for
| exactly SECS seconds (that is, only once for each time Emacs becomes idle).
| 
| This function returns a timer object which you can use in `cancel-timer'.
`----


HTH,
Nick

Re: Scaling org-mode

[Dave Täht]

Matt Lundin <mdl@imapmail.org> writes:

> d@teklibre.org (Dave Täht) writes:
>
>> I have really been enjoying importing my life into org-mode, which I've
>> been doing for about two months now. 
>>
>> But.
>>
>> It currently visits about 100 files and 10k of text to construct the
>> agenda. It's starting to get kind of slow and interrupt my workflow,
>> particularly the background process that scans them. 
>>
>> While the system is effectively frozen, my message buffer fills up with
>> messages about setting the flyspell dictionary to en, etc. This is quite
>> annoying with text to speech turned on. I ended up just having appts
>> spoken.
>
> My guess is that this is a flyspell problem. I only get messages about
> flyspell if I don't have aspell or the proper dictionary installed. Do
> you get these messages when you open buffers in other text modes? What

Yes. Opening up a random text file gives me:

"Local Ispell dictionary set to en" in the message buffer. It's not
annoying when opening a single file. I can hack ispell.el to suppress
the message, but I think the core problem is more starting up the aspell
process for 100+ files. Can't tell without profiling. 

When org is running in the background, I actually get two messages,
alternating, that and... Darn it, I can't get it to happen right now.

> exactly are the messages? What is your flyspell config?

It is using aspell.

I work in both spanish and english, mostly english.

;; Useful because my LANG variable is usually set to es_ES.UTF-8
;; Maybe there is another default dictionary I could set or a better way
;; to suppress the message (like, maybe suppressing the language
;; variable somewhere else), or doing this in ispell rather than flyspell

(setq flyspell-default-dictionary "en")

;; Relevant hooks
;; Maybe find something that suppresses all these hooks until the buffer
;; is actually displayed on screen would help.

(dolist (hook '(text-mode-hook))
  (add-hook hook (lambda () (flyspell-mode 1))))

(dolist (hook '(erc-mode-hook
		emacs-lisp-mode-hook
		text-mode-hook))
		(add-hook hook (lambda () (abbrev-mode 1))
		(add-hook hook (lambda () (auto-capitalization-mode 1))
))



>
>> Solution #1) cut the number of files down - is a good one. I probably
>> can cut those files easily in half right now. The problem is that I have
>> about 600 more files to import (scenes from a book), and I really like
>> the idea of being able to know what my characters are doing in 2023, and
>> separate files was kind of useful at one point.
>
> One recommendation would be to leave files that are purely notes out of
> your agenda. I noticed from your blog posts that you are adding
> timestamps to events from your novel. Are these active or inactive
> timestamps? My guess is that parsing all that data is slowing org-mode
> down. You can always create a timeline from within a particular file
> without including it in your normal agenda files. You can also create
> custom agenda views to work on different sets of org files.

Thank you for reading! I haven't got further than wanting to use
embedded logic in the book for various things like calculating relative
delta-v, I'd like to sort out some more simple stuff first (like
blogging and managing my life. I'm going to get off of blogger, that's
certain, but moving to what (and getting pretty html output) remains an
open question. 

Some percentage of 898 blog posts to convert, too. Sigh.

I'll want to keep draft blog posts in the agenda.

> http://orgmode.org/worg/org-tutorials/org-custom-agenda-commands.php

Reading... Brain dumping core... 

>
> While org-mode is exceptionally robust as a plain text tool, if you need
> to manage an ever-growing collection of thousands and thousands of
> pieces of data, you'll probably be better served by 1) using a database

Heck, no, that's what gobs of memory is for! Emacs is, for once, not the
biggest process on my system, even with over a hundred buffers loaded.

Partial top listing:

 764 d         20   0  663m 184m  13m S  0.0 19.2  10:56.43 firefox     
 3964 d         20   0  343m 164m 9112 S  0.0 17.0  11:32.16 emacs     

:)

I have tried to organize my personal collection of messes before using
everything from a zillion text files, to database based stuff like
evolution, to writing my own XML outliner, to MS-project. All fell down
somewhere.

What works best is an outliner. I really, really, really missed
MORE. Org is better than MORE in almost every way. Yep, I'm willing to
adapt to how it works and adapt it to how I work. 


> or 2) dividing your org-mode files into separate "collections" (i.e.,
> reserving your regular agenda for only a subset of org files).

That strikes me as a good option, but what I'll want to do is add the
book-mode collection of files to the normal org-mode files when in
"book-mode", so I can stay on task.

I think learning how to profile emacs is moving higher up on my list of
smart things to learn.

>
> I keep my org files lean by regularly archiving subtrees. I can always
> search the archives when I need to.

I am going to be doing that. My current issue with archiving is that I'd
like a view that shows my DONE, non-repeating tasks, so I can make sure
I've scheduled the next step for all of them before I archive them, and
also have a report of what I did for the month that I can review later.

(Maybe one exists, I'll go re-read the doc, I remember reading about it...)

I LOVE seeing the DONE tasks, it gives me a sense of accomplishment out of
a chaotic month, but yea, they need to go OOSOOM as soon as I figure out
the above.

LOTS of DONE tasks in the last two months. I feel productive.

It would be kind of cool to be able to "explode" a set of agenda items
into visible on-screen buffers.

>> so thought 2) would be to have it only attempt to construct background
>> agendas when the system is otherwise idle for a few minutes. I don't
>> know how to do that, I figure wrapping this bit with something that
>> could detect idleness instead of just running arbitrarily would be good.
>>
>>   (run-at-time nil 3600 'org-agenda-to-appt)
>>
>> don't know how to detect idleness.
>
> http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html

Bueno. Writing that bit of code right now. Hmmm... Maybe I can tie this into erc
which also has annoying blocking behavior on netsplits and lost connectivity.

So much depth to Emacs!

Thanks for the thoughts!

-- 
Dave Taht
http://the-edge.blogspot.com

Re: Re: Scaling org-mode

[Dave Täht]

d@teklibre.org (Dave Täht) writes:

> Matt Lundin <mdl@imapmail.org> writes:
>
>> d@teklibre.org (Dave Täht) writes:
>
>>> so thought 2) would be to have it only attempt to construct background
>>> agendas when the system is otherwise idle for a few minutes. I don't
>>> know how to do that, I figure wrapping this bit with something that
>>> could detect idleness instead of just running arbitrarily would be good.
>>>
>>>   (run-at-time nil 3600 'org-agenda-to-appt)
>>>
>>> don't know how to detect idleness.
>>
>> http://www.gnu.org/software/emacs/elisp/html_node/Idle-Timers.html
>
> Bueno. Writing that bit of code right now. Hmmm... Maybe I can tie this into erc
> which also has annoying blocking behavior on netsplits and lost connectivity.

First cut at making org more idle friendly.

It appears to be working... but I've been typing too much to really
notice. :)

See:

http://pastebin.com/dbb75ca

for details. I have not written much emacs lisp for years, so review
would be nice.

I changed it to actually try to reconstruct the agenda every 10 minutes
in this kinder, gentler fashion.

Still have a bit more to do but this takes out the main problem.


-- 
Dave Taht
http://the-edge.blogspot.com

Productiviy tools (was: Scaling org-mode)

[Daniel Clemente]

Off-topic.

El dom, sep 13 2009 a les 07:45, Dave Täht va escriure:
> ;; my personal fav, run every 15 minutes
>
> (defun nag-timer () "Nag me when there isn't a clock running"  
>   (interactive)
>   (unless (marker-buffer org-clock-marker)
>     (say "Are you mating now?")))
>

  I like this very much and have started using it; let's see how annoying it can be.

  Do you really clock all the time you have Emacs open? That will give very complete statistics about daily computer usage… if only you don't end up clocking everything into a general task „* do some things“.


  I have since long thought of more utilities like this, which watch my work habits and help me correct them in the ways I defined beforehand. It would be something like my org-boss and include:

- warn when I'm not clocking anything (possibly do this only on work hours, not at home)
- check that each work day I work the hours I should, no less
- warn when some tasks or deadlines start to seem difficult to complete on time:
 - e.g. if there are still 30 predicted hours but the deadline is tomorrow (so you won't be able to do those 30 hours)
 - or if I am being too slow (e.g. if after 1h working at a 4h task I am still at 10%. To be on schedule I should have been at 25%)
- motivate me positively when I complete tasks faster than planned
- help me find the effort estimates which proved wrong (because I spent more time than planned)
- warn when I have too many scheduled tasks for today in my agenda (I should reschedule them)
- complain if I have many same-level tasks and I haven't assigned priorities to them
- complain if I hadn't estimated the effort of task which has taken a lot of time
- …

  I see there is much work to do. Many productivity improvements are personal, so a single mode can't match all corrective needs. A single file with a collection of working functions would be better; then users can adapt to their needs the functions they want.


  How does this utopia sound?
  I alone can't develop this in time, but: if we put a file in Worg or contrib/, could we collect all our productiviy improvement tools and ideas?


-- Daniel

Re: Productiviy tools

[Dave Täht]

Daniel Clemente <n142857@gmail.com> writes:

> Off-topic.
>
> El dom, sep 13 2009 a les 07:45, Dave Täht va escriure:
>> ;; my personal fav, run every 15 minutes
>>
>> (defun nag-timer () "Nag me when there isn't a clock running"  
>>   (interactive)
>>   (unless (marker-buffer org-clock-marker)
>>     (say "Are you mating now?")))
>>
>
>   I like this very much and have started using it; let's see how
>   annoying it can be.

Well, say is a wrapper around a call to the cepstral speech synth, it
used to talk to espeak. Both are having grave difficulties sending stuff
over ESD to my nokia 770 with the latest pulseaudio in ubuntu 9.04. I
really like using ESD. I can listen to good quality music on that handheld,
AND still have various announcements play over the music. I can wander the
house on headphones and still function. 

I'm about ready to dump pulseaudio and write a wrapper for cepstral to
send stuff directly to esd with no intermediaries, and/or tie cepstral
into espeak properly.

I am not big on visual, pop-up reminders, or things that beep or
bong. You really can (with procmail + ssml) convert the old "You have
mail" Aol thing into "You have mail from John Doe", if you want. 

>   Do you really clock all the time you have Emacs open? That will give
>   very complete statistics about daily computer usage… if only you
>   don't end up clocking everything into a general task „* do some
>   things“.

Not yet, but I'm getting better at it. I had to move more and more of my
life into emacs to be able to do it, and the big piece missing is
tracking web usage. I have really begun to hate the default black on
white display of most web pages, and am considering adopting conqueror
I'd like to have that general task "do some things" be a bit bucket for
a daily:

"You spent X time editing Y file"
"You spent X time in an erc window"
"You spent X time in email" (actually, finer grained would be good)
"You spent X time not on the clock"

Which I could then re-allocate over my current projects, but am very
unsure as to how to go about it. I've seen a few tools that track
Xwindow usage....

>   I have since long thought of more utilities like this, which watch
>   my work habits and help me correct them in the ways I defined
>   beforehand. It would be something like my org-boss and include:

I like to think of the voices I use as virtual secretaries, not
bosses. They work for me, not vice versa. If I ever abstract the
functionalities of the various say scripts enough, I'll end up calling
the package "majel" after Majel Barret and come up with some suitable
backronym.

Majel would gather up stuff overnight and tell me in the morning that
VIPs Bob, Doc, and Joe sent me mail, that the surf is up, and winds
offshore, and yesterday I didn't do a drop of billable work, and tell me
a joke, and tomorrow I have to pay some bills, while I blurrily consumed
coffee...

> - warn when I'm not clocking anything (possibly do this only on work
> hours, not at home) 

Yes, my previous mail here has the start at an attempt to divide up work
and play more than I do currently. Don't want to be nagged 24 hours a day.

> - check that each work day I work the hours I should, no less

Heh, for me, I also need reminders to NOT work MORE hours than I should,
and make time for the fun stuff, like playing music and surfing. I
really have to get the tide schedule into this thing somehow.... 

>- warn when some tasks or deadlines start to seem
> difficult to complete on time: - e.g. if there are still 30 predicted
> hours but the deadline is tomorrow (so you won't be able to do those
> 30 hours) - or if I am being too slow (e.g. if after 1h working at a
> 4h task I am still at 10%. To be on schedule I should have been at
> 25%) 

Related to that I have a backlog of tasks that I would like to bulk
reschedule so I stop seeing them. I have about 25, now, and I'm moving
next week, so I'm just not going to get them done and have to push them
all into the future somehow.

Also related to that would be some sort of ms-project-like load leveler,
where I could see that I had 100 hrs of stuff scheduled for next week
and have an easier way of sorting it out.

Similarly, a bulk estimator to mark a set of tasks and put in estimates
for them all, fast.

(I'm pretty sure I can do this last already. Again, I'm only 2 months
into org-mode, many things, like publishing, remain a mystery. I had an
abortive experience with blorg this afternoon, for example. I liked what
I saw of blorg, didn't realize it had been discontinued, don't want to
run ruby for a blog server, and haven't figured out if org-publish does
enough of what I want)

> - motivate me positively when I complete tasks faster than
> planned - help me find the effort estimates which proved wrong
> (because I spent more time than planned) - warn when I have too many
> scheduled tasks for today in my agenda (I should reschedule them) -
> complain if I have many same-level tasks and I haven't assigned
> priorities to them - complain if I hadn't estimated the effort of task
> which has taken a lot of time - …

All good things. A few more priority levels than 3 would be good. Probably.

>
>   I see there is much work to do. Many productivity improvements are
>   personal, so a single mode can't match all corrective needs. A
>   single file with a collection of working functions would be better;
>   then users can adapt to their needs the functions they want.
>
>
>   How does this utopia sound?  I alone can't develop this in time,
>   but: if we put a file in Worg or contrib/, could we collect all our
>   productiviy improvement tools and ideas?

Sure. At the moment though, the only big thing I was thinking of doing
was full blown SSML (speech synthesis markup language) support for org,
based on the existing html mode, using emphasis to handle various levels
of ***** and using paragraph breaks, and eliding [[][]] stuff more
correctly than what I have now.

>
>
> -- Daniel
>

-- 
Dave Taht
http://the-edge.blogspot.com