Re: How to make agenda generation faster

[Nicolas Goaziou <mail@nicolasgoaziou.fr>]

Hello,

Adam Porter <adam@alphapapa.net> writes:

> From what I've read, the byte-compiler can optimize better when
> lexical-binding is used.

It can, but AFAIK, it doesn't yet. It also means un-optimized lexical
binding may be slightly slower than dynamic scoping for the time being.

> I've thought about this for a while.  It seems to me that the issue is
> that Org buffers are, of course, plain-text buffers.  There is no
> persistent, in-memory representation other than the buffer, so whenever
> Org needs structured/semantic data, it must parse it out of the buffer,
> which is necessarily rather slow.  If there were a way to keep an
> outline tree in memory, parallel to the buffer itself, that would allow
> operations like search, agenda, etc. to be greatly sped up.

I don't think that's necessary. File caching as you suggest below, can
go a long way. Filling cache during idle time, too.

> But how would that work in Emacs?  Theoretically, we could write some
> code, applied on self-insert-command, to update the "parallel tree
> structure" as the user manipulates the plain-text in the buffer
> (e.g. add a new node when the user types a "*" to create a new heading),
> and also apply it to functions that manipulate the outline structurally
> in the buffer.  But, of course, that sounds very complicated.  I would
> not relish the idea of debugging code to keep a cached tree in sync with
> a plain-text buffer outline.  :)

My over-engineering-o-meter flashes red, too.

> Anyway, org-ql tries to do some of what you mentioned.  It does
> rudimentary, per-buffer, per-query caching (as long as the buffer is not
> modified, the cache remains valid), which helps when there are several
> Org files open that are referred to often but not as often modified.

That's what I did in an agenda upgrade I tried a few months ago.
Unfortunately, caching is not compatible with the underlying logic of
current Agenda, in particular with `org-agenda-skip-function'.

> And the query and presentation code are separated (org-ql and
> org-ql-agenda).

That's a very good thing.

> I don't know how widely it's used, but the repo is getting some regular
> traffic, and I'm using it as the backend for my org-sidebar package.
> I'd be happy if it could be made more generally useful, or if it could
> be helpful to Org itself in some way.  Contributions are welcome.

That's not exactly what I'm suggesting. I suggest to move the work in
Org tree, e.g., as an org-agenda-ng.el library, and, from there,
implement back most of the features of the current agenda. 

Org cannot really benefit from libraries living outside Emacs, as we
recently learnt with htmlize issue.

Regards,

-- 
Nicolas Goaziou