Re: How to make agenda generation faster

[Adam Porter <adam@alphapapa.net>]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> It can, but AFAIK, it doesn't yet. It also means un-optimized lexical
> binding may be slightly slower than dynamic scoping for the time
> being.

Well, I can't vouch for it myself, because I haven't studied the code.
But here's one of the resources that suggests it is faster to use
lexical binding:

https://emacs.stackexchange.com/questions/2129/why-is-let-faster-with-lexical-scope

> That's not exactly what I'm suggesting. I suggest to move the work in
> Org tree, e.g., as an org-agenda-ng.el library, and, from there,
> implement back most of the features of the current agenda.
>
> Org cannot really benefit from libraries living outside Emacs, as we
> recently learnt with htmlize issue.

Org is welcome to take any of the org-ql or org-ql-agenda code you think
would be useful.

However, before it could be suitable as a possible replacement, it will
likely require more optimization.  Some queries, especially more complex
ones, are slower than the equivalent searches and agendas in the current
Org Agenda code.  This is because of the way the queries run predicates
on each heading.  Despite the current Org Agenda code's complexity, it
is well optimized and hard to beat.

I have a proof-of-concept branch that begins to implement a relatively
simple optimization that converts one suitable predicate in a query to a
buffer-global regexp search.  It significantly improves speed in some
cases, but a query with several predicates still has to run all but one
of them as predicates.

Another possible optimization would be to convert as many predicates in
a query to buffer regexp searches as possible, collecting a list of
heading positions in the buffer, and then do a final pass with the
appropriate union/intersection/difference operations on the lists.  Then
the list of positions could be used to gather the heading data.  I use a
similar technique in helm-org-rifle, and it seems to work quickly.  It
would require some work on a sort of "query compiler" to do the
transformation and optimization.  I don't have much experience with that
kind of programming; maybe someone else would be interested in helping
with that.

So before taking any of the code into Org itself, you might want to
consider these issues and decide whether it could be a suitable
approach.  Let me know what you'd like to do and how I can help.

Thanks,
Adam