Re: How to make agenda generation faster

[Adam Porter <adam@alphapapa.net>]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> my understanding is that code that runs with lexical-binding enabled
>> is generally faster.
>
> Not really. But it's certainly easier to understand since it removes one
> class of problems.

From what I've read, the byte-compiler can optimize better when
lexical-binding is used.

> Instead of re-inventing the wheel, or putting efforts into a
> wheel-like invention, wouldn't it make sense to actually work on Org
> Agenda itself?
>
> So again, wouldn't it be nice to think about Org Agenda-ng?

As a matter of fact, what's now called org-ql-agenda was originally
called org-agenda-ng.  I factored org-ql out of it and realized that it
should probably be its own, standalone package.  Then I renamed
org-agenda-ng to org-ql-agenda, so I could reasonably keep them in the
same repo, and because I don't know if I will ever develop it far enough
to be worthy of the name org-agenda-ng.  It started as an experiment to
build a foundation for a new, modular agenda implementation, and maybe
it could be.

> I didn't look closely at org-ql, but I had the idea of splitting the
> Agenda in two distinct parts. One would be responsible for collecting,
> possibly asynchronously, and caching data from Org documents. The other
> one would provide a DSL to query and display the results extracted from
> the output of the first part. The second part could even be made generic
> enough to be extracted from Org and become some part of Emacs.
> Displaying filtered data, maybe in a timeline, could be useful for other
> packages. Unfortunately, I don't have time to work on this. Ah well.

I've thought about this for a while.  It seems to me that the issue is
that Org buffers are, of course, plain-text buffers.  There is no
persistent, in-memory representation other than the buffer, so whenever
Org needs structured/semantic data, it must parse it out of the buffer,
which is necessarily rather slow.  If there were a way to keep an
outline tree in memory, parallel to the buffer itself, that would allow
operations like search, agenda, etc. to be greatly sped up.

But how would that work in Emacs?  Theoretically, we could write some
code, applied on self-insert-command, to update the "parallel tree
structure" as the user manipulates the plain-text in the buffer
(e.g. add a new node when the user types a "*" to create a new heading),
and also apply it to functions that manipulate the outline structurally
in the buffer.  But, of course, that sounds very complicated.  I would
not relish the idea of debugging code to keep a cached tree in sync with
a plain-text buffer outline.  :)

Besides that, AFAIK there would be no way to do it asynchronously other
than calling out to a child Emacs process (because elisp is still
single-threaded), printing and reading the data back and forth (which
would tie up the parent process when reading).  Maybe in the future
elisp will be multithreaded...

Anyway, org-ql tries to do some of what you mentioned.  It does
rudimentary, per-buffer, per-query caching (as long as the buffer is not
modified, the cache remains valid), which helps when there are several
Org files open that are referred to often but not as often modified.
And the query and presentation code are separated (org-ql and
org-ql-agenda).

I don't know how widely it's used, but the repo is getting some regular
traffic, and I'm using it as the backend for my org-sidebar package.
I'd be happy if it could be made more generally useful, or if it could
be helpful to Org itself in some way.  Contributions are welcome.