From: Ihor Radchenko <yantar92@gmail.com>
To: Adam Porter <adam@alphapapa.net>, emacs-orgmode@gnu.org
Subject: Re: How to make agenda generation faster
Date: Wed, 17 Oct 2018 15:04:58 +0800 [thread overview]
Message-ID: <874ldlt5np.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> (raw)
In-Reply-To: <87ftx5fx3n.fsf@alphapapa.net>
[-- Attachment #1: Type: text/plain, Size: 4626 bytes --]
> I've thought about this for a while. It seems to me that the issue is
> that Org buffers are, of course, plain-text buffers. There is no
> persistent, in-memory representation other than the buffer, so whenever
> Org needs structured/semantic data, it must parse it out of the buffer,
> which is necessarily rather slow. If there were a way to keep an
> outline tree in memory, parallel to the buffer itself, that would allow
> operations like search, agenda, etc. to be greatly sped up.
FYI
A while ago I saw some cache implementation in org-element.el.
Take a look at org-element--cache variable definition and the code
below.
````
(defvar org-element--cache nil
"AVL tree used to cache elements.
Each node of the tree contains an element. Comparison is done
with `org-element--cache-compare'. This cache is used in
`org-element-at-point'.")
````
Best,
Ihor
Adam Porter <adam@alphapapa.net> writes:
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>>> my understanding is that code that runs with lexical-binding enabled
>>> is generally faster.
>>
>> Not really. But it's certainly easier to understand since it removes one
>> class of problems.
>
> From what I've read, the byte-compiler can optimize better when
> lexical-binding is used.
>
>> Instead of re-inventing the wheel, or putting efforts into a
>> wheel-like invention, wouldn't it make sense to actually work on Org
>> Agenda itself?
>>
>> So again, wouldn't it be nice to think about Org Agenda-ng?
>
> As a matter of fact, what's now called org-ql-agenda was originally
> called org-agenda-ng. I factored org-ql out of it and realized that it
> should probably be its own, standalone package. Then I renamed
> org-agenda-ng to org-ql-agenda, so I could reasonably keep them in the
> same repo, and because I don't know if I will ever develop it far enough
> to be worthy of the name org-agenda-ng. It started as an experiment to
> build a foundation for a new, modular agenda implementation, and maybe
> it could be.
>
>> I didn't look closely at org-ql, but I had the idea of splitting the
>> Agenda in two distinct parts. One would be responsible for collecting,
>> possibly asynchronously, and caching data from Org documents. The other
>> one would provide a DSL to query and display the results extracted from
>> the output of the first part. The second part could even be made generic
>> enough to be extracted from Org and become some part of Emacs.
>> Displaying filtered data, maybe in a timeline, could be useful for other
>> packages. Unfortunately, I don't have time to work on this. Ah well.
>
> I've thought about this for a while. It seems to me that the issue is
> that Org buffers are, of course, plain-text buffers. There is no
> persistent, in-memory representation other than the buffer, so whenever
> Org needs structured/semantic data, it must parse it out of the buffer,
> which is necessarily rather slow. If there were a way to keep an
> outline tree in memory, parallel to the buffer itself, that would allow
> operations like search, agenda, etc. to be greatly sped up.
>
> But how would that work in Emacs? Theoretically, we could write some
> code, applied on self-insert-command, to update the "parallel tree
> structure" as the user manipulates the plain-text in the buffer
> (e.g. add a new node when the user types a "*" to create a new heading),
> and also apply it to functions that manipulate the outline structurally
> in the buffer. But, of course, that sounds very complicated. I would
> not relish the idea of debugging code to keep a cached tree in sync with
> a plain-text buffer outline. :)
>
> Besides that, AFAIK there would be no way to do it asynchronously other
> than calling out to a child Emacs process (because elisp is still
> single-threaded), printing and reading the data back and forth (which
> would tie up the parent process when reading). Maybe in the future
> elisp will be multithreaded...
>
> Anyway, org-ql tries to do some of what you mentioned. It does
> rudimentary, per-buffer, per-query caching (as long as the buffer is not
> modified, the cache remains valid), which helps when there are several
> Org files open that are referred to often but not as often modified.
> And the query and presentation code are separated (org-ql and
> org-ql-agenda).
>
> I don't know how widely it's used, but the repo is getting some regular
> traffic, and I'm using it as the backend for my org-sidebar package.
> I'd be happy if it could be made more generally useful, or if it could
> be helpful to Org itself in some way. Contributions are welcome.
>
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
next prev parent reply other threads:[~2018-10-17 7:06 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-07 4:53 How to make agenda generation faster Marcin Borkowski
2018-10-08 7:20 ` Michael Welle
2018-10-10 20:03 ` Marcin Borkowski
2018-10-10 21:01 ` Samuel Wales
2018-10-11 6:48 ` Michael Welle
2018-10-11 8:48 ` Marcin Borkowski
2018-10-11 19:59 ` Samuel Wales
2018-10-14 8:51 ` Marcin Borkowski
2018-10-09 6:37 ` Adam Porter
2018-10-09 16:11 ` Nicolas Goaziou
2018-10-10 20:01 ` Marcin Borkowski
2018-10-16 20:35 ` Adam Porter
2018-10-17 7:04 ` Ihor Radchenko [this message]
2018-10-17 13:01 ` Nicolas Goaziou
2018-10-17 19:12 ` Adam Porter
2018-10-18 22:48 ` Nicolas Goaziou
2018-10-19 0:04 ` stardiviner
2018-10-20 2:12 ` Adam Porter
2018-10-20 8:12 ` Nicolas Goaziou
2018-10-10 19:59 ` Marcin Borkowski
2018-10-09 11:47 ` Julius Dittmar
2018-10-10 20:03 ` Marcin Borkowski
2018-10-11 6:40 ` Michael Welle
2018-10-14 7:42 ` Marcin Borkowski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=874ldlt5np.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me \
--to=yantar92@gmail.com \
--cc=adam@alphapapa.net \
--cc=emacs-orgmode@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).