From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ihor Radchenko Subject: Re: How to make agenda generation faster Date: Wed, 17 Oct 2018 15:04:58 +0800 Message-ID: <874ldlt5np.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> References: <87h8hy1ho5.fsf@mbork.pl> <87o9c34ocl.fsf@alphapapa.net> <877eirdrqt.fsf@nicolasgoaziou.fr> <87ftx5fx3n.fsf@alphapapa.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:55972) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCfum-0004Nk-GM for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 03:06:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCfuj-0006Oy-BW for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 03:06:44 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]:35775) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gCfuj-0006Jv-27 for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 03:06:41 -0400 Received: by mail-pl1-x636.google.com with SMTP id f8-v6so12242648plb.2 for ; Wed, 17 Oct 2018 00:06:40 -0700 (PDT) In-Reply-To: <87ftx5fx3n.fsf@alphapapa.net> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Adam Porter , emacs-orgmode@gnu.org --=-=-= Content-Type: text/plain > I've thought about this for a while. It seems to me that the issue is > that Org buffers are, of course, plain-text buffers. There is no > persistent, in-memory representation other than the buffer, so whenever > Org needs structured/semantic data, it must parse it out of the buffer, > which is necessarily rather slow. If there were a way to keep an > outline tree in memory, parallel to the buffer itself, that would allow > operations like search, agenda, etc. to be greatly sped up. FYI A while ago I saw some cache implementation in org-element.el. Take a look at org-element--cache variable definition and the code below. ```` (defvar org-element--cache nil "AVL tree used to cache elements. Each node of the tree contains an element. Comparison is done with `org-element--cache-compare'. This cache is used in `org-element-at-point'.") ```` Best, Ihor Adam Porter writes: > Nicolas Goaziou writes: > >>> my understanding is that code that runs with lexical-binding enabled >>> is generally faster. >> >> Not really. But it's certainly easier to understand since it removes one >> class of problems. > > From what I've read, the byte-compiler can optimize better when > lexical-binding is used. > >> Instead of re-inventing the wheel, or putting efforts into a >> wheel-like invention, wouldn't it make sense to actually work on Org >> Agenda itself? >> >> So again, wouldn't it be nice to think about Org Agenda-ng? > > As a matter of fact, what's now called org-ql-agenda was originally > called org-agenda-ng. I factored org-ql out of it and realized that it > should probably be its own, standalone package. Then I renamed > org-agenda-ng to org-ql-agenda, so I could reasonably keep them in the > same repo, and because I don't know if I will ever develop it far enough > to be worthy of the name org-agenda-ng. It started as an experiment to > build a foundation for a new, modular agenda implementation, and maybe > it could be. > >> I didn't look closely at org-ql, but I had the idea of splitting the >> Agenda in two distinct parts. One would be responsible for collecting, >> possibly asynchronously, and caching data from Org documents. The other >> one would provide a DSL to query and display the results extracted from >> the output of the first part. The second part could even be made generic >> enough to be extracted from Org and become some part of Emacs. >> Displaying filtered data, maybe in a timeline, could be useful for other >> packages. Unfortunately, I don't have time to work on this. Ah well. > > I've thought about this for a while. It seems to me that the issue is > that Org buffers are, of course, plain-text buffers. There is no > persistent, in-memory representation other than the buffer, so whenever > Org needs structured/semantic data, it must parse it out of the buffer, > which is necessarily rather slow. If there were a way to keep an > outline tree in memory, parallel to the buffer itself, that would allow > operations like search, agenda, etc. to be greatly sped up. > > But how would that work in Emacs? Theoretically, we could write some > code, applied on self-insert-command, to update the "parallel tree > structure" as the user manipulates the plain-text in the buffer > (e.g. add a new node when the user types a "*" to create a new heading), > and also apply it to functions that manipulate the outline structurally > in the buffer. But, of course, that sounds very complicated. I would > not relish the idea of debugging code to keep a cached tree in sync with > a plain-text buffer outline. :) > > Besides that, AFAIK there would be no way to do it asynchronously other > than calling out to a child Emacs process (because elisp is still > single-threaded), printing and reading the data back and forth (which > would tie up the parent process when reading). Maybe in the future > elisp will be multithreaded... > > Anyway, org-ql tries to do some of what you mentioned. It does > rudimentary, per-buffer, per-query caching (as long as the buffer is not > modified, the cache remains valid), which helps when there are several > Org files open that are referred to often but not as often modified. > And the query and presentation code are separated (org-ql and > org-ql-agenda). > > I don't know how widely it's used, but the repo is getting some regular > traffic, and I'm using it as the backend for my org-sidebar package. > I'd be happy if it could be made more generally useful, or if it could > be helpful to Org itself in some way. Contributions are welcome. > > --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEERZAHPFbUe3JemmzmZHB2Kn2hHYsFAlvG3xoACgkQZHB2Kn2h HYsmaAgAsjAf4T5opK3V9zJ+q4ypFd8vWAL7Fwpa7cbXFpICkea9aJTxgo++vSO+ hHBQs/Uvh+d3fR6rAo1vHnE8Lb/VcNU6vEW7CvfqLX6vx51AWrs4tObaPkLdUeM7 6iZn9h2tZKcHT/i/VfSeK/YMTmgTvWGMK6V9FnKGKGjmW9v20KLfLftIdKG5/bYx aN9jaLi5v4uPj0PTMjEH7IScMiFvZqPane/H/vbPLhaMGHd8JStQ7ajK1bIp4spx 4zOLHDxYQpHwPVYKMIfD7jA1sb446RFFY0y+OLRKmR70KLcq5iI96oH77u7/V7yN RqgnGo1tPfSuFg5oXuea9NEehM5uCA== =yIft -----END PGP SIGNATURE----- --=-=-=--