From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adam Porter Subject: Re: How to make agenda generation faster Date: Tue, 16 Oct 2018 15:35:56 -0500 Message-ID: <87ftx5fx3n.fsf@alphapapa.net> References: <87h8hy1ho5.fsf@mbork.pl> <87o9c34ocl.fsf@alphapapa.net> <877eirdrqt.fsf@nicolasgoaziou.fr> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:55798) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCW4d-0002ry-68 for emacs-orgmode@gnu.org; Tue, 16 Oct 2018 16:36:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCW4Y-0000wp-5B for emacs-orgmode@gnu.org; Tue, 16 Oct 2018 16:36:15 -0400 Received: from [195.159.176.226] (port=49887 helo=blaine.gmane.org) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gCW4X-0000ty-S8 for emacs-orgmode@gnu.org; Tue, 16 Oct 2018 16:36:10 -0400 Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1gCW2O-0007aQ-2k for emacs-orgmode@gnu.org; Tue, 16 Oct 2018 22:33:56 +0200 List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: emacs-orgmode@gnu.org Nicolas Goaziou writes: >> my understanding is that code that runs with lexical-binding enabled >> is generally faster. > > Not really. But it's certainly easier to understand since it removes one > class of problems. >From what I've read, the byte-compiler can optimize better when lexical-binding is used. > Instead of re-inventing the wheel, or putting efforts into a > wheel-like invention, wouldn't it make sense to actually work on Org > Agenda itself? > > So again, wouldn't it be nice to think about Org Agenda-ng? As a matter of fact, what's now called org-ql-agenda was originally called org-agenda-ng. I factored org-ql out of it and realized that it should probably be its own, standalone package. Then I renamed org-agenda-ng to org-ql-agenda, so I could reasonably keep them in the same repo, and because I don't know if I will ever develop it far enough to be worthy of the name org-agenda-ng. It started as an experiment to build a foundation for a new, modular agenda implementation, and maybe it could be. > I didn't look closely at org-ql, but I had the idea of splitting the > Agenda in two distinct parts. One would be responsible for collecting, > possibly asynchronously, and caching data from Org documents. The other > one would provide a DSL to query and display the results extracted from > the output of the first part. The second part could even be made generic > enough to be extracted from Org and become some part of Emacs. > Displaying filtered data, maybe in a timeline, could be useful for other > packages. Unfortunately, I don't have time to work on this. Ah well. I've thought about this for a while. It seems to me that the issue is that Org buffers are, of course, plain-text buffers. There is no persistent, in-memory representation other than the buffer, so whenever Org needs structured/semantic data, it must parse it out of the buffer, which is necessarily rather slow. If there were a way to keep an outline tree in memory, parallel to the buffer itself, that would allow operations like search, agenda, etc. to be greatly sped up. But how would that work in Emacs? Theoretically, we could write some code, applied on self-insert-command, to update the "parallel tree structure" as the user manipulates the plain-text in the buffer (e.g. add a new node when the user types a "*" to create a new heading), and also apply it to functions that manipulate the outline structurally in the buffer. But, of course, that sounds very complicated. I would not relish the idea of debugging code to keep a cached tree in sync with a plain-text buffer outline. :) Besides that, AFAIK there would be no way to do it asynchronously other than calling out to a child Emacs process (because elisp is still single-threaded), printing and reading the data back and forth (which would tie up the parent process when reading). Maybe in the future elisp will be multithreaded... Anyway, org-ql tries to do some of what you mentioned. It does rudimentary, per-buffer, per-query caching (as long as the buffer is not modified, the cache remains valid), which helps when there are several Org files open that are referred to often but not as often modified. And the query and presentation code are separated (org-ql and org-ql-agenda). I don't know how widely it's used, but the repo is getting some regular traffic, and I'm using it as the backend for my org-sidebar package. I'd be happy if it could be made more generally useful, or if it could be helpful to Org itself in some way. Contributions are welcome.