From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nicolas Goaziou Subject: Re: How to make agenda generation faster Date: Wed, 17 Oct 2018 15:01:50 +0200 Message-ID: <87pnw8engh.fsf@nicolasgoaziou.fr> References: <87h8hy1ho5.fsf@mbork.pl> <87o9c34ocl.fsf@alphapapa.net> <877eirdrqt.fsf@nicolasgoaziou.fr> <87ftx5fx3n.fsf@alphapapa.net> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:54018) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gClSr-0008FF-Jg for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 09:02:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gClSi-0006wB-ES for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 09:02:15 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:48157) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gClSd-0006KL-7b for emacs-orgmode@gnu.org; Wed, 17 Oct 2018 09:02:05 -0400 In-Reply-To: <87ftx5fx3n.fsf@alphapapa.net> (Adam Porter's message of "Tue, 16 Oct 2018 15:35:56 -0500") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Adam Porter Cc: emacs-orgmode@gnu.org Hello, Adam Porter writes: > From what I've read, the byte-compiler can optimize better when > lexical-binding is used. It can, but AFAIK, it doesn't yet. It also means un-optimized lexical binding may be slightly slower than dynamic scoping for the time being. > I've thought about this for a while. It seems to me that the issue is > that Org buffers are, of course, plain-text buffers. There is no > persistent, in-memory representation other than the buffer, so whenever > Org needs structured/semantic data, it must parse it out of the buffer, > which is necessarily rather slow. If there were a way to keep an > outline tree in memory, parallel to the buffer itself, that would allow > operations like search, agenda, etc. to be greatly sped up. I don't think that's necessary. File caching as you suggest below, can go a long way. Filling cache during idle time, too. > But how would that work in Emacs? Theoretically, we could write some > code, applied on self-insert-command, to update the "parallel tree > structure" as the user manipulates the plain-text in the buffer > (e.g. add a new node when the user types a "*" to create a new heading), > and also apply it to functions that manipulate the outline structurally > in the buffer. But, of course, that sounds very complicated. I would > not relish the idea of debugging code to keep a cached tree in sync with > a plain-text buffer outline. :) My over-engineering-o-meter flashes red, too. > Anyway, org-ql tries to do some of what you mentioned. It does > rudimentary, per-buffer, per-query caching (as long as the buffer is not > modified, the cache remains valid), which helps when there are several > Org files open that are referred to often but not as often modified. That's what I did in an agenda upgrade I tried a few months ago. Unfortunately, caching is not compatible with the underlying logic of current Agenda, in particular with `org-agenda-skip-function'. > And the query and presentation code are separated (org-ql and > org-ql-agenda). That's a very good thing. > I don't know how widely it's used, but the repo is getting some regular > traffic, and I'm using it as the backend for my org-sidebar package. > I'd be happy if it could be made more generally useful, or if it could > be helpful to Org itself in some way. Contributions are welcome. That's not exactly what I'm suggesting. I suggest to move the work in Org tree, e.g., as an org-agenda-ng.el library, and, from there, implement back most of the features of the current agenda. Org cannot really benefit from libraries living outside Emacs, as we recently learnt with htmlize issue. Regards, -- Nicolas Goaziou