From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matt Lundin Subject: Re: [PATCH] Re: Problems with org publish cache checking Date: Thu, 26 Nov 2015 19:30:40 -0600 Message-ID: <871tbci5z3.fsf@fastmail.fm> References: <87r3jfh1js.fsf@fastmail.fm> <87lh9llcf5.fsf@fastmail.fm> <8737vtjhf2.fsf@nicolasgoaziou.fr> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35549) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a27s6-0002qO-8f for emacs-orgmode@gnu.org; Thu, 26 Nov 2015 20:30:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a27s3-0001co-0x for emacs-orgmode@gnu.org; Thu, 26 Nov 2015 20:30:46 -0500 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:58635) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a27s2-0001cc-T9 for emacs-orgmode@gnu.org; Thu, 26 Nov 2015 20:30:42 -0500 Received: from archthink (c-50-172-132-15.hsd1.il.comcast.net [50.172.132.15]) by mail.messagingengine.com (Postfix) with ESMTPA id CB1A568017A for ; Thu, 26 Nov 2015 20:30:40 -0500 (EST) In-Reply-To: <8737vtjhf2.fsf@nicolasgoaziou.fr> (Nicolas Goaziou's message of "Thu, 26 Nov 2015 09:25:53 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Org Mode --=-=-= Content-Type: text/plain Nicolas Goaziou writes: > Hello, > > Matt Lundin writes: > >> OK, I've worked up a patch that solves several of these issues. The >> basic idea is to check when publishing an org file whether it includes >> other org files and then to store that data in the cache. That way, >> org-publish-cache-file-needs-publishing does not need to open each >> buffer but rather can compare the stored timestamp data against the >> actual modified times of the included files. > > This is much better, indeed. Thank you. > > One suggestion: wouldn't it make sense to also apply check to SETUPFILE > keywords? Yes, that's a great idea. I've added it to the patch. One caveat: this patch does not implement recursive checking of included files (i.e., included files that include other files), but this could be added in the future. Thanks, Matt --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=0001-Speed-up-publishing-by-caching-included-file-data.patch >From cc66884f2836bee1203d06618828c0339ea2e4e2 Mon Sep 17 00:00:00 2001 From: Matt Lundin Date: Thu, 26 Nov 2015 19:22:00 -0600 Subject: [PATCH] Speed up publishing by caching included file data * lisp/ox-publish.el: (org-publish-cache-get-included-files): New function (org-publish-org-to): Use new function (org-publish-cache-file-needs-publishing): Use cache instead of visiting every file in a project. Org-publish can now quickly determine a) whether an org source includes other files (either via #+INCLUDE or #+SETUPFILE) and b) whether those files have changed. This speeds up the publishing process and makes tracking of changes in included files more reliable. --- lisp/ox-publish.el | 70 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el index 90f307c..02eb06d 100644 --- a/lisp/ox-publish.el +++ b/lisp/ox-publish.el @@ -574,6 +574,7 @@ Return output file name." (let ((output-file (org-export-output-file-name extension nil pub-dir)) (body-p (plist-get plist :body-only))) + (when org-publish-cache (org-publish-cache-get-included-files)) (org-export-to-file backend output-file nil nil nil body-p ;; Add `org-publish--collect-references' and @@ -1221,42 +1222,49 @@ If FREE-CACHE, empty the cache." (defun org-publish-cache-file-needs-publishing (filename &optional pub-dir pub-func _base-dir) "Check the timestamp of the last publishing of FILENAME. -Return non-nil if the file needs publishing. Also check if -any included files have been more recently published, so that -the file including them will be republished as well." +Return non-nil if the file needs publishing. Also use the cache +to check if any included files have changed, so that the file +including them will be republished." (unless org-publish-cache (error "`org-publish-cache-file-needs-publishing' called, but no cache present")) - (let* ((case-fold-search t) - (key (org-publish-timestamp-filename filename pub-dir pub-func)) + (let* ((key (org-publish-timestamp-filename filename pub-dir pub-func)) (pstamp (org-publish-cache-get key)) - (org-inhibit-startup t) - (visiting (find-buffer-visiting filename)) - included-files-ctime buf) - (when (equal (file-name-extension filename) "org") - (setq buf (find-file (expand-file-name filename))) - (with-current-buffer buf - (goto-char (point-min)) - (while (re-search-forward "^[ \t]*#\\+INCLUDE:" nil t) - (let* ((element (org-element-at-point)) - (included-file - (and (eq (org-element-type element) 'keyword) - (let ((value (org-element-property :value element))) - (and value - (string-match "^\\(\".+?\"\\|\\S-+\\)" value) - ;; Ignore search suffix. - (car (split-string - (org-remove-double-quotes - (match-string 1 value))))))))) - (when included-file - (push (org-publish-cache-ctime-of-src - (expand-file-name included-file)) - included-files-ctime))))) - (unless visiting (kill-buffer buf))) + (ctime (when pstamp (org-publish-cache-ctime-of-src filename)))) (or (null pstamp) - (let ((ctime (org-publish-cache-ctime-of-src filename))) - (or (< pstamp ctime) - (cl-some (lambda (ct) (< ctime ct)) included-files-ctime)))))) + (< pstamp ctime) + (cl-some (lambda (incl) + ;; See if cached time is before modification time. + (< (cdr incl) + (org-publish-cache-ctime-of-src (car incl)))) + (org-publish-cache-get-file-property filename :includes))))) + +(defun org-publish-cache-get-included-files () + "Get data about included files in current buffer. +Store file names and modification times in cache. Also store data +about setupfiles." + (let ((case-fold-search t) + included) + (save-excursion + (goto-char (point-min)) + (while (re-search-forward "^[ \t]*#\\+\\(INCLUDE\\|SETUPFILE\\):" nil t) + (let* ((element (org-element-at-point)) + (included-file + (and (eq (org-element-type element) 'keyword) + (let ((value (org-element-property :value element))) + (and value + (string-match "^\\(\".+?\"\\|\\S-+\\)" value) + ;; Ignore search suffix. + (car (split-string + (org-remove-double-quotes + (match-string 1 value))))))))) + (when included-file + (let ((iname (expand-file-name included-file))) + (push (cons iname (org-publish-cache-ctime-of-src + (expand-file-name iname))) + included)))))) + (org-publish-cache-set-file-property (buffer-file-name) + :includes included))) (defun org-publish-cache-set-file-property (filename property value &optional project-name) -- 2.6.2 --=-=-=--