From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id yit3NraWlWNrcAEAbAwnHQ (envelope-from ) for ; Sun, 11 Dec 2022 09:37:10 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id yLrJNbaWlWPhEAAA9RJhRA (envelope-from ) for ; Sun, 11 Dec 2022 09:37:10 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CF594EB1B for ; Sun, 11 Dec 2022 09:37:09 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p4Hot-00075S-0r; Sun, 11 Dec 2022 03:36:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p4Hon-00074m-6w for emacs-orgmode@gnu.org; Sun, 11 Dec 2022 03:36:17 -0500 Received: from out3-smtp.messagingengine.com ([66.111.4.27]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p4Hok-0003x8-Qe for emacs-orgmode@gnu.org; Sun, 11 Dec 2022 03:36:16 -0500 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id E558D5C00D6 for ; Sun, 11 Dec 2022 03:36:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sun, 11 Dec 2022 03:36:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tecosaur.net; h= cc:content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t= 1670747770; x=1670834170; bh=8F8hs8/KlOCR5mp+RrOjDx498PjCv4v5IT2 lQ2vz99Q=; b=RmoCEQh5kevmxlxmTbmW/X25tkSHIQckTFupa3jazvIgxklBOst brIWjdJRJmYtZK+NiVmDomTnKZksy7XcB5PTQCEWUh1ZT79lP9WvcM7ofJVSSg0g zLDgUiZCdaFKv5DiIPq3H5qmTXOx389DPngsA3jkYCISi65ZK5l9XDWfXowkF8U7 cXf7rkrq6yWR6dEeSRvfKwo1grgAEnlbJp9wsMi9eyk/1E/uhk0W41KmT7nE1wEl sDywoDdlt8+lyq3jwnZdPePMsiFrPQli1nWtSTxgnPG6dFwYdL4EX/ULImBvYA2d gi2lhtn9Urk+Zt2ibnC0xsGF9qwOwXwFr8g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1670747770; x= 1670834170; bh=8F8hs8/KlOCR5mp+RrOjDx498PjCv4v5IT2lQ2vz99Q=; b=q PSro2qdikmv/IMmb//9qi6pD1X283TXQQ7dNpAhUCkiNNYDxQGGflf8n/XHzR+yy Nc/XQ6tSKG1dhUxNBUQrL4SjjU6GMuiyhwns3oCmNEYU7y8dIDNpqg1ux+rPkrHw G2gugOl8KiGWtjLarsB2KGR4S7o4NQsFpzMmDSlFWMBI9GeTA+OJ0xKi9WbKdKmE 6l9VvS4MYok7on8iQteza69duu5sCIBLoB0UKA7J3ClNOi5uxXupM87FDImvm11V P9k/I432BUWrc//kaX79NCHEYVu3h8ICmAXRDaVIMwU2JsDyXzd3Ua1hMsPc2i+R hqH5aO6WalnJUZvz0rG8g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdehgdduvdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffuffgfkfggtgesmhdtreertd erjeenucfhrhhomhepvfhimhhothhhhicuoehorhhgmhhouggvsehtvggtrdhtvggtohhs rghurhdrnhgvtheqnecuggftrfgrthhtvghrnhepteelveelvddtfeehteeihfeuveejge egtdeuteeljeeghfejheevudeiieetvefgnecuffhomhgrihhnpehorhhgmhhouggvrdho rhhgpdhlihgsvghrrghprgihrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrg hrrghmpehmrghilhhfrhhomhepohhrghhmohguvgesthgvtgdrthgvtghoshgruhhrrdhn vght X-ME-Proxy: Feedback-ID: iecf94634:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 11 Dec 2022 03:36:09 -0500 (EST) From: Timothy To: emacs-orgmode@gnu.org Subject: [PATCH] Merge loaded org-persist index with index file contents Date: Sun, 11 Dec 2022 15:59:41 +0800 User-agent: mu4e 1.8.11; emacs 28.2.50 Message-ID: <87359mqqvf.fsf@tec.tecosaur.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: none client-ip=66.111.4.27; envelope-from=orgmode@tec.tecosaur.net; helo=out3-smtp.messagingengine.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1670747830; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=8F8hs8/KlOCR5mp+RrOjDx498PjCv4v5IT2lQ2vz99Q=; b=r97cVb4i3CQyc27NLAKwG7FpsLyQzx1EwPOGcnJH6AmabqlAeSSf1bqvaIyK/hQ7kQvjJd 6dkC1Bzqq6VdNqCLdxyeRTTEkA3azPlfdaegGnHImGdM1/hAp7oFMG5z3ffaovtjZQXqyx EvSfpL0QEpKPfM0f1iBH9FM49foOUD9QPqYzY5BsEuWtwWrNehWlNowcE9pVR+ja0+QC5S BpBNWhIAOzvK2y2XlHzZ/NrHnWcSpR8+P9bZQapEu5OYlAVgvJFto41aMEJMyYbynE6xlL HCmenSRhv1cQC1gYpQ/Fe6az/MQuFM7RSfp2kPAwUh3DDZAFWBmqUz4KO5zVvA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1670747830; a=rsa-sha256; cv=none; b=lUbbp6TsP62nKnr3dXXadljIQsjLRMB4OoTk1EtegA1/ENjt+nZkHwQaiFwK9nU7pJRcKP X2eZ2OcebdH/5Iga6GGSu/YuQvPCgI85rSFRvt6+jIp/hPF3ojXOqHC86udTNP0UuCQYE4 DjZbtDEQi+eoxgVudyslsTnpnp46Y8SNrTVhgqi71FhT+2D8OEPdyBY0h4HrP37D1zD3X1 6MtqTSNa1yW5tv88n87QjYBk4R8yMOYMkEpV8jGP9WOJFQuJcerBQD04lqKZ0ITL6JIv2T wwtPXhWvl/tUu8Bl499LAK18g8GHcdIyAJWVkIe6EQV/V24wuOdsItkmy/4Umg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tecosaur.net header.s=fm1 header.b=RmoCEQh5; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="q PSro2q"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 1.15 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tecosaur.net header.s=fm1 header.b=RmoCEQh5; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="q PSro2q"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: CF594EB1B X-Spam-Score: 1.15 X-Migadu-Scanner: scn1.migadu.com X-TUID: Oc6DtlBd+XrQ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi All, I=E2=80=99ve recently pushed a batch of improvements/fixes to org-persist, = particularly with the handling of url containers. Expect to actually see caching of remo= te images now =F0=9F=99=82. I=E2=80=99ve also noticed that the org-persist index is written when Emacs = closes, and so this sequence of event can happen: 1. Initial empty index file 2. Start Emacs session #1, add 20 entries to the index 3. Start Emacs session #2, add 1 entry to the index 4. Close Emacs session #1 =E2=81=83 The index file now has 20 entries in it 5. Close Emacs session #2 =E2=81=83 The index file now has 1 entry in it I think ideally at the end we=E2=80=99d have 21 entries in the index. So, I= =E2=80=99ve added a simple merge function that checks to see if the index file has been updated since loading, and if so just grabs new index entries from the file. We don= =E2=80=99t bother trying to merge modified entries. How does this sound? All the best, Timothy --=20 Timothy (=E2=80=98tecosaur=E2=80=99/=E2=80=98TEC=E2=80=99), Org mode contri= butor. Learn more about Org mode at . Support Org development at , or support my work at . --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-org-persist-Merge-index-with-index-file-content.patch >From 6a12fca24f1b89129424b8fc2902719f5e053832 Mon Sep 17 00:00:00 2001 From: TEC Date: Sat, 10 Dec 2022 23:53:44 +0800 Subject: [PATCH] org-persist: Merge index with index file content * lisp/org-persist.el (org-persist-write, org-persist-load, org-persist--index-age): Check if the index file has been externally updated since loading, and if so try to perform basic merging of the current index file contents and the loaded index before performing GC or overwriting the index file. --- lisp/org-persist.el | 53 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 7 deletions(-) diff --git a/lisp/org-persist.el b/lisp/org-persist.el index f215310a2..e9310d172 100644 --- a/lisp/org-persist.el +++ b/lisp/org-persist.el @@ -259,6 +259,9 @@ (defvar org-persist--index-hash nil "Hash table storing `org-persist--index'. Used for quick access. They keys are conses of (container . associated).") +(defvar org-persist--index-age nil + "The time at which the index was loaded, as given by `current-time'.") + (defvar org-persist--report-time 0.5 "Whether to report read/write time. @@ -589,8 +592,9 @@ (defalias 'org-persist-load:file #'org-persist-read:file) (defun org-persist-load:index (container index-file _) "Load `org-persist--index' from INDEX-FILE according to CONTAINER." (unless org-persist--index - (setq org-persist--index (org-persist-read:index container index-file nil)) - (setq org-persist--index-hash nil) + (setq org-persist--index (org-persist-read:index container index-file nil) + org-persist--index-hash nil + org-persist--index-age (current-time)) (if org-persist--index (mapc (lambda (collection) (org-persist--add-to-index collection 'hash)) org-persist--index) (setq org-persist--index nil) @@ -690,17 +694,49 @@ (defun org-persist-write:index (container _) (message "Missing write access rights to org-persist-directory: %S" org-persist-directory)))) (when (file-exists-p org-persist-directory) - (org-persist--write-elisp-file - (org-file-name-concat org-persist-directory org-persist-index-file) - org-persist--index - t t) - (org-file-name-concat org-persist-directory org-persist-index-file))) + (let ((index-file + (org-file-name-concat org-persist-directory org-persist-index-file))) + (org-persist--merge-index-with-disk) + (org-persist--write-elisp-file index-file org-persist--index t t) + (setq org-persist--index-age (current-time)) + index-file))) (defun org-persist--save-index () "Save `org-persist--index'." (org-persist-write:index `(index ,org-persist--storage-version) nil)) +(defun org-persist--merge-index-with-disk () + "Merge `org-persist--index' with the current index file on disk." + (let* ((index-file + (org-file-name-concat org-persist-directory org-persist-index-file)) + (disk-index + (and (file-exists-p index-file) + (org-file-newer-than-p index-file org-persist--index-age) + (org-persist-read:index `(index ,org-persist--storage-version) index-file nil))) + (combined-index + (org-persist--merge-index org-persist--index disk-index))) + (when disk-index + (setq org-persist--index combined-index + org-persist--index-age (current-time))))) + +(defun org-persist--merge-index (base other) + "Attempt to merge new index items in OTHER into BASE. +Items with different details are considered too difficult, and skipped." + (if other + (let ((new (cl-set-difference other base :test #'equal)) + (base-files (mapcar (lambda (s) (plist-get s :persist-file)) base)) + (combined (reverse base))) + (dolist (item (nreverse new)) + (unless (or (memq 'index (mapcar #'car (plist-get collection :container))) + (not (file-exists-p + (org-file-name-concat org-persist-directory + (plist-get item :persist-file)))) + (member (plist-get item :persist-file) base-files)) + (push item combined))) + (nreverse combined)) + base)) + ;;;; Public API (cl-defun org-persist-register (container &optional associated &rest misc @@ -951,6 +987,9 @@ (defun org-persist-associated-files:url (container collection) (defun org-persist-gc () "Remove expired or unregistered containers and orphaned files. Also, remove containers associated with non-existing files." + (if org-persist--index + (org-persist--merge-index-with-disk) + (org-persist--load-index)) (unless (and org-persist-disable-when-emacs-Q ;; FIXME: This is relying on undocumented fact that ;; Emacs sets `user-init-file' to nil when loaded with -- 2.38.1 --=-=-=--