* org-persist
@ 2021-10-20 5:46 Colin Baxter 😺
2021-10-20 6:23 ` org-persist Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Colin Baxter 😺 @ 2021-10-20 5:46 UTC (permalink / raw)
To: emacs-orgmode
After today's pull of org-mode I get the warning that org-persist cannot
read its index. This is an entirely new warning (as of today) and I
assume is the result of the recent commits in org-persist.
Please, can this be corrected. The warning is irritating, especially so
since I do not use org-persist and have no wish to do so.
Best wishes,
Colin Baxter.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 5:46 org-persist Colin Baxter 😺
@ 2021-10-20 6:23 ` Ihor Radchenko
2021-10-20 6:37 ` org-persist Colin Baxter 😺
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-10-20 6:23 UTC (permalink / raw)
To: Colin Baxter 😺; +Cc: emacs-orgmode
Colin Baxter 😺 <m43cap@yandex.com> writes:
> After today's pull of org-mode I get the warning that org-persist cannot
> read its index. This is an entirely new warning (as of today) and I
> assume is the result of the recent commits in org-persist.
> Please, can this be corrected. The warning is irritating, especially so
> since I do not use org-persist and have no wish to do so.
Fixed. Note that org-persist is enabled by default together with
org-element-cache. You can indeed disable it, but, when enabled,
org-persist should improve loading times and overall performance.
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 6:23 ` org-persist Ihor Radchenko
@ 2021-10-20 6:37 ` Colin Baxter 😺
2021-10-20 6:55 ` org-persist Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Colin Baxter 😺 @ 2021-10-20 6:37 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
>>>>> Ihor Radchenko <yantar92@gmail.com> writes:
> Colin Baxter 😺 <m43cap@yandex.com> writes:
>> After today's pull of org-mode I get the warning that org-persist
>> cannot read its index. This is an entirely new warning (as of
>> today) and I assume is the result of the recent commits in
>> org-persist.
>> Please, can this be corrected. The warning is irritating,
>> especially so since I do not use org-persist and have no wish to
>> do so.
> Fixed.
Thank you. I cloned org-mode afresh and got the same warning during
'make' - I presume that too is now fixed.
> Note that org-persist is enabled by default together with
> org-element-cache. You can indeed disable it,
How can I do this?
Best wishes,
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 6:37 ` org-persist Colin Baxter 😺
@ 2021-10-20 6:55 ` Ihor Radchenko
2021-10-20 9:12 ` org-persist Colin Baxter 😺
2021-10-20 20:25 ` org-persist Samuel Wales
0 siblings, 2 replies; 21+ messages in thread
From: Ihor Radchenko @ 2021-10-20 6:55 UTC (permalink / raw)
To: Colin Baxter 😺; +Cc: emacs-orgmode
Colin Baxter 😺 <m43cap@yandex.com> writes:
> Thank you. I cloned org-mode afresh and got the same warning during
> 'make' - I presume that too is now fixed.
The warning should never appear now. I removed it completely.
> > Note that org-persist is enabled by default together with
> > org-element-cache. You can indeed disable it,
>
> How can I do this?
If you want to disable cache persistence, just set
`org-element-cache-persistent' to nil. It is documented in ORG-NEWS.
If you find ORG-NEWS not clear, please let me know. I will try to
improve the news record.
If you want to disable cache completely, you can set
`org-element-use-cache' to nil. However, I am hoping to make cache
enabled all the time in future. If you have specific reasons to avoid
org-element-cache, may you share them?
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 6:55 ` org-persist Ihor Radchenko
@ 2021-10-20 9:12 ` Colin Baxter 😺
2021-10-20 9:41 ` org-persist Ihor Radchenko
2021-10-20 20:25 ` org-persist Samuel Wales
1 sibling, 1 reply; 21+ messages in thread
From: Colin Baxter 😺 @ 2021-10-20 9:12 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
>>>>> Ihor Radchenko <yantar92@gmail.com> writes:
r> Colin Baxter 😺 <m43cap@yandex.com> writes:
>> Thank you. I cloned org-mode afresh and got the same warning
>> during 'make' - I presume that too is now fixed.
r> The warning should never appear now. I removed it completely.
>> > Note that org-persist is enabled by default together with >
>> org-element-cache. You can indeed disable it,
>>
>> How can I do this?
r> If you want to disable cache persistence, just set
r> `org-element-cache-persistent' to nil. It is documented in
r> ORG-NEWS. If you find ORG-NEWS not clear, please let me know. I
r> will try to improve the news record.
Thank you.
r> If you want to disable cache completely, you can set
r> `org-element-use-cache' to nil. However, I am hoping to make
r> cache enabled all the time in future. If you have specific
r> reasons to avoid org-element-cache, may you share them?
I may not have understood things fully, but it seems that the
org-persist directory is set only by the
org-persist-path. Unfortunately, the resulting org-persist directory in
~/.cache is no good for me since I daily clear out my ~/.cache using
bleachbit. I can't find a way of changing org-persist-path and grep
finds the variable only in ORG-NEWS.
Best wishes,
Colin.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 9:12 ` org-persist Colin Baxter 😺
@ 2021-10-20 9:41 ` Ihor Radchenko
2021-10-20 11:45 ` org-persist Colin Baxter 😺
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-10-20 9:41 UTC (permalink / raw)
To: Colin Baxter 😺; +Cc: emacs-orgmode
Colin Baxter 😺 <m43cap@yandex.com> writes:
> I may not have understood things fully, but it seems that the
> org-persist directory is set only by the
> org-persist-path. Unfortunately, the resulting org-persist directory in
> ~/.cache is no good for me since I daily clear out my ~/.cache using
> bleachbit. I can't find a way of changing org-persist-path and grep
> finds the variable only in ORG-NEWS.
Oops. It was my bad (fixed now). Should be `org-persist-directory'. You
can set it to other location in your init.el.
Also, deleting the cache would not make any harm, except possible slower
Org startup. org-persist should handle such case routinely. If you are
using bleachbit for privacy reasons, you may want to keep the default
cache directory: i.e. headline titles are stored in cache (unless you
are using org-crypt).
Best,
Ihot
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 9:41 ` org-persist Ihor Radchenko
@ 2021-10-20 11:45 ` Colin Baxter 😺
2021-10-20 12:44 ` org-persist Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Colin Baxter 😺 @ 2021-10-20 11:45 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
>>>>> Ihor Radchenko <yantar92@gmail.com> writes:
> Colin Baxter 😺 <m43cap@yandex.com> writes:
>> I may not have understood things fully, but it seems that the
>> org-persist directory is set only by the
>> org-persist-path. Unfortunately, the resulting org-persist
>> directory in ~/.cache is no good for me since I daily clear out
>> my ~/.cache using bleachbit. I can't find a way of changing
>> org-persist-path and grep finds the variable only in ORG-NEWS.
> Oops. It was my bad (fixed now). Should be
> `org-persist-directory'. You can set it to other location in your
> init.el.
Ok, thanks - that works - I used ~/.emacs.d/org-persist/. It might be
worth mentioning that the user should not create the
org-persist-directory himself. I did this originally (mkdir
~/.emacs.d/org-persist <RET>, touch ~/.emacs.d/org-persist/index <RET>)
and got various parsing error messages. Everything worked fine after I
deleted my own handiwork and left the directory creation to org-persist.
I had originally set org-element-use-cache to nil because I remember
reading somewhere that the nil setting would help prevent emacs from
hanging in org-mode.
I'll certainly begin using org-persist and see what happens.
Thanks for your help.
Best wishes,
Colin.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 11:45 ` org-persist Colin Baxter 😺
@ 2021-10-20 12:44 ` Ihor Radchenko
2021-12-17 21:05 ` org-persist Lele Gaifax
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-10-20 12:44 UTC (permalink / raw)
To: Colin Baxter 😺; +Cc: emacs-orgmode
Colin Baxter 😺 <m43cap@yandex.com> writes:
> Ok, thanks - that works - I used ~/.emacs.d/org-persist/. It might be
> worth mentioning that the user should not create the
> org-persist-directory himself. I did this originally (mkdir
> ~/.emacs.d/org-persist <RET>, touch ~/.emacs.d/org-persist/index <RET>)
> and got various parsing error messages. Everything worked fine after I
> deleted my own handiwork and left the directory creation to org-persist.
Thanks for reporting! I did not consider corrupted index file. Fixed now.
> I had originally set org-element-use-cache to nil because I remember
> reading somewhere that the nil setting would help prevent emacs from
> hanging in org-mode.
The cache code has been refactored. I tried my best to fix all the bugs
causing the hangs and I am not seeing hangs anymore. That's why I took a
risk to enable cache by default on main.
> I'll certainly begin using org-persist and see what happens.
Thanks! The new cache makes agenda/sparse tree searches at least several
times faster (I even got >10x on some agendas). But we need more testing.
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 6:55 ` org-persist Ihor Radchenko
2021-10-20 9:12 ` org-persist Colin Baxter 😺
@ 2021-10-20 20:25 ` Samuel Wales
2021-10-21 2:18 ` org-persist Ihor Radchenko
1 sibling, 1 reply; 21+ messages in thread
From: Samuel Wales @ 2021-10-20 20:25 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: Colin Baxter 😺, emacs-orgmode
On 10/19/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> enabled all the time in future. If you have specific reasons to avoid
> org-element-cache, may you share them?
fwiw, long ago, i disabled org element cache due to buffer corruption.
perhaps that has been fixed and tested.
also i presume this is all on main not bugfix [i haven't upgraded
maint yet; first thing i will do is try the archiving improvement].
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 20:25 ` org-persist Samuel Wales
@ 2021-10-21 2:18 ` Ihor Radchenko
0 siblings, 0 replies; 21+ messages in thread
From: Ihor Radchenko @ 2021-10-21 2:18 UTC (permalink / raw)
To: Samuel Wales; +Cc: Colin Baxter 😺, emacs-orgmode
Samuel Wales <samologist@gmail.com> writes:
> On 10/19/21, Ihor Radchenko <yantar92@gmail.com> wrote:
>> enabled all the time in future. If you have specific reasons to avoid
>> org-element-cache, may you share them?
>
> fwiw, long ago, i disabled org element cache due to buffer corruption.
> perhaps that has been fixed and tested.
I hope I fixed it. At least on my side. Need other users to test.
Currently, cache even self-verifies itself to double check corruption.
> also i presume this is all on main not bugfix [i haven't upgraded
> maint yet; first thing i will do is try the archiving improvement].
It is indeed not on bugfix. Just on main.
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist
2021-10-20 12:44 ` org-persist Ihor Radchenko
@ 2021-12-17 21:05 ` Lele Gaifax
2021-12-18 4:22 ` org-persist cache for remote files (was: org-persist) Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Lele Gaifax @ 2021-12-17 21:05 UTC (permalink / raw)
To: emacs-orgmode
Hi,
Ihor Radchenko <yantar92@gmail.com> writes:
>> I had originally set org-element-use-cache to nil because I remember
>> reading somewhere that the nil setting would help prevent emacs from
>> hanging in org-mode.
>
> The cache code has been refactored. I tried my best to fix all the bugs
> causing the hangs and I am not seeing hangs anymore. That's why I took a
> risk to enable cache by default on main.
today I spent the required time to investigate on one issue I had with my
Doom-based profile, that uses the "master" version of Orgmode, and found it
related to the org-persist functionality.
Basically since some time ago, quitting Emacs I was sometime prompted for a
password, some other time it took a considerable time to close the instance.
Digging down, I found that it was due to the persistent cache, that contained
several entries about remote (Tramp) documents: upon closing Emacs, the
org-persist-gc function is triggered, and the index is traversed to remove
dangling entries, performing a file-exist-p on each filename. For remote
files, that obviously mean prompting for the remote password (when the
ssh-agent does not carry it) or spending a considerable time in the network
round trip.
I wonder if there is some mechanism I could use to either prevent caching of
non-local documents or to avoid the check on existence in the org-persist-gc.
Thanks for any hint,
bye, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* org-persist cache for remote files (was: org-persist)
2021-12-17 21:05 ` org-persist Lele Gaifax
@ 2021-12-18 4:22 ` Ihor Radchenko
2021-12-18 6:46 ` org-persist cache for remote files Lele Gaifax
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-12-18 4:22 UTC (permalink / raw)
To: Lele Gaifax; +Cc: emacs-orgmode
Lele Gaifax <lele@metapensiero.it> writes:
> Digging down, I found that it was due to the persistent cache, that contained
> several entries about remote (Tramp) documents: upon closing Emacs, the
> org-persist-gc function is triggered, and the index is traversed to remove
> dangling entries, performing a file-exist-p on each filename. For remote
> files, that obviously mean prompting for the remote password (when the
> ssh-agent does not carry it) or spending a considerable time in the network
> round trip.
Thanks for reporting! I am not using tramp regularly and I did not think
about this use case.
> I wonder if there is some mechanism I could use to either prevent caching of
> non-local documents or to avoid the check on existence in the org-persist-gc.
I can easily make a change that always garbage-collect non-local
documents without trying to connect to remote server. This can be one
solution of your problem. However, I am not sure if it is the best
solution.
May I know more about your use cases? Do you frequently open large
remote Org files? If you do, trashing their cache may not be optimal.
Maybe we should better make this a user option?
Or we can go a different route and just not garbage collect remote files
until cache gets too large.
WDYT?
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 4:22 ` org-persist cache for remote files (was: org-persist) Ihor Radchenko
@ 2021-12-18 6:46 ` Lele Gaifax
2021-12-18 14:21 ` Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Lele Gaifax @ 2021-12-18 6:46 UTC (permalink / raw)
To: emacs-orgmode
Ihor Radchenko <yantar92@gmail.com> writes:
> Lele Gaifax <lele@metapensiero.it> writes:
>
>> I wonder if there is some mechanism I could use to either prevent caching
>> of non-local documents or to avoid the check on existence in the
>> org-persist-gc.
>
> I can easily make a change that always garbage-collect non-local
> documents without trying to connect to remote server. This can be one
> solution of your problem. However, I am not sure if it is the best
> solution.
>
> May I know more about your use cases? Do you frequently open large
> remote Org files? If you do, trashing their cache may not be optimal.
I do not open remote Org files very often, but surely happens now and then,
typically from my laptop to see/edit something I keep on my desktop PC.
> Maybe we should better make this a user option?
Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes
is enabled, the cleanup procedure would ignore remote Org documents and a new
explicit (interactive) org-persist-gc-remotes would take care of them.
> Or we can go a different route and just not garbage collect remote files
> until cache gets too large.
Yes, a threshold would work too.
Thanks a lot,
ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 6:46 ` org-persist cache for remote files Lele Gaifax
@ 2021-12-18 14:21 ` Ihor Radchenko
2021-12-18 14:36 ` Lele Gaifax
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-12-18 14:21 UTC (permalink / raw)
To: Lele Gaifax; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 615 bytes --]
Lele Gaifax <lele@metapensiero.it> writes:
>> Maybe we should better make this a user option?
>
> Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes
> is enabled, the cleanup procedure would ignore remote Org documents and a new
> explicit (interactive) org-persist-gc-remotes would take care of them.
>
>> Or we can go a different route and just not garbage collect remote files
>> until cache gets too large.
>
> Yes, a threshold would work too.
Can you test the attached 2 patches? They do not seem to break local
files, but I do not have remote file caches to test on.
Best,
Ihor
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-persist.el-Make-org-persist-directory-a-custom-v.patch --]
[-- Type: text/x-diff, Size: 2638 bytes --]
From 592608d3c48cb1e63f9b4427576f0698ca190e9a Mon Sep 17 00:00:00 2001
Message-Id: <592608d3c48cb1e63f9b4427576f0698ca190e9a.1639837163.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Dec 2021 22:13:24 +0800
Subject: [PATCH] org-persist.el: Make org-persist-directory a custom variable
* lisp/org-persist.el (org-persist): Create org-persist customisation
group.
(org-persist-directory): Convert into defcustom.
---
lisp/org-persist.el | 34 +++++++++++++++++++++-------------
1 file changed, 21 insertions(+), 13 deletions(-)
diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index f06d4fc61..90f9a3bc9 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -35,19 +35,27 @@ (declare-function org-back-to-heading "org" (&optional invisible-ok))
(declare-function org-next-visible-heading "org" (arg))
(declare-function org-at-heading-p "org" (&optional invisible-not-ok))
-(defvar org-persist-directory (expand-file-name
- (org-file-name-concat
- (let ((cache-dir (when (fboundp 'xdg-cache-home)
- (xdg-cache-home))))
- (if (or (seq-empty-p cache-dir)
- (not (file-exists-p cache-dir))
- (file-exists-p (org-file-name-concat
- user-emacs-directory
- "org-persist")))
- user-emacs-directory
- cache-dir))
- "org-persist/"))
- "Directory where the data is stored.")
+
+(defgroup org-persist nil
+ "Persistent cache for Org mode."
+ :tag "Org persist"
+ :group 'org)
+
+(defcustom org-persist-directory (expand-file-name
+ (org-file-name-concat
+ (let ((cache-dir (when (fboundp 'xdg-cache-home)
+ (xdg-cache-home))))
+ (if (or (seq-empty-p cache-dir)
+ (not (file-exists-p cache-dir))
+ (file-exists-p (org-file-name-concat
+ user-emacs-directory
+ "org-persist")))
+ user-emacs-directory
+ cache-dir))
+ "org-persist/"))
+ "Directory where the data is stored."
+ :group 'org-persist
+ :type 'directory)
(defvar org-persist-index-file "index"
"File name used to store the data index.")
--
2.32.0
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-org-persist.el-Customise-persistence-of-data-for-rem.patch --]
[-- Type: text/x-diff, Size: 3480 bytes --]
From de89eca758ca1972d75c8830aee224376a1e364e Mon Sep 17 00:00:00 2001
Message-Id: <de89eca758ca1972d75c8830aee224376a1e364e.1639837255.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Dec 2021 22:14:24 +0800
Subject: [PATCH] org-persist.el: Customise persistence of data for remote
files
* lisp/org-persist.el (org-persist-remote-files): New defcustom
defining strategy for persisting data associated with remote files.
Separate strategy is necessary to avoid unexpected password promtps
while garbage collecting the cache: remote files may require server
login.
(org-persist-gc): Decide whether to store remote file data according
to `org-persist-remote-files'.
Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u
---
lisp/org-persist.el | 41 ++++++++++++++++++++++++++++++++++-------
1 file changed, 34 insertions(+), 7 deletions(-)
diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 90f9a3bc9..613290afc 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -57,6 +57,22 @@ (defcustom org-persist-directory (expand-file-name
:group 'org-persist
:type 'directory)
+(defcustom org-persist-remote-files 100
+ "Whether to keep persistent data for remote files.
+
+When this variable is nil, never save persitent data associated with
+remote files. When `t', always keep the data. When
+`check-existance', contact remote server containing the file and only
+keep the data when the file exists on the server.
+
+Note that the last option `check-existance' may cause Emacs to show
+password prompts to log in."
+ :group 'org-persist
+ :type '(choice (const :tag "Never" nil)
+ (const :tag "Always" t)
+ (number :tag "Keep note more than X files")
+ (const :tag "Check if exist on remote" 'check-existence)))
+
(defvar org-persist-index-file "index"
"File name used to store the data index.")
@@ -287,15 +303,26 @@ (defun org-persist-read-all-buffer ()
(defun org-persist-gc ()
"Remove stored data for not existing files or unregistered variables."
- (let (new-index)
+ (let (new-index (remote-files-num 0))
(dolist (index org-persist--index)
- (let ((file (plist-get index :path))
- (persist-file (when (plist-get index :persist-file)
- (org-file-name-concat
- org-persist-directory
- (plist-get index :persist-file)))))
+ (let* ((file (plist-get index :path))
+ (file-remote (when file (file-remote-p file)))
+ (persist-file (when (plist-get index :persist-file)
+ (org-file-name-concat
+ org-persist-directory
+ (plist-get index :persist-file)))))
(when (and file persist-file)
- (if (file-exists-p file)
+ (when (and file-remote persist-file)
+ (cl-incf remote-files-num))
+ (if (if (not file-remote)
+ (file-exists-p file)
+ (pcase org-persist-remote-files
+ ('t t)
+ ('check-existence
+ (file-exists-p file))
+ ((pred #'numberp)
+ (<= org-persist-remote-files remote-files-num))
+ (_ nil)))
(push index new-index)
(when (file-exists-p persist-file)
(delete-file persist-file)
--
2.32.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 14:21 ` Ihor Radchenko
@ 2021-12-18 14:36 ` Lele Gaifax
2021-12-18 14:47 ` Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Lele Gaifax @ 2021-12-18 14:36 UTC (permalink / raw)
To: emacs-orgmode
Ihor Radchenko <yantar92@gmail.com> writes:
> Lele Gaifax <lele@metapensiero.it> writes:
>
>>> Maybe we should better make this a user option?
>>
>> Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes
>> is enabled, the cleanup procedure would ignore remote Org documents and a new
>> explicit (interactive) org-persist-gc-remotes would take care of them.
>>
>>> Or we can go a different route and just not garbage collect remote files
>>> until cache gets too large.
>>
>> Yes, a threshold would work too.
>
> Can you test the attached 2 patches? They do not seem to break local
> files, but I do not have remote file caches to test on.
Yes, I'll be able to do some test in a couple of hours.
Thank you!
In the meanwhile, here below some notes:
> ...
> Subject: [PATCH] org-persist.el: Customise persistence of data for remote
> files
>
> * lisp/org-persist.el (org-persist-remote-files): New defcustom
> defining strategy for persisting data associated with remote files.
> Separate strategy is necessary to avoid unexpected password promtps
> while garbage collecting the cache: remote files may require server
> login.
> ...
s/promtps/prompts
> ...
> +(defcustom org-persist-remote-files 100
> + "Whether to keep persistent data for remote files.
> +
> +When this variable is nil, never save persitent data associated with
> +remote files. When `t', always keep the data. When
> +`check-existance', contact remote server containing the file and only
> +keep the data when the file exists on the server.
s/check-existance/check-existence
Also, shouldn't the doc mention the meaning of the numberp, expecially since
that's the default?
> +Note that the last option `check-existance' may cause Emacs to show
> +password prompts to log in."
Thanks again,
bye, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 14:36 ` Lele Gaifax
@ 2021-12-18 14:47 ` Ihor Radchenko
2021-12-18 15:28 ` Lele Gaifax
0 siblings, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-12-18 14:47 UTC (permalink / raw)
To: Lele Gaifax; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 132 bytes --]
Lele Gaifax <lele@metapensiero.it> writes:
> In the meanwhile, here below some notes:
Thanks! See the updated patch.
Best,
Ihor
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-persist.el-Customise-persistence-of-data-for-rem.patch --]
[-- Type: text/x-diff, Size: 3556 bytes --]
From 92878fab54c26800562409f3c686f065b7523a61 Mon Sep 17 00:00:00 2001
Message-Id: <92878fab54c26800562409f3c686f065b7523a61.1639838789.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Dec 2021 22:14:24 +0800
Subject: [PATCH] org-persist.el: Customise persistence of data for remote
files
* lisp/org-persist.el (org-persist-remote-files): New defcustom
defining strategy for persisting data associated with remote files.
Separate strategy is necessary to avoid unexpected password promtps
while garbage collecting the cache: remote files may require server
login.
(org-persist-gc): Decide whether to store remote file data according
to `org-persist-remote-files'.
Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u
---
lisp/org-persist.el | 42 +++++++++++++++++++++++++++++++++++-------
1 file changed, 35 insertions(+), 7 deletions(-)
diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 90f9a3bc9..4c0673c72 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -57,6 +57,23 @@ (defcustom org-persist-directory (expand-file-name
:group 'org-persist
:type 'directory)
+(defcustom org-persist-remote-files 100
+ "Whether to keep persistent data for remote files.
+
+When this variable is nil, never save persitent data associated with
+remote files. When `t', always keep the data. When
+`check-existence', contact remote server containing the file and only
+keep the data when the file exists on the server. When a number, keep
+up to that number persistent values for remote files.
+
+Note that the last option `check-existence' may cause Emacs to show
+password prompts to log in."
+ :group 'org-persist
+ :type '(choice (const :tag "Never" nil)
+ (const :tag "Always" t)
+ (number :tag "Keep note more than X files")
+ (const :tag "Check if exist on remote" 'check-existence)))
+
(defvar org-persist-index-file "index"
"File name used to store the data index.")
@@ -287,15 +304,26 @@ (defun org-persist-read-all-buffer ()
(defun org-persist-gc ()
"Remove stored data for not existing files or unregistered variables."
- (let (new-index)
+ (let (new-index (remote-files-num 0))
(dolist (index org-persist--index)
- (let ((file (plist-get index :path))
- (persist-file (when (plist-get index :persist-file)
- (org-file-name-concat
- org-persist-directory
- (plist-get index :persist-file)))))
+ (let* ((file (plist-get index :path))
+ (file-remote (when file (file-remote-p file)))
+ (persist-file (when (plist-get index :persist-file)
+ (org-file-name-concat
+ org-persist-directory
+ (plist-get index :persist-file)))))
(when (and file persist-file)
- (if (file-exists-p file)
+ (when (and file-remote persist-file)
+ (cl-incf remote-files-num))
+ (if (if (not file-remote)
+ (file-exists-p file)
+ (pcase org-persist-remote-files
+ ('t t)
+ ('check-existence
+ (file-exists-p file))
+ ((pred #'numberp)
+ (<= org-persist-remote-files remote-files-num))
+ (_ nil)))
(push index new-index)
(when (file-exists-p persist-file)
(delete-file persist-file)
--
2.32.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 14:47 ` Ihor Radchenko
@ 2021-12-18 15:28 ` Lele Gaifax
2021-12-18 18:18 ` Lele Gaifax
2021-12-19 9:56 ` Ihor Radchenko
0 siblings, 2 replies; 21+ messages in thread
From: Lele Gaifax @ 2021-12-18 15:28 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
Ihor Radchenko <yantar92@gmail.com> writes:
> Lele Gaifax <lele@metapensiero.it> writes:
>
>> In the meanwhile, here below some notes:
>
> Thanks! See the updated patch.
Great, you're so fast! :-)
I couldn't test yet, but I saw a couple of other glitches, sorry for not
noticing first.
> ...
> +(defcustom org-persist-remote-files 100
> + "Whether to keep persistent data for remote files.
> +
> +When this variable is nil, never save persitent data associated with
> +remote files. When `t', always keep the data. When
> +`check-existence', contact remote server containing the file and only
> +keep the data when the file exists on the server. When a number, keep
> +up to that number persistent values for remote files.
> ...
s/persitent/persistent
> +Note that the last option `check-existence' may cause Emacs to show
> +password prompts to log in."
I'd remove "last" above, since `check-existence' is not the last (described)
option anymore.
> + :group 'org-persist
> + :type '(choice (const :tag "Never" nil)
> + (const :tag "Always" t)
> + (number :tag "Keep note more than X files")
Now that you explained the meaning, maybe better to be explicit here,
mentioning "X remote files"?
Thanks&bye, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 15:28 ` Lele Gaifax
@ 2021-12-18 18:18 ` Lele Gaifax
2021-12-19 9:56 ` Ihor Radchenko
1 sibling, 0 replies; 21+ messages in thread
From: Lele Gaifax @ 2021-12-18 18:18 UTC (permalink / raw)
To: Ihor Radchenko; +Cc: emacs-orgmode
Lele Gaifax <lele@metapensiero.it> writes:
>> + :group 'org-persist
>> + :type '(choice (const :tag "Never" nil)
>> + (const :tag "Always" t)
>> + (number :tag "Keep note more than X files")
>
> Now that you explained the meaning, maybe better to be explicit here,
> mentioning "X remote files"?
Sigh, re-reading, I see one more glitch there: s/Keep note/Keep not/
I'm sorry, but real work swallowed all my day, and could not try this out yet.
Hopefully I will be able to do some test tomorrow morning.
bye, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-18 15:28 ` Lele Gaifax
2021-12-18 18:18 ` Lele Gaifax
@ 2021-12-19 9:56 ` Ihor Radchenko
2021-12-21 8:11 ` Lele Gaifax
1 sibling, 1 reply; 21+ messages in thread
From: Ihor Radchenko @ 2021-12-19 9:56 UTC (permalink / raw)
To: Lele Gaifax; +Cc: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 214 bytes --]
Lele Gaifax <lele@metapensiero.it> writes:
> I couldn't test yet, but I saw a couple of other glitches, sorry for not
> noticing first.
Thanks for the proof reading.
I updated the patch accordingly.
Best,
Ihor
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-persist.el-Customise-persistence-of-data-for-rem.patch --]
[-- Type: text/x-diff, Size: 3579 bytes --]
From 557e7cec6dd22b09468b82650feed2020f97d781 Mon Sep 17 00:00:00 2001
Message-Id: <557e7cec6dd22b09468b82650feed2020f97d781.1639907677.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Dec 2021 22:14:24 +0800
Subject: [PATCH] org-persist.el: Customise persistence of data for remote
files
* lisp/org-persist.el (org-persist-remote-files): New defcustom
defining strategy for persisting data associated with remote files.
Separate strategy is necessary to avoid unexpected password prompts
while garbage collecting the cache: remote files may require server
login.
(org-persist-gc): Decide whether to store remote file data according
to `org-persist-remote-files'.
Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u
---
lisp/org-persist.el | 42 +++++++++++++++++++++++++++++++++++-------
1 file changed, 35 insertions(+), 7 deletions(-)
diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 90f9a3bc9..8e8ecc196 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -57,6 +57,23 @@ (defcustom org-persist-directory (expand-file-name
:group 'org-persist
:type 'directory)
+(defcustom org-persist-remote-files 100
+ "Whether to keep persistent data for remote files.
+
+When this variable is nil, never save persistent data associated with
+remote files. When `t', always keep the data. When a number, keep up
+to that number persistent values for remote files. When
+`check-existence', contact remote server containing the file and only
+keep the data when the file exists on the server.
+
+Note that setting value to `check-existence' will require logging in
+to remote servers. Emacs may show multiple password prompts."
+ :group 'org-persist
+ :type '(choice (const :tag "Never" nil)
+ (const :tag "Always" t)
+ (number :tag "Up to X files")
+ (const :tag "Check if exist on remote" 'check-existence)))
+
(defvar org-persist-index-file "index"
"File name used to store the data index.")
@@ -287,15 +304,26 @@ (defun org-persist-read-all-buffer ()
(defun org-persist-gc ()
"Remove stored data for not existing files or unregistered variables."
- (let (new-index)
+ (let (new-index (remote-files-num 0))
(dolist (index org-persist--index)
- (let ((file (plist-get index :path))
- (persist-file (when (plist-get index :persist-file)
- (org-file-name-concat
- org-persist-directory
- (plist-get index :persist-file)))))
+ (let* ((file (plist-get index :path))
+ (file-remote (when file (file-remote-p file)))
+ (persist-file (when (plist-get index :persist-file)
+ (org-file-name-concat
+ org-persist-directory
+ (plist-get index :persist-file)))))
(when (and file persist-file)
- (if (file-exists-p file)
+ (when (and file-remote persist-file)
+ (cl-incf remote-files-num))
+ (if (if (not file-remote)
+ (file-exists-p file)
+ (pcase org-persist-remote-files
+ ('t t)
+ ('check-existence
+ (file-exists-p file))
+ ((pred #'numberp)
+ (<= org-persist-remote-files remote-files-num))
+ (_ nil)))
(push index new-index)
(when (file-exists-p persist-file)
(delete-file persist-file)
--
2.32.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-19 9:56 ` Ihor Radchenko
@ 2021-12-21 8:11 ` Lele Gaifax
2021-12-21 9:03 ` Ihor Radchenko
0 siblings, 1 reply; 21+ messages in thread
From: Lele Gaifax @ 2021-12-21 8:11 UTC (permalink / raw)
To: emacs-orgmode
Hi Ihor,
I had some trouble testing your patches, and the outcome is still unclear to
me: apparently (again, cannot be sure this is not caused by "improper"
application of the patches) the new version of org-persist, with default value
for o-p-remote-files (100), removes any remote file from the persistent index,
without even prompting for a password.
I will try again (and harder) as time permits.
ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: org-persist cache for remote files
2021-12-21 8:11 ` Lele Gaifax
@ 2021-12-21 9:03 ` Ihor Radchenko
0 siblings, 0 replies; 21+ messages in thread
From: Ihor Radchenko @ 2021-12-21 9:03 UTC (permalink / raw)
To: Lele Gaifax; +Cc: emacs-orgmode
Lele Gaifax <lele@metapensiero.it> writes:
> Hi Ihor,
>
> I had some trouble testing your patches, and the outcome is still unclear to
> me: apparently (again, cannot be sure this is not caused by "improper"
> application of the patches) the new version of org-persist, with default value
> for o-p-remote-files (100), removes any remote file from the persistent index,
> without even prompting for a password.
Oops. Try:
- (<= org-persist-remote-files remote-files-num))
+ (>= org-persist-remote-files remote-files-num))
Best,
Ihor
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-12-21 9:04 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-20 5:46 org-persist Colin Baxter 😺
2021-10-20 6:23 ` org-persist Ihor Radchenko
2021-10-20 6:37 ` org-persist Colin Baxter 😺
2021-10-20 6:55 ` org-persist Ihor Radchenko
2021-10-20 9:12 ` org-persist Colin Baxter 😺
2021-10-20 9:41 ` org-persist Ihor Radchenko
2021-10-20 11:45 ` org-persist Colin Baxter 😺
2021-10-20 12:44 ` org-persist Ihor Radchenko
2021-12-17 21:05 ` org-persist Lele Gaifax
2021-12-18 4:22 ` org-persist cache for remote files (was: org-persist) Ihor Radchenko
2021-12-18 6:46 ` org-persist cache for remote files Lele Gaifax
2021-12-18 14:21 ` Ihor Radchenko
2021-12-18 14:36 ` Lele Gaifax
2021-12-18 14:47 ` Ihor Radchenko
2021-12-18 15:28 ` Lele Gaifax
2021-12-18 18:18 ` Lele Gaifax
2021-12-19 9:56 ` Ihor Radchenko
2021-12-21 8:11 ` Lele Gaifax
2021-12-21 9:03 ` Ihor Radchenko
2021-10-20 20:25 ` org-persist Samuel Wales
2021-10-21 2:18 ` org-persist Ihor Radchenko
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).