* [BUG] org-goto slows down org-set-property
@ 2021-07-10 15:47 Maxim Nikulin
2021-07-11 9:25 ` Maxim Nikulin
2024-10-13 11:45 ` Ihor Radchenko
0 siblings, 2 replies; 11+ messages in thread
From: Maxim Nikulin @ 2021-07-10 15:47 UTC (permalink / raw)
To: emacs-orgmode
It is rather annoying that C-c C-x p `org-set-property' may be quite
slow if a file is large enough. (I do not consider it huge however.)
There is noticeable delay before the prompt for property name appears in
minibuffer. Accidentally I have noticed that sometimes it is fast.
Problem appears after first jump using C-u C-c C-j `org-goto'. Actually
a jump is not required, it is enough to get jump (refile) targets.
I hope, the following numbers are convincing enough. Second attempt to
get list of properties is much slower than the first one.
#+begin_src elisp
(require 'org-refile)
(let ((org-refile-targets '((nil :maxlevel . 6)))
(org-refile-use-outline-path 'file)
(org-refile-use-cache t)
(iters 10))
(cons
(list (org-version))
(mapcar (lambda (f)
(append (list f) (benchmark-run iters (funcall f))))
(list #'org-buffer-property-keys
#'org-refile-get-targets
#'org-buffer-property-keys))))
#+end_src
git master:
#+RESULTS:
| 9.4.6 | | | |
| org-buffer-property-keys | 0.065695 | 0 | 0.0 |
| org-refile-get-targets | 1.836473 | 12 | 0.39498671199999985 |
| org-buffer-property-keys | 0.563227 | 0 | 0.0 |
elpa-org system package for Ubuntu-20.04 is even more slower:
#+RESULTS:
| 9.3.1 | | | |
| org-buffer-property-keys | 0.242574778 | 0 | 0.0 |
| org-refile-get-targets | 2.669112479 | 13 | 0.285175928 |
| org-buffer-property-keys | 1.231223427 | 0 | 0.0 |
The effect is noticeable for the doc/org-manual.org file from org
sources though absolute numbers are well below delay that makes
interaction incomfortable.
In the configuration above effect of `org-refile-use-cache' is crucial,
contribution of `org-refile-use-outline-path' is noticeable but not so
important.
My expectation is that neither `org-refile-use-cache' nor `org-goto'
command should make adding property slower.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [BUG] org-goto slows down org-set-property
2021-07-10 15:47 [BUG] org-goto slows down org-set-property Maxim Nikulin
@ 2021-07-11 9:25 ` Maxim Nikulin
2021-07-11 11:49 ` Ihor Radchenko
2024-07-07 12:38 ` Ihor Radchenko
2024-10-13 11:45 ` Ihor Radchenko
1 sibling, 2 replies; 11+ messages in thread
From: Maxim Nikulin @ 2021-07-11 9:25 UTC (permalink / raw)
To: emacs-orgmode
On 10/07/2021 22:47, Maxim Nikulin wrote:
> It is rather annoying that C-c C-x p `org-set-property' may be quite
> slow if a file is large enough. (I do not consider it huge however.)
> There is noticeable delay before the prompt for property name appears in
> minibuffer. Accidentally I have noticed that sometimes it is fast.
>
> Problem appears after first jump using C-u C-c C-j `org-goto'. Actually
> a jump is not required, it is enough to get jump (refile) targets.
It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb
file ruin performance of regexps in that buffer. I would expect either
no influence of markers at all since search is a read-only operation or
some negligible penalty proportional to fraction of markers in relation
to total amount of characters.
Look at the numbers for nm-test that is just some regexp searches from
`org-buffer-property-keys'. Performance is restored after clearing of
the cache.
#+begin_src elisp
(defun nm-tst ()
(org-with-wide-buffer
(goto-char (point-min))
(while (re-search-forward org-property-start-re nil t)
(org-get-property-block)
(outline-next-heading))
nil))
(org-refile-cache-clear)
(let ((org-refile-targets '((nil :maxlevel . 6)))
(org-refile-use-outline-path 'file)
(org-refile-use-cache t)
(iters 10))
(cons
(list (org-version))
(mapcar (lambda (f)
(append (list f) (benchmark-run iters (funcall f))))
(list #'nm-tst ; #'org-buffer-property-keys
#'org-refile-get-targets
#'nm-tst ; #'org-buffer-property-keys
#'org-refile-cache-clear
#'nm-tst))))
#+end_src
#+RESULTS:
| 9.4.6 | | | |
| nm-tst | 0.044981942 | 0 | 0.0 |
| org-refile-get-targets | 1.748572638 | 8 | 0.3876601379999993 |
| nm-tst | 0.393022201 | 0 | 0.0 |
| org-refile-cache-clear | 0.001787078 | 0 | 0.0 |
| nm-tst | 0.035731803 | 0 | 0.0 |
It seems, it is better to drop org-refile-cache completely. It has some
design problems making it rather fragile. At the same time a patch can
be applied that improves performance of `org-refile-get-targets' by a
factor of at least 2 for (org-refile-use-outline-path 'file).
Emacs-26.3
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [BUG] org-goto slows down org-set-property
2021-07-11 9:25 ` Maxim Nikulin
@ 2021-07-11 11:49 ` Ihor Radchenko
2021-07-11 12:55 ` Maxim Nikulin
2024-07-07 12:38 ` Ihor Radchenko
1 sibling, 1 reply; 11+ messages in thread
From: Ihor Radchenko @ 2021-07-11 11:49 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: emacs-orgmode
Maxim Nikulin <manikulin@gmail.com> writes:
> It seems, it is better to drop org-refile-cache completely. It has some
> design problems making it rather fragile. At the same time a patch can
> be applied that improves performance of `org-refile-get-targets' by a
> factor of at least 2 for (org-refile-use-outline-path 'file).
I tend agree. I got rid of org-refile-cache some time ago in favour of
org-ql. The performance is much better subjectively, which indicates
that alternative caching mechanisms could be much better.
Your test on my 12M chars 18k headings org file (Emacs master, Org mode
master). I cannot use org-set-property even without org-refile-cache :(
#+RESULTS[e3fd8ffd9dd1a1d1cb8cf2ab3927bf647e944dd8]:
| 9.4.6 | | | |
| nm-tst | 13.638543295000002 | 4 | 1.9903503830000204 |
| org-refile-get-targets | 5.355179174 | 2 | 0.97954138099999 |
| nm-tst | 22.599429478 | 4 | 2.0785714209999924 |
| org-refile-cache-clear | 0.003903494 | 0 | 0.0 |
| nm-tst | 13.969909913 | 4 | 2.06943181500003 |
Best,
Ihor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 11:49 ` Ihor Radchenko
@ 2021-07-11 12:55 ` Maxim Nikulin
2021-07-11 14:20 ` Ihor Radchenko
0 siblings, 1 reply; 11+ messages in thread
From: Maxim Nikulin @ 2021-07-11 12:55 UTC (permalink / raw)
To: emacs-orgmode
On 11/07/2021 18:49, Ihor Radchenko wrote:
>
> Your test on my 12M chars 18k headings org file (Emacs master, Org mode
> master). I cannot use org-set-property even without org-refile-cache :(
>
> | nm-tst | 13.969909913 | 4 | 2.06943181500003 |
Certainly 13 seconds is no go especially for interactive usage. I use
`org-set-property' to add CUSTOM_ID when necessary. I do not like purely
random ID. Do you have an alternative for adding properties?
Unfortunately `org-set-property-and-value' C-c C-x P does not allow to
set even fixed list of options for completion.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 12:55 ` Maxim Nikulin
@ 2021-07-11 14:20 ` Ihor Radchenko
2021-07-11 15:07 ` Maxim Nikulin
0 siblings, 1 reply; 11+ messages in thread
From: Ihor Radchenko @ 2021-07-11 14:20 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: emacs-orgmode
Maxim Nikulin <manikulin@gmail.com> writes:
> Certainly 13 seconds is no go especially for interactive usage.
Yeah. So, I do not use it for now until me or someone else fixes the
issue.
> Do you have an alternative for adding properties?
> Unfortunately `org-set-property-and-value' C-c C-x P does not allow to
> set even fixed list of options for completion.
For now, I just edit the properties drawer manually or use custom
setters utilising org-set-property.
You can always generate a list of options using `completing-read'.
Example:
(completing-read
"Category: "
(org-ql-select (org-agenda-files t t)
'(property "CATEGORY")
:action (lambda () (substring-no-properties (org-entry-get (point) "CATEGORY")))))
> I use
> `org-set-property' to add CUSTOM_ID when necessary. I do not like purely
> random ID.
You may want to customise `org-id-method'. Also, I often use
`org-capture-ref-generate-key-human-readable' from my org-capture-ref
package [1].
[1] https://github.com/yantar92/org-capture-ref
Best,
Ihor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 14:20 ` Ihor Radchenko
@ 2021-07-11 15:07 ` Maxim Nikulin
2021-07-11 16:00 ` Ihor Radchenko
0 siblings, 1 reply; 11+ messages in thread
From: Maxim Nikulin @ 2021-07-11 15:07 UTC (permalink / raw)
To: emacs-orgmode
On 11/07/2021 21:20, Ihor Radchenko wrote:
> Maxim Nikulin writes > | nm-tst | 22.599429478 | 4 | 2.0785714209999924 |
> | org-refile-cache-clear | 0.003903494 | 0 | 0.0 |
> | nm-tst | 13.969909913 | 4 | 2.06943181500003 |
>
>> Certainly 13 seconds is no go especially for interactive usage.
>
> Yeah. So, I do not use it for now until me or someone else fixes the
> issue.
Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
have twice more markers in the buffer than `org-refile-get-targets'
usually generates. If my guess is correct it may be necessary to file a
bug that org-ql or some other package abuses markers by generation to
much of them.
P.S. Maybe generating of custom IDs automatically is a viable idea.
Before I have not considered such variant believing that they would be
noticeably longer and would have weaker associations.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 15:07 ` Maxim Nikulin
@ 2021-07-11 16:00 ` Ihor Radchenko
2021-07-12 14:59 ` Maxim Nikulin
0 siblings, 1 reply; 11+ messages in thread
From: Ihor Radchenko @ 2021-07-11 16:00 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: emacs-orgmode
Maxim Nikulin <manikulin@gmail.com> writes:
> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
> have twice more markers in the buffer than `org-refile-get-targets'
> usually generates. If my guess is correct it may be necessary to file a
> bug that org-ql or some other package abuses markers by generation to
> much of them.
With emacs -Q things are twice better, though still not good enough:
#+RESULTS[b9f5ebaf91a21d8da95b8840c70f6d9aeb2ab3ae]:
| 9.4.4 | | | |
| nm-tst | 5.572397445999999 | 38 | 1.0039881250000002 |
| org-refile-get-targets | 13.029634993 | 53 | 1.3204672779999997 |
| nm-tst | 24.142289434000002 | 25 | 0.7106094290000002 |
| org-refile-cache-clear | 0.00502349 | 0 | 0.0 |
| nm-tst | 5.043277087 | 38 | 0.8292470489999997 |
org-ql might be one of the reasons of slowdown. Without it, the nm-tst
after clearing the cache is slightly faster (was 13 sec):
#+RESULTS[e5ee2bee7b348265a19834d5ca2697618a9bf261]:
| 9.4.6 | | | |
| nm-tst | 10.106726732 | 34 | 3.0993166650000004 |
| org-refile-get-targets | 5.721277742000001 | 19 | 1.6862241439999996 |
| nm-tst | 22.056917422 | 33 | 3.000191514999999 |
| org-refile-cache-clear | 0.001498722 | 0 | 0.0 |
| nm-tst | 10.066160733 | 34 | 3.203417302 |
However, markers are also created by agendas for example. The last
result is also on fresh Emacs. Fresh Emacs tends to be faster in general.
Do you know a good way to find out all the markers associated with a buffer?
> P.S. Maybe generating of custom IDs automatically is a viable idea.
> Before I have not considered such variant believing that they would be
> noticeably longer and would have weaker associations.
Some examples of autogenerated IDs using bibtex-autokey:
du-2021-recen-studies-in-mechan-proper-e4d
Youtube-veritasium-2021-you-cant-prove-every-thats-true-346
schuh-2005-quant-insig-into-disloc-nucleat-a1d
Karl-Voit-voit-2020-draft-workf-advan-projec-manag
You can judge for yourself if it is too long or not. Also, the length
can be customised as described in bibtex-generate-autokey docstring.
Best,
Ihor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 16:00 ` Ihor Radchenko
@ 2021-07-12 14:59 ` Maxim Nikulin
2021-07-13 14:21 ` John Kitchin
0 siblings, 1 reply; 11+ messages in thread
From: Maxim Nikulin @ 2021-07-12 14:59 UTC (permalink / raw)
To: emacs-orgmode
On 11/07/2021 23:00, Ihor Radchenko wrote:
> Maxim Nikulin writes:
>
>> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
>> have twice more markers in the buffer than `org-refile-get-targets'
>> usually generates. If my guess is correct it may be necessary to file a
>> bug that org-ql or some other package abuses markers by generation to
>> much of them.
>
> With emacs -Q things are twice better, though still not good enough:
...
> org-ql might be one of the reasons of slowdown. Without it, the nm-tst
> after clearing the cache is slightly faster (was 13 sec):
13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So
contribution of org-ql is not major one, but it is still significant.
> However, markers are also created by agendas for example. The last
> result is also on fresh Emacs. Fresh Emacs tends to be faster in general.
>
> Do you know a good way to find out all the markers associated with a buffer?
I was going to ask you, but forgot to do it. Perhaps it is impossible by
design.
(length org-agenda-markers)
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-12 14:59 ` Maxim Nikulin
@ 2021-07-13 14:21 ` John Kitchin
0 siblings, 0 replies; 11+ messages in thread
From: John Kitchin @ 2021-07-13 14:21 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: org-mode-email
[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]
This link (
https://stackoverflow.com/questions/36926513/are-there-elisp-functions-that-list-markers-in-a-given-buffer)
suggests there is not a good way to find all the markers associated with a
buffer.
John
-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
On Mon, Jul 12, 2021 at 11:01 AM Maxim Nikulin <manikulin@gmail.com> wrote:
> On 11/07/2021 23:00, Ihor Radchenko wrote:
> > Maxim Nikulin writes:
> >
> >> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
> >> have twice more markers in the buffer than `org-refile-get-targets'
> >> usually generates. If my guess is correct it may be necessary to file a
> >> bug that org-ql or some other package abuses markers by generation to
> >> much of them.
> >
> > With emacs -Q things are twice better, though still not good enough:
> ...
> > org-ql might be one of the reasons of slowdown. Without it, the nm-tst
> > after clearing the cache is slightly faster (was 13 sec):
>
> 13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So
> contribution of org-ql is not major one, but it is still significant.
>
> > However, markers are also created by agendas for example. The last
> > result is also on fresh Emacs. Fresh Emacs tends to be faster in general.
> >
> > Do you know a good way to find out all the markers associated with a
> buffer?
>
> I was going to ask you, but forgot to do it. Perhaps it is impossible by
> design.
>
> (length org-agenda-markers)
>
>
>
[-- Attachment #2: Type: text/html, Size: 2503 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-11 9:25 ` Maxim Nikulin
2021-07-11 11:49 ` Ihor Radchenko
@ 2024-07-07 12:38 ` Ihor Radchenko
1 sibling, 0 replies; 11+ messages in thread
From: Ihor Radchenko @ 2024-07-07 12:38 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: emacs-orgmode
Maxim Nikulin <manikulin@gmail.com> writes:
> It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb
> file ruin performance of regexps in that buffer. I would expect either
> no influence of markers at all since search is a read-only operation or
> some negligible penalty proportional to fraction of markers in relation
> to total amount of characters.
>
> Look at the numbers for nm-test that is just some regexp searches from
> `org-buffer-property-keys'. Performance is restored after clearing of
> the cache.
> ...
This is no longer reproducible on the latest Emacs master, after a
couple of improvements in the marker handling.
Canceled.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] org-goto slows down org-set-property
2021-07-10 15:47 [BUG] org-goto slows down org-set-property Maxim Nikulin
2021-07-11 9:25 ` Maxim Nikulin
@ 2024-10-13 11:45 ` Ihor Radchenko
1 sibling, 0 replies; 11+ messages in thread
From: Ihor Radchenko @ 2024-10-13 11:45 UTC (permalink / raw)
To: Maxim Nikulin; +Cc: emacs-orgmode
Maxim Nikulin <manikulin@gmail.com> writes:
> It is rather annoying that C-c C-x p `org-set-property' may be quite
> slow if a file is large enough. (I do not consider it huge however.)
> There is noticeable delay before the prompt for property name appears in
> minibuffer. Accidentally I have noticed that sometimes it is fast.
>
> Problem appears after first jump using C-u C-c C-j `org-goto'. Actually
> a jump is not required, it is enough to get jump (refile) targets.
>
> I hope, the following numbers are convincing enough. Second attempt to
> get list of properties is much slower than the first one.
> ...
This is no longer a problem in the newer Emacs versions.
Canceled.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-10-13 11:43 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-07-10 15:47 [BUG] org-goto slows down org-set-property Maxim Nikulin
2021-07-11 9:25 ` Maxim Nikulin
2021-07-11 11:49 ` Ihor Radchenko
2021-07-11 12:55 ` Maxim Nikulin
2021-07-11 14:20 ` Ihor Radchenko
2021-07-11 15:07 ` Maxim Nikulin
2021-07-11 16:00 ` Ihor Radchenko
2021-07-12 14:59 ` Maxim Nikulin
2021-07-13 14:21 ` John Kitchin
2024-07-07 12:38 ` Ihor Radchenko
2024-10-13 11:45 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).