It is rather annoying that C-c C-x p `org-set-property' may be quite slow if a file is large enough. (I do not consider it huge however.) There is noticeable delay before the prompt for property name appears in minibuffer. Accidentally I have noticed that sometimes it is fast. Problem appears after first jump using C-u C-c C-j `org-goto'. Actually a jump is not required, it is enough to get jump (refile) targets. I hope, the following numbers are convincing enough. Second attempt to get list of properties is much slower than the first one. #+begin_src elisp (require 'org-refile) (let ((org-refile-targets '((nil :maxlevel . 6))) (org-refile-use-outline-path 'file) (org-refile-use-cache t) (iters 10)) (cons (list (org-version)) (mapcar (lambda (f) (append (list f) (benchmark-run iters (funcall f)))) (list #'org-buffer-property-keys #'org-refile-get-targets #'org-buffer-property-keys)))) #+end_src git master: #+RESULTS: | 9.4.6 | | | | | org-buffer-property-keys | 0.065695 | 0 | 0.0 | | org-refile-get-targets | 1.836473 | 12 | 0.39498671199999985 | | org-buffer-property-keys | 0.563227 | 0 | 0.0 | elpa-org system package for Ubuntu-20.04 is even more slower: #+RESULTS: | 9.3.1 | | | | | org-buffer-property-keys | 0.242574778 | 0 | 0.0 | | org-refile-get-targets | 2.669112479 | 13 | 0.285175928 | | org-buffer-property-keys | 1.231223427 | 0 | 0.0 | The effect is noticeable for the doc/org-manual.org file from org sources though absolute numbers are well below delay that makes interaction incomfortable. In the configuration above effect of `org-refile-use-cache' is crucial, contribution of `org-refile-use-outline-path' is noticeable but not so important. My expectation is that neither `org-refile-use-cache' nor `org-goto' command should make adding property slower.
On 10/07/2021 22:47, Maxim Nikulin wrote:
> It is rather annoying that C-c C-x p `org-set-property' may be quite
> slow if a file is large enough. (I do not consider it huge however.)
> There is noticeable delay before the prompt for property name appears in
> minibuffer. Accidentally I have noticed that sometimes it is fast.
>
> Problem appears after first jump using C-u C-c C-j `org-goto'. Actually
> a jump is not required, it is enough to get jump (refile) targets.
It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb
file ruin performance of regexps in that buffer. I would expect either
no influence of markers at all since search is a read-only operation or
some negligible penalty proportional to fraction of markers in relation
to total amount of characters.
Look at the numbers for nm-test that is just some regexp searches from
`org-buffer-property-keys'. Performance is restored after clearing of
the cache.
#+begin_src elisp
(defun nm-tst ()
(org-with-wide-buffer
(goto-char (point-min))
(while (re-search-forward org-property-start-re nil t)
(org-get-property-block)
(outline-next-heading))
nil))
(org-refile-cache-clear)
(let ((org-refile-targets '((nil :maxlevel . 6)))
(org-refile-use-outline-path 'file)
(org-refile-use-cache t)
(iters 10))
(cons
(list (org-version))
(mapcar (lambda (f)
(append (list f) (benchmark-run iters (funcall f))))
(list #'nm-tst ; #'org-buffer-property-keys
#'org-refile-get-targets
#'nm-tst ; #'org-buffer-property-keys
#'org-refile-cache-clear
#'nm-tst))))
#+end_src
#+RESULTS:
| 9.4.6 | | | |
| nm-tst | 0.044981942 | 0 | 0.0 |
| org-refile-get-targets | 1.748572638 | 8 | 0.3876601379999993 |
| nm-tst | 0.393022201 | 0 | 0.0 |
| org-refile-cache-clear | 0.001787078 | 0 | 0.0 |
| nm-tst | 0.035731803 | 0 | 0.0 |
It seems, it is better to drop org-refile-cache completely. It has some
design problems making it rather fragile. At the same time a patch can
be applied that improves performance of `org-refile-get-targets' by a
factor of at least 2 for (org-refile-use-outline-path 'file).
Emacs-26.3
Maxim Nikulin <manikulin@gmail.com> writes:
> It seems, it is better to drop org-refile-cache completely. It has some
> design problems making it rather fragile. At the same time a patch can
> be applied that improves performance of `org-refile-get-targets' by a
> factor of at least 2 for (org-refile-use-outline-path 'file).
I tend agree. I got rid of org-refile-cache some time ago in favour of
org-ql. The performance is much better subjectively, which indicates
that alternative caching mechanisms could be much better.
Your test on my 12M chars 18k headings org file (Emacs master, Org mode
master). I cannot use org-set-property even without org-refile-cache :(
#+RESULTS[e3fd8ffd9dd1a1d1cb8cf2ab3927bf647e944dd8]:
| 9.4.6 | | | |
| nm-tst | 13.638543295000002 | 4 | 1.9903503830000204 |
| org-refile-get-targets | 5.355179174 | 2 | 0.97954138099999 |
| nm-tst | 22.599429478 | 4 | 2.0785714209999924 |
| org-refile-cache-clear | 0.003903494 | 0 | 0.0 |
| nm-tst | 13.969909913 | 4 | 2.06943181500003 |
Best,
Ihor
On 11/07/2021 18:49, Ihor Radchenko wrote:
>
> Your test on my 12M chars 18k headings org file (Emacs master, Org mode
> master). I cannot use org-set-property even without org-refile-cache :(
>
> | nm-tst | 13.969909913 | 4 | 2.06943181500003 |
Certainly 13 seconds is no go especially for interactive usage. I use
`org-set-property' to add CUSTOM_ID when necessary. I do not like purely
random ID. Do you have an alternative for adding properties?
Unfortunately `org-set-property-and-value' C-c C-x P does not allow to
set even fixed list of options for completion.
Maxim Nikulin <manikulin@gmail.com> writes: > Certainly 13 seconds is no go especially for interactive usage. Yeah. So, I do not use it for now until me or someone else fixes the issue. > Do you have an alternative for adding properties? > Unfortunately `org-set-property-and-value' C-c C-x P does not allow to > set even fixed list of options for completion. For now, I just edit the properties drawer manually or use custom setters utilising org-set-property. You can always generate a list of options using `completing-read'. Example: (completing-read "Category: " (org-ql-select (org-agenda-files t t) '(property "CATEGORY") :action (lambda () (substring-no-properties (org-entry-get (point) "CATEGORY"))))) > I use > `org-set-property' to add CUSTOM_ID when necessary. I do not like purely > random ID. You may want to customise `org-id-method'. Also, I often use `org-capture-ref-generate-key-human-readable' from my org-capture-ref package [1]. [1] https://github.com/yantar92/org-capture-ref Best, Ihor
On 11/07/2021 21:20, Ihor Radchenko wrote:
> Maxim Nikulin writes > | nm-tst | 22.599429478 | 4 | 2.0785714209999924 |
> | org-refile-cache-clear | 0.003903494 | 0 | 0.0 |
> | nm-tst | 13.969909913 | 4 | 2.06943181500003 |
>
>> Certainly 13 seconds is no go especially for interactive usage.
>
> Yeah. So, I do not use it for now until me or someone else fixes the
> issue.
Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
have twice more markers in the buffer than `org-refile-get-targets'
usually generates. If my guess is correct it may be necessary to file a
bug that org-ql or some other package abuses markers by generation to
much of them.
P.S. Maybe generating of custom IDs automatically is a viable idea.
Before I have not considered such variant believing that they would be
noticeably longer and would have weaker associations.
Maxim Nikulin <manikulin@gmail.com> writes: > Ihor, did you get that numbers using emacs -Q? If not, I suspect you may > have twice more markers in the buffer than `org-refile-get-targets' > usually generates. If my guess is correct it may be necessary to file a > bug that org-ql or some other package abuses markers by generation to > much of them. With emacs -Q things are twice better, though still not good enough: #+RESULTS[b9f5ebaf91a21d8da95b8840c70f6d9aeb2ab3ae]: | 9.4.4 | | | | | nm-tst | 5.572397445999999 | 38 | 1.0039881250000002 | | org-refile-get-targets | 13.029634993 | 53 | 1.3204672779999997 | | nm-tst | 24.142289434000002 | 25 | 0.7106094290000002 | | org-refile-cache-clear | 0.00502349 | 0 | 0.0 | | nm-tst | 5.043277087 | 38 | 0.8292470489999997 | org-ql might be one of the reasons of slowdown. Without it, the nm-tst after clearing the cache is slightly faster (was 13 sec): #+RESULTS[e5ee2bee7b348265a19834d5ca2697618a9bf261]: | 9.4.6 | | | | | nm-tst | 10.106726732 | 34 | 3.0993166650000004 | | org-refile-get-targets | 5.721277742000001 | 19 | 1.6862241439999996 | | nm-tst | 22.056917422 | 33 | 3.000191514999999 | | org-refile-cache-clear | 0.001498722 | 0 | 0.0 | | nm-tst | 10.066160733 | 34 | 3.203417302 | However, markers are also created by agendas for example. The last result is also on fresh Emacs. Fresh Emacs tends to be faster in general. Do you know a good way to find out all the markers associated with a buffer? > P.S. Maybe generating of custom IDs automatically is a viable idea. > Before I have not considered such variant believing that they would be > noticeably longer and would have weaker associations. Some examples of autogenerated IDs using bibtex-autokey: du-2021-recen-studies-in-mechan-proper-e4d Youtube-veritasium-2021-you-cant-prove-every-thats-true-346 schuh-2005-quant-insig-into-disloc-nucleat-a1d Karl-Voit-voit-2020-draft-workf-advan-projec-manag You can judge for yourself if it is too long or not. Also, the length can be customised as described in bibtex-generate-autokey docstring. Best, Ihor
On 11/07/2021 23:00, Ihor Radchenko wrote: > Maxim Nikulin writes: > >> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may >> have twice more markers in the buffer than `org-refile-get-targets' >> usually generates. If my guess is correct it may be necessary to file a >> bug that org-ql or some other package abuses markers by generation to >> much of them. > > With emacs -Q things are twice better, though still not good enough: ... > org-ql might be one of the reasons of slowdown. Without it, the nm-tst > after clearing the cache is slightly faster (was 13 sec): 13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So contribution of org-ql is not major one, but it is still significant. > However, markers are also created by agendas for example. The last > result is also on fresh Emacs. Fresh Emacs tends to be faster in general. > > Do you know a good way to find out all the markers associated with a buffer? I was going to ask you, but forgot to do it. Perhaps it is impossible by design. (length org-agenda-markers)
[-- Attachment #1: Type: text/plain, Size: 1649 bytes --] This link ( https://stackoverflow.com/questions/36926513/are-there-elisp-functions-that-list-markers-in-a-given-buffer) suggests there is not a good way to find all the markers associated with a buffer. John ----------------------------------- Professor John Kitchin (he/him/his) Doherty Hall A207F Department of Chemical Engineering Carnegie Mellon University Pittsburgh, PA 15213 412-268-7803 @johnkitchin http://kitchingroup.cheme.cmu.edu On Mon, Jul 12, 2021 at 11:01 AM Maxim Nikulin <manikulin@gmail.com> wrote: > On 11/07/2021 23:00, Ihor Radchenko wrote: > > Maxim Nikulin writes: > > > >> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may > >> have twice more markers in the buffer than `org-refile-get-targets' > >> usually generates. If my guess is correct it may be necessary to file a > >> bug that org-ql or some other package abuses markers by generation to > >> much of them. > > > > With emacs -Q things are twice better, though still not good enough: > ... > > org-ql might be one of the reasons of slowdown. Without it, the nm-tst > > after clearing the cache is slightly faster (was 13 sec): > > 13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So > contribution of org-ql is not major one, but it is still significant. > > > However, markers are also created by agendas for example. The last > > result is also on fresh Emacs. Fresh Emacs tends to be faster in general. > > > > Do you know a good way to find out all the markers associated with a > buffer? > > I was going to ask you, but forgot to do it. Perhaps it is impossible by > design. > > (length org-agenda-markers) > > > [-- Attachment #2: Type: text/html, Size: 2503 bytes --]