* [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
@ 2020-04-24 6:55 Ihor Radchenko
2020-04-24 8:02 ` Nicolas Goaziou
0 siblings, 1 reply; 192+ messages in thread
From: Ihor Radchenko @ 2020-04-24 6:55 UTC (permalink / raw)
To: emacs-orgmode
Emacs becomes very slow when opening and moving around huge org files
with many drawers. I have reported this issue last year in bug-gnu-emacs
[1] and there have been other reports on the same problem in the
internet [2]. You can easily see this problem using the attached file if
you try to move down the lines when all the headings are folded. Moving
a single line down may take over 10 seconds in the file.
According to the reply to my initial emacs bug report [1], the reasons
of performance degradation is huge number of overlays created by org in
the PROPERTY and LOGBOOK drawers. Emacs must loop over all those
overlays every time it calculates where the next visible line is
located. So, one way to improve the performance would be reducing the
number of overlays.
I have been looking into usage of overlays in the org-mode code recently
and tried to redefine org-flag-region to use text properties instead of
overlays:
#+begin_src emacs-lisp
(defun org-flag-region (from to flag spec)
"Hide or show lines from FROM to TO, according to FLAG.
SPEC is the invisibility spec, as a symbol."
(pcase spec
;; outlines must still use overlays because they rely on
;; 'reveal-toggle-invisible feature from reveal.el
;; That only works for overlays
('outline
(remove-overlays from to 'invisible spec)
;; Use `front-advance' since text right before to the beginning of
;; the overlay belongs to the visible line than to the contents.
(when flag
(let ((o (make-overlay from to nil 'front-advance)))
(overlay-put o 'evaporate t)
(overlay-put o 'invisible spec)
(overlay-put o 'isearch-open-invisible #'delete-overlay))))
(_
(let ((inhibit-modification-hooks t))
(remove-text-properties from to '(invisible nil))
;; Use `front-advance' since text right before to the beginning of
;; the overlay belongs to the visible line than to the contents.
(when flag
(put-text-property from to 'rear-non-sticky t)
(put-text-property from to 'front-sticky t)
(put-text-property from to 'invisible spec)
;; no idea if 'isearch-open-invisible is needed for text
;; properties
;; (overlay-put o 'isearch-open-invisible #'delete-overlay)
)))))
#+end_src
To my surprise, the patch did not break org to unusable state and
the performance on the sample org file [3] improved drastically. You can
try by yourself!
However, this did introduce some visual glitches with drawer display.
Though drawers can still be folded/unfolded with <tab>, they are not
folded on org-mode startup for some reason (can be fixed by running
(org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
are unfolded for no apparent reason sometimes. A blind guess is that it
is something to do with lack of 'isearch-open-invisible, which I am not
sure how to set via text properties.
Any thoughts about the use of text properties or about the patch
suggestion are welcome.
Best,
Ihor
[1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/msg01387.html
[2] https://www.reddit.com/r/orgmode/comments/e9p84n/scaling_org_better_to_use_more_medsize_files_or/
[3] See the attached org file in my Emacs bug report: https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/txte6kQp35VOm.txt
--
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
^ permalink raw reply [flat|nested] 192+ messages in thread* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-24 6:55 [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko @ 2020-04-24 8:02 ` Nicolas Goaziou 2020-04-25 0:29 ` stardiviner 2020-04-26 16:04 ` Ihor Radchenko 0 siblings, 2 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-04-24 8:02 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > To my surprise, the patch did not break org to unusable state and > the performance on the sample org file [3] improved drastically. You can > try by yourself! It is not a surprise, really. Text properties are much faster than overlays, and very close to them features-wise. They are a bit more complex to handle, however. > However, this did introduce some visual glitches with drawer display. > Though drawers can still be folded/unfolded with <tab>, they are not > folded on org-mode startup for some reason (can be fixed by running > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) > are unfolded for no apparent reason sometimes. A blind guess is that it > is something to do with lack of 'isearch-open-invisible, which I am not > sure how to set via text properties. You cannot. You may however mimic it with `cursor-sensor-functions' text property. These assume Cursor Sensor minor mode is active, tho. I haven't tested it, but I assume it would slow down text properties a bit, too, but hopefully not as much as overlays. Note there are clear advantages using text properties. For example, when you move contents around, text properties are preserved. So there's no more need for the `org-cycle-hide-drawer' dance, i.e., it is not necessary anymore to re-hide drawers. > Any thoughts about the use of text properties or about the patch > suggestion are welcome. Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth experimenting with `cursor-sensor-functions'. We could also use text properties for property drawers, and overlays for regular ones. This might give us a reasonable speed-up with an acceptable feature trade-off. Anyway, the real fix should come from Emacs itself. There are ways to make overlays faster. These ways have already been discussed on the Emacs devel mailing list, but no one implemented them. It is a bit sad that we have to find workarounds for that. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-24 8:02 ` Nicolas Goaziou @ 2020-04-25 0:29 ` stardiviner 2020-04-26 16:04 ` Ihor Radchenko 1 sibling, 0 replies; 192+ messages in thread From: stardiviner @ 2020-04-25 0:29 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode, Ihor Radchenko -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> To my surprise, the patch did not break org to unusable state and >> the performance on the sample org file [3] improved drastically. You can >> try by yourself! > > It is not a surprise, really. Text properties are much faster than > overlays, and very close to them features-wise. They are a bit more > complex to handle, however. > >> However, this did introduce some visual glitches with drawer display. >> Though drawers can still be folded/unfolded with <tab>, they are not >> folded on org-mode startup for some reason (can be fixed by running >> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) >> are unfolded for no apparent reason sometimes. A blind guess is that it >> is something to do with lack of 'isearch-open-invisible, which I am not >> sure how to set via text properties. > > You cannot. You may however mimic it with `cursor-sensor-functions' text > property. These assume Cursor Sensor minor mode is active, tho. > I haven't tested it, but I assume it would slow down text properties > a bit, too, but hopefully not as much as overlays. > > Note there are clear advantages using text properties. For example, when > you move contents around, text properties are preserved. So there's no > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > necessary anymore to re-hide drawers. > >> Any thoughts about the use of text properties or about the patch >> suggestion are welcome. > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > experimenting with `cursor-sensor-functions'. > > We could also use text properties for property drawers, and overlays for > regular ones. This might give us a reasonable speed-up with an > acceptable feature trade-off. That's great, making Org Mode faster will be great. (Even thought I have not found big performance problem on Org Mode yet.) I like Thor's try. This indeed is is an acceptable feature trade-off, if only related to `isearch-open-invisible'. > > Anyway, the real fix should come from Emacs itself. There are ways to > make overlays faster. These ways have already been discussed on the > Emacs devel mailing list, but no one implemented them. It is a bit sad > that we have to find workarounds for that. > > Regards, - -- [ stardiviner ] I try to make every word tell the meaning what I want to express. Blog: https://stardiviner.github.io/ IRC(freenode): stardiviner, Matrix: stardiviner GPG: F09F650D7D674819892591401B5DF1C95AE89AC3 -----BEGIN PGP SIGNATURE----- iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl6jhG0UHG51bWJjaGls ZEBnbWFpbC5jb20ACgkQG13xyVromsPHDAf+OVnhOq5H5MYm1/RK+9xSzwAT6qc8 ajSNVNzI31q6CIesvO65GoiZ3Rpaiq/O31B9JQ1mTyXvyX81tFecKrDpsrqIc/bR Xo3Z4dCXzCbRKD1861t4tcphtPBk+rABpl83YpXafYNDKHnp2MuWSheV0ogF7LYd 6HWCl9D351onGAHGcebXEUTvvDiqLGx5qVnrpjomH00uCj5RoSI4cpdzXydBcIYY B6lDvsat8AHhvbPXqJc4PHOd4hPtNVehWyPfOGaAXhp/pS0y+c4cJMbHjXCwFCkj r8bUfdK+ZyMubNiboNI9xO8EwINvZLl+C5Lt5siYs/v2mrt1+UiVrxYWTw== =dnH4 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-24 8:02 ` Nicolas Goaziou 2020-04-25 0:29 ` stardiviner @ 2020-04-26 16:04 ` Ihor Radchenko 2020-05-04 16:56 ` Karl Voit ` (2 more replies) 1 sibling, 3 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-04-26 16:04 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > You cannot. You may however mimic it with `cursor-sensor-functions' text > property. These assume Cursor Sensor minor mode is active, tho. > I haven't tested it, but I assume it would slow down text properties > a bit, too, but hopefully not as much as overlays. Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil internally. Anyway, I came up with some workaround, which seems to work (see below). Though it would be better if isearch supported hidden text in addition to overlays. > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > experimenting with `cursor-sensor-functions'. So far, I came up with the following partial solution searching and showing hidden text. ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we ;; cannot even use cursor-sensor-functions as a workaround ;; I used a less ideas approach with advice to isearch-search-string as ;; a workaround (defun org-find-text-property-region (pos prop) "Find a region containing PROP text property around point POS." (require 'org-macs) ;; org-with-point-at (org-with-point-at pos (let* ((beg (and (get-text-property pos prop) pos)) (end beg)) (when beg (setq beg (or (previous-single-property-change pos prop) beg)) (setq end (or (next-single-property-change pos prop) end)) (unless (equal beg end) (cons beg end)))))) ;; :FIXME: re-hide properties when point moves away (define-advice isearch-search-string (:after (&rest _) put-overlay) "Reveal hidden text at point." (when-let ((region (org-find-text-property-region (point) 'invisible))) (with-silent-modifications (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible))) (remove-text-properties (car region) (cdr region) '(invisible nil)))) ;; this seems to be unstable, but I cannot figure out why (defun org-restore-invisibility-specs (&rest _) "" (let ((pos (point-min))) (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max)) (when-let ((region (org-find-text-property-region pos 'org-invisible))) (with-silent-modifications (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible)) (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) (add-hook 'post-command-hook #'org-restore-invisibility-specs) (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." (pcase spec ('outline (remove-overlays from to 'invisible spec) ;; Use `front-advance' since text right before to the beginning of ;; the overlay belongs to the visible line than to the contents. (when flag (let ((o (make-overlay from to nil 'front-advance))) (overlay-put o 'evaporate t) (overlay-put o 'invisible spec) (overlay-put o 'isearch-open-invisible #'delete-overlay)))) (_ (with-silent-modifications (remove-text-properties from to '(invisible nil)) (when flag (put-text-property from to 'invisible spec) ))))) ;; This normally deletes invisible text property. We do not want this now. (defun org-unfontify-region (beg end &optional _maybe_loudly) "Remove fontification and activation overlays from links." (font-lock-default-unfontify-region beg end) (let* ((buffer-undo-list t) (inhibit-read-only t) (inhibit-point-motion-hooks t) (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) (remove-text-properties beg end '(mouse-face t keymap t org-linked-text t ;; Do not remove invisible during fontification ;; invisible t intangible t org-emphasis t)) (org-remove-font-lock-display-properties beg end))) > Anyway, the real fix should come from Emacs itself. There are ways to > make overlays faster. These ways have already been discussed on the > Emacs devel mailing list, but no one implemented them. It is a bit sad > that we have to find workarounds for that. I guess that it is a very old story starting from the times when XEmacs was a thing [1]. I recently heard about binary tree implementation of overlays (there should be a branch in emacs git repo) [2], but there was no update on that branch for a while. So, I do not have much hope on Emacs implementing efficient overlay access in the near future. (And I have problems with huge org files already). [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> To my surprise, the patch did not break org to unusable state and >> the performance on the sample org file [3] improved drastically. You can >> try by yourself! > > It is not a surprise, really. Text properties are much faster than > overlays, and very close to them features-wise. They are a bit more > complex to handle, however. > >> However, this did introduce some visual glitches with drawer display. >> Though drawers can still be folded/unfolded with <tab>, they are not >> folded on org-mode startup for some reason (can be fixed by running >> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) >> are unfolded for no apparent reason sometimes. A blind guess is that it >> is something to do with lack of 'isearch-open-invisible, which I am not >> sure how to set via text properties. > > You cannot. You may however mimic it with `cursor-sensor-functions' text > property. These assume Cursor Sensor minor mode is active, tho. > I haven't tested it, but I assume it would slow down text properties > a bit, too, but hopefully not as much as overlays. > > Note there are clear advantages using text properties. For example, when > you move contents around, text properties are preserved. So there's no > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > necessary anymore to re-hide drawers. > >> Any thoughts about the use of text properties or about the patch >> suggestion are welcome. > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > experimenting with `cursor-sensor-functions'. > > We could also use text properties for property drawers, and overlays for > regular ones. This might give us a reasonable speed-up with an > acceptable feature trade-off. > > Anyway, the real fix should come from Emacs itself. There are ways to > make overlays faster. These ways have already been discussed on the > Emacs devel mailing list, but no one implemented them. It is a bit sad > that we have to find workarounds for that. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-26 16:04 ` Ihor Radchenko @ 2020-05-04 16:56 ` Karl Voit 2020-05-07 7:18 ` Karl Voit 2020-05-09 15:43 ` Ihor Radchenko 2020-05-07 11:04 ` Christian Heinrich 2020-05-08 16:38 ` Nicolas Goaziou 2 siblings, 2 replies; 192+ messages in thread From: Karl Voit @ 2020-05-04 16:56 UTC (permalink / raw) To: emacs-orgmode Hi Ihor, * Ihor Radchenko <yantar92@gmail.com> wrote: > > So far, I came up with the following partial solution searching and > showing hidden text. > > (defun org-find-text-property-region (pos prop) > (define-advice isearch-search-string (:after (&rest _) put-overlay) > (defun org-restore-invisibility-specs (&rest _) > (add-hook 'post-command-hook #'org-restore-invisibility-specs) > (defun org-flag-region (from to flag spec) > (defun org-unfontify-region (beg end &optional _maybe_loudly) After a couple of hours working with these patches, my feedback is very positive. Besides some visual glitches when creating a new heading with org-expiry-insinuate activated (which automatically adds :CREATED: properties), I could not detect any side-effect so far (will keep testing). The visual glitch looks like that: :PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] X ... with "X" being my character that symbolizes collapsed content. The way it looked without the patch was a simple collapsed property drawer. To me, this is acceptable considering the huge performance gain I got. THANK YOU VERY MUCH! I can't remember where I had this way of working within my large Org files[3] since ages. >> Anyway, the real fix should come from Emacs itself. There are ways to >> make overlays faster. These ways have already been discussed on the >> Emacs devel mailing list, but no one implemented them. It is a bit sad >> that we have to find workarounds for that. > > I guess that it is a very old story starting from the times when XEmacs > was a thing [1]. I recently heard about binary tree implementation of > overlays (there should be a branch in emacs git repo) [2], but there was > no update on that branch for a while. So, I do not have much hope on > Emacs implementing efficient overlay access in the near future. (And I > have problems with huge org files already). I can not express how this also reflects my personal situation. > [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ > [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html [3] https://karl-voit.at/2020/05/03/current-org-files -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/ ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-04 16:56 ` Karl Voit @ 2020-05-07 7:18 ` Karl Voit 2020-05-09 15:43 ` Ihor Radchenko 1 sibling, 0 replies; 192+ messages in thread From: Karl Voit @ 2020-05-07 7:18 UTC (permalink / raw) To: emacs-orgmode Hi, * Karl Voit <devnull@Karl-Voit.at> wrote: > Hi Ihor, > > * Ihor Radchenko <yantar92@gmail.com> wrote: >> >> So far, I came up with the following partial solution searching and >> showing hidden text. >> >> (defun org-find-text-property-region (pos prop) >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> (defun org-restore-invisibility-specs (&rest _) >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> (defun org-flag-region (from to flag spec) >> (defun org-unfontify-region (beg end &optional _maybe_loudly) > > After a couple of hours working with these patches, my feedback is > very positive. Besides some visual glitches when creating a new > heading with org-expiry-insinuate activated (which automatically > adds :CREATED: properties), I could not detect any side-effect so > far (will keep testing). > > The visual glitch looks like that: > >:PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] > X > > ... with "X" being my character that symbolizes collapsed content. > The way it looked without the patch was a simple collapsed property > drawer. Here some hard numbers to demonstrate the impact: my-org-agenda: from 11-16s down to 10 -> not much of a difference helm-org-contacts-refresh-cache: 29-59s down to 2½ -> HUGE Emacs boot time: 50-65s down to 10 -> HUGE Navigating the cursor in large Org files -> HUGE subjective impact >>> Anyway, the real fix should come from Emacs itself. There are ways to >>> make overlays faster. These ways have already been discussed on the >>> Emacs devel mailing list, but no one implemented them. It is a bit sad >>> that we have to find workarounds for that. >> >> I guess that it is a very old story starting from the times when XEmacs >> was a thing [1]. I recently heard about binary tree implementation of >> overlays (there should be a branch in emacs git repo) [2], but there was >> no update on that branch for a while. So, I do not have much hope on >> Emacs implementing efficient overlay access in the near future. (And I >> have problems with huge org files already). > > I can not express how this also reflects my personal situation. > >> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html > > [3] https://karl-voit.at/2020/05/03/current-org-files > -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/ ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-04 16:56 ` Karl Voit 2020-05-07 7:18 ` Karl Voit @ 2020-05-09 15:43 ` Ihor Radchenko 1 sibling, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 15:43 UTC (permalink / raw) To: Karl Voit, emacs-orgmode > The visual glitch looks like that: > > :PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] > X Should be partially fixed in the latest patch I just sent. OLD <<< :PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] NEW >>> :PROPERTIES:X X Best, Ihor Karl Voit <devnull@Karl-Voit.at> writes: > Hi Ihor, > > * Ihor Radchenko <yantar92@gmail.com> wrote: >> >> So far, I came up with the following partial solution searching and >> showing hidden text. >> >> (defun org-find-text-property-region (pos prop) >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> (defun org-restore-invisibility-specs (&rest _) >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> (defun org-flag-region (from to flag spec) >> (defun org-unfontify-region (beg end &optional _maybe_loudly) > > After a couple of hours working with these patches, my feedback is > very positive. Besides some visual glitches when creating a new > heading with org-expiry-insinuate activated (which automatically > adds :CREATED: properties), I could not detect any side-effect so > far (will keep testing). > > The visual glitch looks like that: > > :PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] > X > > ... with "X" being my character that symbolizes collapsed content. > The way it looked without the patch was a simple collapsed property > drawer. > > To me, this is acceptable considering the huge performance gain I > got. > > THANK YOU VERY MUCH! I can't remember where I had this way of > working within my large Org files[3] since ages. > >>> Anyway, the real fix should come from Emacs itself. There are ways to >>> make overlays faster. These ways have already been discussed on the >>> Emacs devel mailing list, but no one implemented them. It is a bit sad >>> that we have to find workarounds for that. >> >> I guess that it is a very old story starting from the times when XEmacs >> was a thing [1]. I recently heard about binary tree implementation of >> overlays (there should be a branch in emacs git repo) [2], but there was >> no update on that branch for a while. So, I do not have much hope on >> Emacs implementing efficient overlay access in the near future. (And I >> have problems with huge org files already). > > I can not express how this also reflects my personal situation. > >> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html > > [3] https://karl-voit.at/2020/05/03/current-org-files > > -- > get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > > get Memacs from https://github.com/novoid/Memacs < > Personal Information Management > http://Karl-Voit.at/tags/pim/ > Emacs-related > http://Karl-Voit.at/tags/emacs/ > > -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-26 16:04 ` Ihor Radchenko 2020-05-04 16:56 ` Karl Voit @ 2020-05-07 11:04 ` Christian Heinrich 2020-05-09 15:46 ` Ihor Radchenko 2020-05-08 16:38 ` Nicolas Goaziou 2 siblings, 1 reply; 192+ messages in thread From: Christian Heinrich @ 2020-05-07 11:04 UTC (permalink / raw) To: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 8181 bytes --] Hi, thanks for your (initial) patch! I traced another error down today and found your code by chance. I tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using outline-show-all which I had to use anyways to deal with my huge file.) I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind sending a patch file? I hope that this ends up in the master branch at some point. Thanks again! Christian On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote: > > You cannot. You may however mimic it with `cursor-sensor-functions' text > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. > > Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil > internally. Anyway, I came up with some workaround, which seems to work > (see below). Though it would be better if isearch supported hidden text > in addition to overlays. > > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > > experimenting with `cursor-sensor-functions'. > > So far, I came up with the following partial solution searching and > showing hidden text. > > ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we > ;; cannot even use cursor-sensor-functions as a workaround > ;; I used a less ideas approach with advice to isearch-search-string as > ;; a workaround > > (defun org-find-text-property-region (pos prop) > "Find a region containing PROP text property around point POS." > (require 'org-macs) ;; org-with-point-at > (org-with-point-at pos > (let* ((beg (and (get-text-property pos prop) pos)) > (end beg)) > (when beg > (setq beg (or (previous-single-property-change pos prop) > beg)) > (setq end (or (next-single-property-change pos prop) > end)) > (unless (equal beg end) > (cons beg end)))))) > > ;; :FIXME: re-hide properties when point moves away > (define-advice isearch-search-string (:after (&rest _) put-overlay) > "Reveal hidden text at point." > (when-let ((region (org-find-text-property-region (point) 'invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) > 'invisible))) > (remove-text-properties (car region) (cdr region) '(invisible nil)))) > > ;; this seems to be unstable, but I cannot figure out why > (defun org-restore-invisibility-specs (&rest _) > "" > (let ((pos (point-min))) > (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point- > max)) > (when-let ((region (org-find-text-property-region pos 'org-invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org- > invisible)) > (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) > > (add-hook 'post-command-hook #'org-restore-invisibility-specs) > > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > (pcase spec > ('outline > (remove-overlays from to 'invisible spec) > ;; Use `front-advance' since text right before to the beginning of > ;; the overlay belongs to the visible line than to the contents. > (when flag > (let ((o (make-overlay from to nil 'front-advance))) > (overlay-put o 'evaporate t) > (overlay-put o 'invisible spec) > (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > (_ > (with-silent-modifications > (remove-text-properties from to '(invisible nil)) > (when flag > (put-text-property from to 'invisible spec) > ))))) > > ;; This normally deletes invisible text property. We do not want this now. > (defun org-unfontify-region (beg end &optional _maybe_loudly) > "Remove fontification and activation overlays from links." > (font-lock-default-unfontify-region beg end) > (let* ((buffer-undo-list t) > (inhibit-read-only t) (inhibit-point-motion-hooks t) > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > (remove-text-properties beg end > '(mouse-face t keymap t org-linked-text t > ;; Do not remove invisible during fontification > > ;; invisible t > intangible t > org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) > > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. > > I guess that it is a very old story starting from the times when XEmacs > was a thing [1]. I recently heard about binary tree implementation of > overlays (there should be a branch in emacs git repo) [2], but there was > no update on that branch for a while. So, I do not have much hope on > Emacs implementing efficient overlay access in the near future. (And I > have problems with huge org files already). > > [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ > [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html > > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > > > Hello, > > > > Ihor Radchenko <yantar92@gmail.com> writes: > > > > > To my surprise, the patch did not break org to unusable state and > > > the performance on the sample org file [3] improved drastically. You can > > > try by yourself! > > > > It is not a surprise, really. Text properties are much faster than > > overlays, and very close to them features-wise. They are a bit more > > complex to handle, however. > > > > > However, this did introduce some visual glitches with drawer display. > > > Though drawers can still be folded/unfolded with <tab>, they are not > > > folded on org-mode startup for some reason (can be fixed by running > > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) > > > are unfolded for no apparent reason sometimes. A blind guess is that it > > > is something to do with lack of 'isearch-open-invisible, which I am not > > > sure how to set via text properties. > > > > You cannot. You may however mimic it with `cursor-sensor-functions' text > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. > > > > Note there are clear advantages using text properties. For example, when > > you move contents around, text properties are preserved. So there's no > > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > > necessary anymore to re-hide drawers. > > > > > Any thoughts about the use of text properties or about the patch > > > suggestion are welcome. > > > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > > experimenting with `cursor-sensor-functions'. > > > > We could also use text properties for property drawers, and overlays for > > regular ones. This might give us a reasonable speed-up with an > > acceptable feature trade-off. > > > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. > > > > Regards, > > > > -- > > Nicolas Goaziou [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-07 11:04 ` Christian Heinrich @ 2020-05-09 15:46 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 15:46 UTC (permalink / raw) To: Christian Heinrich, emacs-orgmode > I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind > sending a patch file? I hope that this ends up in the master branch at some point. I have sent the patch in another email. Will appreciate any feedback. Best, Ihor Christian Heinrich <christian@gladbachcity.de> writes: > Hi, > > thanks for your (initial) patch! I traced another error down today and found your code by chance. I > tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this > patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when > outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using > outline-show-all which I had to use anyways to deal with my huge file.) > > I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind > sending a patch file? I hope that this ends up in the master branch at some point. > > Thanks again! > Christian > > On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote: >> > You cannot. You may however mimic it with `cursor-sensor-functions' text >> > property. These assume Cursor Sensor minor mode is active, tho. >> > I haven't tested it, but I assume it would slow down text properties >> > a bit, too, but hopefully not as much as overlays. >> >> Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil >> internally. Anyway, I came up with some workaround, which seems to work >> (see below). Though it would be better if isearch supported hidden text >> in addition to overlays. >> >> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth >> > experimenting with `cursor-sensor-functions'. >> >> So far, I came up with the following partial solution searching and >> showing hidden text. >> >> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we >> ;; cannot even use cursor-sensor-functions as a workaround >> ;; I used a less ideas approach with advice to isearch-search-string as >> ;; a workaround >> >> (defun org-find-text-property-region (pos prop) >> "Find a region containing PROP text property around point POS." >> (require 'org-macs) ;; org-with-point-at >> (org-with-point-at pos >> (let* ((beg (and (get-text-property pos prop) pos)) >> (end beg)) >> (when beg >> (setq beg (or (previous-single-property-change pos prop) >> beg)) >> (setq end (or (next-single-property-change pos prop) >> end)) >> (unless (equal beg end) >> (cons beg end)))))) >> >> ;; :FIXME: re-hide properties when point moves away >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> "Reveal hidden text at point." >> (when-let ((region (org-find-text-property-region (point) 'invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) >> 'invisible))) >> (remove-text-properties (car region) (cdr region) '(invisible nil)))) >> >> ;; this seems to be unstable, but I cannot figure out why >> (defun org-restore-invisibility-specs (&rest _) >> "" >> (let ((pos (point-min))) >> (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point- >> max)) >> (when-let ((region (org-find-text-property-region pos 'org-invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org- >> invisible)) >> (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) >> >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> >> (defun org-flag-region (from to flag spec) >> "Hide or show lines from FROM to TO, according to FLAG. >> SPEC is the invisibility spec, as a symbol." >> (pcase spec >> ('outline >> (remove-overlays from to 'invisible spec) >> ;; Use `front-advance' since text right before to the beginning of >> ;; the overlay belongs to the visible line than to the contents. >> (when flag >> (let ((o (make-overlay from to nil 'front-advance))) >> (overlay-put o 'evaporate t) >> (overlay-put o 'invisible spec) >> (overlay-put o 'isearch-open-invisible #'delete-overlay)))) >> (_ >> (with-silent-modifications >> (remove-text-properties from to '(invisible nil)) >> (when flag >> (put-text-property from to 'invisible spec) >> ))))) >> >> ;; This normally deletes invisible text property. We do not want this now. >> (defun org-unfontify-region (beg end &optional _maybe_loudly) >> "Remove fontification and activation overlays from links." >> (font-lock-default-unfontify-region beg end) >> (let* ((buffer-undo-list t) >> (inhibit-read-only t) (inhibit-point-motion-hooks t) >> (inhibit-modification-hooks t) >> deactivate-mark buffer-file-name buffer-file-truename) >> (decompose-region beg end) >> (remove-text-properties beg end >> '(mouse-face t keymap t org-linked-text t >> ;; Do not remove invisible during fontification >> >> ;; invisible t >> intangible t >> org-emphasis t)) >> (org-remove-font-lock-display-properties beg end))) >> >> > Anyway, the real fix should come from Emacs itself. There are ways to >> > make overlays faster. These ways have already been discussed on the >> > Emacs devel mailing list, but no one implemented them. It is a bit sad >> > that we have to find workarounds for that. >> >> I guess that it is a very old story starting from the times when XEmacs >> was a thing [1]. I recently heard about binary tree implementation of >> overlays (there should be a branch in emacs git repo) [2], but there was >> no update on that branch for a while. So, I do not have much hope on >> Emacs implementing efficient overlay access in the near future. (And I >> have problems with huge org files already). >> >> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html >> >> >> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: >> >> > Hello, >> > >> > Ihor Radchenko <yantar92@gmail.com> writes: >> > >> > > To my surprise, the patch did not break org to unusable state and >> > > the performance on the sample org file [3] improved drastically. You can >> > > try by yourself! >> > >> > It is not a surprise, really. Text properties are much faster than >> > overlays, and very close to them features-wise. They are a bit more >> > complex to handle, however. >> > >> > > However, this did introduce some visual glitches with drawer display. >> > > Though drawers can still be folded/unfolded with <tab>, they are not >> > > folded on org-mode startup for some reason (can be fixed by running >> > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) >> > > are unfolded for no apparent reason sometimes. A blind guess is that it >> > > is something to do with lack of 'isearch-open-invisible, which I am not >> > > sure how to set via text properties. >> > >> > You cannot. You may however mimic it with `cursor-sensor-functions' text >> > property. These assume Cursor Sensor minor mode is active, tho. >> > I haven't tested it, but I assume it would slow down text properties >> > a bit, too, but hopefully not as much as overlays. >> > >> > Note there are clear advantages using text properties. For example, when >> > you move contents around, text properties are preserved. So there's no >> > more need for the `org-cycle-hide-drawer' dance, i.e., it is not >> > necessary anymore to re-hide drawers. >> > >> > > Any thoughts about the use of text properties or about the patch >> > > suggestion are welcome. >> > >> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth >> > experimenting with `cursor-sensor-functions'. >> > >> > We could also use text properties for property drawers, and overlays for >> > regular ones. This might give us a reasonable speed-up with an >> > acceptable feature trade-off. >> > >> > Anyway, the real fix should come from Emacs itself. There are ways to >> > make overlays faster. These ways have already been discussed on the >> > Emacs devel mailing list, but no one implemented them. It is a bit sad >> > that we have to find workarounds for that. >> > >> > Regards, >> > >> > -- >> > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-04-26 16:04 ` Ihor Radchenko 2020-05-04 16:56 ` Karl Voit 2020-05-07 11:04 ` Christian Heinrich @ 2020-05-08 16:38 ` Nicolas Goaziou 2020-05-09 13:58 ` Nicolas Goaziou 2020-05-09 15:40 ` Ihor Radchenko 2 siblings, 2 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-08 16:38 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we > ;; cannot even use cursor-sensor-functions as a workaround > ;; I used a less ideas approach with advice to isearch-search-string as > ;; a workaround OK. > (defun org-find-text-property-region (pos prop) > "Find a region containing PROP text property around point POS." > (require 'org-macs) ;; org-with-point-at > (org-with-point-at pos Do we really need that since every function has a POS argument anyway? Is it for the `widen' part? > (let* ((beg (and (get-text-property pos prop) pos)) > (end beg)) > (when beg > (setq beg (or (previous-single-property-change pos prop) > beg)) Shouldn't fall-back be (point-min)? > (setq end (or (next-single-property-change pos prop) > end)) And (point-max) here? > (unless (equal beg end) Nitpick: `equal' -> = > (cons beg end)))))) > ;; :FIXME: re-hide properties when point moves away > (define-advice isearch-search-string (:after (&rest _) put-overlay) > "Reveal hidden text at point." > (when-let ((region (org-find-text-property-region (point) 'invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible))) > (remove-text-properties (car region) (cdr region) '(invisible nil)))) Could we use `isearch-update-post-hook' here? Or, it seems nicer to `add-function' around `isearch-filter-predicate' and extend isearch-filter-visible to support (i.e., stop at, and display) invisible text through text properties. > ;; this seems to be unstable, but I cannot figure out why > (defun org-restore-invisibility-specs (&rest _) > "" > (let ((pos (point-min))) > (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max)) > (when-let ((region (org-find-text-property-region pos 'org-invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible)) > (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) Could you use the hook above to store all visited invisible texts, and re-hide them at the end of the search, e.g., using `isearch-mode-end-hook'? > (add-hook 'post-command-hook #'org-restore-invisibility-specs) Ouch. I hope we can avoid that. I wonder how it compares to drawers using the same invisible spec as headlines, as it was the case before. Could you give it a try? I think hiding all property drawers right after opening a subtree is fast enough. Another option, as I already suggested, would be to use text-properties on property drawers only. Ignoring isearch inside those sounds tolerable, at least. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-08 16:38 ` Nicolas Goaziou @ 2020-05-09 13:58 ` Nicolas Goaziou 2020-05-09 16:22 ` Ihor Radchenko 2020-05-09 15:40 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-09 13:58 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > I wonder how it compares to drawers using the same invisible spec as > headlines, as it was the case before. Could you give it a try? > > I think hiding all property drawers right after opening a subtree is > fast enough. As a follow-up, I switched property drawers (and only those) back to using `outline' invisible spec in master branch. Hopefully, navigating in large folded files should be faster. Of course, this doesn't prevent us to continue exploring text-properties. In particular, the problem is still open for regular drawers (e.g., LOGBOOK). ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 13:58 ` Nicolas Goaziou @ 2020-05-09 16:22 ` Ihor Radchenko 2020-05-09 17:21 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 16:22 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > As a follow-up, I switched property drawers (and only those) back to > using `outline' invisible spec in master branch. Hopefully, navigating > in large folded files should be faster. Just tested the master branch. Three observations on large org file: 1. Next/previous line on folder buffer is still terribly slow 2. Unfolding speed does not seem to be affected by the last commits - it is still much slower than text property version. There might be some improvement if I run Emacs for longer time though (Emacs generally becomes slower over time). 3. <TAB> <TAB> on a headline with several levels of subheadings moves the cursor to the end of subtree, which did not happen in the past. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> I wonder how it compares to drawers using the same invisible spec as >> headlines, as it was the case before. Could you give it a try? >> >> I think hiding all property drawers right after opening a subtree is >> fast enough. > > As a follow-up, I switched property drawers (and only those) back to > using `outline' invisible spec in master branch. Hopefully, navigating > in large folded files should be faster. > > Of course, this doesn't prevent us to continue exploring > text-properties. In particular, the problem is still open for regular > drawers (e.g., LOGBOOK). -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 16:22 ` Ihor Radchenko @ 2020-05-09 17:21 ` Nicolas Goaziou 2020-05-10 5:25 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-09 17:21 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > Just tested the master branch. > Three observations on large org file: > > 1. Next/previous line on folder buffer is still terribly slow Oops, you are right. I fixed this. It should be way faster. I can navigate in your example file without much trouble. Please let me know how it goes. > 2. Unfolding speed does not seem to be affected by the last commits - it > is still much slower than text property version. There might be some > improvement if I run Emacs for longer time though (Emacs generally > becomes slower over time). The last commits have nothing to do with unfolding. I'm not pretending that overlays are faster than text properties, either. With the current implementation property drawers add no overhead : last commits reduce drastically the number of overlays active in a buffer at a given time. > 3. <TAB> <TAB> on a headline with several levels of subheadings moves > the cursor to the end of subtree, which did not happen in the past. Indeed. I fixed that, too. Thank you! Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 17:21 ` Nicolas Goaziou @ 2020-05-10 5:25 ` Ihor Radchenko 2020-05-10 9:47 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-10 5:25 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > Oops, you are right. I fixed this. It should be way faster. I can > navigate in your example file without much trouble. > > Please let me know how it goes. I tested with master + my personal config + native compilation of org, Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae. Did not see much improvement. Vertical motion in the folded buffer is still quite slow. > The last commits have nothing to do with unfolding. Apparently I misunderstood the purpose of: 1027e0256 "Implement `org-cycle-hide-property-drawers'" Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> Just tested the master branch. >> Three observations on large org file: >> >> 1. Next/previous line on folder buffer is still terribly slow > > Oops, you are right. I fixed this. It should be way faster. I can > navigate in your example file without much trouble. > > Please let me know how it goes. > >> 2. Unfolding speed does not seem to be affected by the last commits - it >> is still much slower than text property version. There might be some >> improvement if I run Emacs for longer time though (Emacs generally >> becomes slower over time). > > The last commits have nothing to do with unfolding. > > I'm not pretending that overlays are faster than text properties, > either. > > With the current implementation property drawers add no overhead : last > commits reduce drastically the number of overlays active in a buffer at > a given time. > >> 3. <TAB> <TAB> on a headline with several levels of subheadings moves >> the cursor to the end of subtree, which did not happen in the past. > > Indeed. I fixed that, too. Thank you! > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 5:25 ` Ihor Radchenko @ 2020-05-10 9:47 ` Nicolas Goaziou 2020-05-10 13:29 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 9:47 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: >> Oops, you are right. I fixed this. It should be way faster. I can >> navigate in your example file without much trouble. >> >> Please let me know how it goes. > > I tested with master + my personal config + native compilation of org, > Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae. > Did not see much improvement. > Vertical motion in the folded buffer is still quite slow. Oh! This is embarrassing. I improved speed, then broke it again in a later commit. Sorry for wasting your time. I think I fixed it again. Thank you for the feedback. Could you have a look again? > Apparently I misunderstood the purpose of: 1027e0256 > "Implement `org-cycle-hide-property-drawers'" The function is meant to re-hide only property drawers after visibility cycling. Its purpose is not to improve /unfolding/ speed. Unfolding is very fast already, faster than using text properties. Folding has roughly the same speed in both cases: most time is spent looking for the next location to fold. However, folding with text properties is more resilient, so you fold less often. As a side note, your file contains 5217 headlines and 5215 property drawers. I'll ignore the 3989 regular drawers for the time being (although they do contribute to the slow navigation). In current master, it means there is at most 5217 overlays in the buffer. With text properties, the worse situation in the same. Of course, that case happens less often with text properties. For example, it happens in "contents" view in both cases. However, in "show all" view, it is only a problem with overlays. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 9:47 ` Nicolas Goaziou @ 2020-05-10 13:29 ` Ihor Radchenko 2020-05-10 14:46 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-10 13:29 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode >> I tested with master + my personal config + native compilation of org, >> Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae. >> Did not see much improvement. >> Vertical motion in the folded buffer is still quite slow. > > Oh! This is embarrassing. I improved speed, then broke it again in > a later commit. Sorry for wasting your time. I think I fixed it again. > Thank you for the feedback. > > Could you have a look again? I still do not feel much difference, so I used elp to quantify if there is any difference I cannot notice by myself. I tested the time to move from to bottom of the example file with next-logical-line. org master (7801e9236): 6(#calls) 2.852953989(total time, sec) 0.4754923315(average) org e39365e32: 6 2.991771891 0.4986286485 org feature/drawertextprop: 6 0.149731379 0.0249552298 There is small improvement in speed, but it is not obvious. > ... In current master, > it means there is at most 5217 overlays in the buffer. With text > properties, the worse situation in the same. Do you mean that number of overlays is same with text properties? I feel that I misunderstand what you want to say. > Of course, that case happens less often with text properties. For > example, it happens in "contents" view in both cases. However, in "show > all" view, it is only a problem with overlays. I am completely lost. What do you mean by "that case"? Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >>> Oops, you are right. I fixed this. It should be way faster. I can >>> navigate in your example file without much trouble. >>> >>> Please let me know how it goes. >> >> I tested with master + my personal config + native compilation of org, >> Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae. >> Did not see much improvement. >> Vertical motion in the folded buffer is still quite slow. > > Oh! This is embarrassing. I improved speed, then broke it again in > a later commit. Sorry for wasting your time. I think I fixed it again. > Thank you for the feedback. > > Could you have a look again? > >> Apparently I misunderstood the purpose of: 1027e0256 >> "Implement `org-cycle-hide-property-drawers'" > > The function is meant to re-hide only property drawers after visibility > cycling. Its purpose is not to improve /unfolding/ speed. Unfolding is > very fast already, faster than using text properties. > > Folding has roughly the same speed in both cases: most time is spent > looking for the next location to fold. However, folding with text > properties is more resilient, so you fold less often. > > As a side note, your file contains 5217 headlines and 5215 property > drawers. I'll ignore the 3989 regular drawers for the time being > (although they do contribute to the slow navigation). In current master, > it means there is at most 5217 overlays in the buffer. With text > properties, the worse situation in the same. > > Of course, that case happens less often with text properties. For > example, it happens in "contents" view in both cases. However, in "show > all" view, it is only a problem with overlays. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 13:29 ` Ihor Radchenko @ 2020-05-10 14:46 ` Nicolas Goaziou 2020-05-10 16:21 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 14:46 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > I still do not feel much difference, so I used elp to quantify if there > is any difference I cannot notice by myself. I tested the time to move > from to bottom of the example file with next-logical-line. > > org master (7801e9236): > 6(#calls) 2.852953989(total time, sec) 0.4754923315(average) > > org e39365e32: > 6 2.991771891 0.4986286485 > > org feature/drawertextprop: > 6 0.149731379 0.0249552298 > > There is small improvement in speed, but it is not obvious. I don't know how you made your test. You probably didn't remove :LOGBOOK: lines. When headlines are fully folded, there are 8 overlays in the buffer, where there used to be 10k. It cannot be a "small improvement". Ah, well. It doesn't matter. At least the situation improved in some cases, and the code is better. >> ... In current master, >> it means there is at most 5217 overlays in the buffer. With text >> properties, the worse situation in the same. > > Do you mean that number of overlays is same with text properties? I feel > that I misunderstand what you want to say. AFAIU, you still use overlays for headlines. If you activate so-called "contents view", all headlines are visible, and are all folded. You get 5217 overlays in the buffer. >> Of course, that case happens less often with text properties. For >> example, it happens in "contents" view in both cases. However, in "show >> all" view, it is only a problem with overlays. > > I am completely lost. What do you mean by "that case"? I am talking about the "worse case" situation just above. I'll comment your patch in another message. Regards, ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 14:46 ` Nicolas Goaziou @ 2020-05-10 16:21 ` Ihor Radchenko 2020-05-10 16:38 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-10 16:21 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > I don't know how you made your test. You probably didn't > remove :LOGBOOK: lines. When headlines are fully folded, there are > 8 overlays in the buffer, where there used to be 10k. It cannot be > a "small improvement". Ouch. I did not remove :LOGBOOK: lines. I thought you referred to the original file in "I can navigate in your example file without much trouble." If you want, I can test the file without :LOGBOOK: lines tomorrow. >>> ... In current master, >>> it means there is at most 5217 overlays in the buffer. With text >>> properties, the worse situation in the same. >> >> Do you mean that number of overlays is same with text properties? I feel >> that I misunderstand what you want to say. > > AFAIU, you still use overlays for headlines. If you activate so-called > "contents view", all headlines are visible, and are all folded. You get > 5217 overlays in the buffer. No, there are only 9 'outline overlays in the folded buffer if we do not create overlays for drawers. This is because outline-hide-sublevels called by org-overview is calling outline-flag-region on the whole buffer thus removing all the 'outline overlays in buffer (remove-overlays from to 'invisible 'outline) and re-creating a single overlay for each top-level heading. Now, thinking second time about this, using the following for org-flag-region would achieve similar effect: (remove-overlays from to 'invisible 'outline) (remove-overlays from to 'invisible 'org-hide-drawer) Now sure if it is going to break org-cycle though. What do you think? Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> I still do not feel much difference, so I used elp to quantify if there >> is any difference I cannot notice by myself. I tested the time to move >> from to bottom of the example file with next-logical-line. >> >> org master (7801e9236): >> 6(#calls) 2.852953989(total time, sec) 0.4754923315(average) >> >> org e39365e32: >> 6 2.991771891 0.4986286485 >> >> org feature/drawertextprop: >> 6 0.149731379 0.0249552298 >> >> There is small improvement in speed, but it is not obvious. > > I don't know how you made your test. You probably didn't > remove :LOGBOOK: lines. When headlines are fully folded, there are > 8 overlays in the buffer, where there used to be 10k. It cannot be > a "small improvement". > > Ah, well. It doesn't matter. At least the situation improved in some > cases, and the code is better. > >>> ... In current master, >>> it means there is at most 5217 overlays in the buffer. With text >>> properties, the worse situation in the same. >> >> Do you mean that number of overlays is same with text properties? I feel >> that I misunderstand what you want to say. > > AFAIU, you still use overlays for headlines. If you activate so-called > "contents view", all headlines are visible, and are all folded. You get > 5217 overlays in the buffer. > >>> Of course, that case happens less often with text properties. For >>> example, it happens in "contents" view in both cases. However, in "show >>> all" view, it is only a problem with overlays. >> >> I am completely lost. What do you mean by "that case"? > > I am talking about the "worse case" situation just above. > > I'll comment your patch in another message. > > Regards, -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 16:21 ` Ihor Radchenko @ 2020-05-10 16:38 ` Nicolas Goaziou 2020-05-10 17:08 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 16:38 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > If you want, I can test the file without :LOGBOOK: lines tomorrow. Don't worry, it doesn't matter now. > No, there are only 9 'outline overlays in the folded buffer if we do not > create overlays for drawers. This is because outline-hide-sublevels > called by org-overview is calling outline-flag-region on the whole > buffer thus removing all the 'outline overlays in buffer > (remove-overlays from to 'invisible 'outline) and re-creating a single > overlay for each top-level heading. You're talking about "overview" (org-overview), whereas I'm talking about "contents view" (org-content). They are not the same. In the latter, you show every headline in the buffer, so you have one overlay per headline. > Now, thinking second time about this, using the following for > org-flag-region would achieve similar effect: > > (remove-overlays from to 'invisible 'outline) > (remove-overlays from to 'invisible 'org-hide-drawer) > > Now sure if it is going to break org-cycle though. > What do you think? This is already the case. See first line of `org-flag-region'. Regards, ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 16:38 ` Nicolas Goaziou @ 2020-05-10 17:08 ` Ihor Radchenko 2020-05-10 19:38 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-10 17:08 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > You're talking about "overview" (org-overview), whereas I'm talking > about "contents view" (org-content). They are not the same. In the > latter, you show every headline in the buffer, so you have one overlay > per headline. Thanks for the explanation. I finally understand you initial note. I was thinking about org-overview mostly because it is the case when next/previous-line was extremely slow with many overlays jammed between two subsequent lines. >> Now, thinking second time about this, using the following for >> org-flag-region would achieve similar effect: >> >> (remove-overlays from to 'invisible 'outline) >> (remove-overlays from to 'invisible 'org-hide-drawer) >> >> Now sure if it is going to break org-cycle though. >> What do you think? > > This is already the case. See first line of `org-flag-region'. Currently, `org-flag-region' only removes one SPEC type of overlays: (remove-overlays from to 'invisible spec) If we change it to (remove-overlays from to 'invisible spec) (when flag (remove-overlays from to 'invisible 'org-hide-drawer) ... ) then all the extra drawer overlays in the flagged region will be removed. It will require re-creating those extra overlays later if the region is revealed again though. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> If you want, I can test the file without :LOGBOOK: lines tomorrow. > > Don't worry, it doesn't matter now. > >> No, there are only 9 'outline overlays in the folded buffer if we do not >> create overlays for drawers. This is because outline-hide-sublevels >> called by org-overview is calling outline-flag-region on the whole >> buffer thus removing all the 'outline overlays in buffer >> (remove-overlays from to 'invisible 'outline) and re-creating a single >> overlay for each top-level heading. > > You're talking about "overview" (org-overview), whereas I'm talking > about "contents view" (org-content). They are not the same. In the > latter, you show every headline in the buffer, so you have one overlay > per headline. > >> Now, thinking second time about this, using the following for >> org-flag-region would achieve similar effect: >> >> (remove-overlays from to 'invisible 'outline) >> (remove-overlays from to 'invisible 'org-hide-drawer) >> >> Now sure if it is going to break org-cycle though. >> What do you think? > > This is already the case. See first line of `org-flag-region'. > > Regards, -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 17:08 ` Ihor Radchenko @ 2020-05-10 19:38 ` Nicolas Goaziou 0 siblings, 0 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 19:38 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Currently, `org-flag-region' only removes one SPEC type of overlays: > > (remove-overlays from to 'invisible spec) > > If we change it to > > (remove-overlays from to 'invisible spec) > (when flag > (remove-overlays from to 'invisible 'org-hide-drawer) > ... > ) > > then all the extra drawer overlays in the flagged region will be > removed. It will require re-creating those extra overlays later if the > region is revealed again though. Exactly. This would be equivalent to drop `org-hide-drawer' altogether, which we did for property drawers. You have to fold again every drawer after each visibility change. For the record, this is the initial bug `org-hide-drawer' was trying to solve. Back to square one. Also, we would have the same problem with blocks. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-08 16:38 ` Nicolas Goaziou 2020-05-09 13:58 ` Nicolas Goaziou @ 2020-05-09 15:40 ` Ihor Radchenko 2020-05-09 16:30 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 15:40 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 4813 bytes --] I have prepared a patch taking into account your comments and fixing other issues, reported by Karl Voit and found by myself. Summary of what is done in the patch: 1. iSearching in drawers is rewritten using using isearch-filter-predicate and isearch-mode-end-hook. The idea is to create temporary overlays in place of drawers to make isearch work as usual. 2. Change org-show-set-visibility to consider text properties. This makes helm-occur open drawers. 3. Make sure (partially) that text inserted into hidden drawers is also hidden (to avoid glitches reported by Karl Voit). The reason why it was happening was because `insert' does not inherit text properties by default, which means that all the inserted text is visible by default. I have changes some instances of insert and insert-before-markers to thair *-and-inherit versions. Still looking into where else I need to do the replacement. Note that "glitch" might appear in many external packages writing into org drawers. I do not think that insert-and-inherit is often used or even known. Remaining problems: 1. insert-* -> insert-*-and-inherit replacement will at least need to be done in org-table.el and probably other places 2. I found hi-lock re-opening drawers after exiting isearch for some reason. This happens when hi-lock tries to highlight isearch matches. Not sure about the cause. 3. There is still some visual glitch when unnecessary org-ellipsis is shown when text was inserted into hidden property drawer, though the inserted text itself is hidden. >> (defun org-find-text-property-region (pos prop) >> "Find a region containing PROP text property around point POS." >> (require 'org-macs) ;; org-with-point-at >> (org-with-point-at pos > > Do we really need that since every function has a POS argument anyway? > Is it for the `widen' part? Yes, it is not needed. Fixed. >> (let* ((beg (and (get-text-property pos prop) pos)) >> (end beg)) >> (when beg >> (setq beg (or (previous-single-property-change pos prop) >> beg)) > > Shouldn't fall-back be (point-min)? > >> (setq end (or (next-single-property-change pos prop) >> end)) > > And (point-max) here? No, (point-min) and (point-max) may cause problems there. previous/next-single-property-change returns nil when called at the beginning/end of the region with given text property. Falling back to (point-min/max) may wrongly return too large region. > Nitpick: `equal' -> = Fixed. > Or, it seems nicer to `add-function' around `isearch-filter-predicate' > and extend isearch-filter-visible to support (i.e., stop at, and > display) invisible text through text properties. Done. I used (setq-local isearch-filter-predicate #'org--isearch-filter-predicate), which should be even cleaner. > I wonder how it compares to drawers using the same invisible spec as > headlines, as it was the case before. Could you give it a try? > I think hiding all property drawers right after opening a subtree is > fast enough. I am not sure what you refer to. Just saw your relevant commit. Will test ASAP. Without testing, the code does not seem to change the number of overlays. A new overlay is still created for each property drawer. As I mentioned in the first email, the large number of overlays is what makes Emacs slow. Citing Eli Zaretskii's reply to my Bug#354553, explaining why Emacs becomes slow on large org file: "... When C-n calls vertical-motion, the latter needs to find the buffer position displayed directly below the place where you typed C-n. Since much of the text between these places, vertical-motion needs to skip the invisible text as quickly as possible, because from the user's POV that text "doesn't exist": it isn't on the screen. However, Org makes this skipping exceedingly hard, because (1) it uses overlays (as opposed to text properties) to hide text, and (2) it puts an awful lot of overlays on the hidden text: there are 18400 overlays in this file's buffer, 17500 of them between the 3rd and the 4th heading. Because of this, vertical-motion must examine each and every overlay as it moves through the text, because each overlay can potentially change the 'invisible' property of text, or it might have a display string that needs to be displayed. So instead of skipping all that hidden text in one go, vertical-motion loops over those 17.5K overlays examining the properties of each one of them. And that takes time." I imagine that opening subtree will also require cycling over the [many] overlays in the subtree. > Another option, as I already suggested, would be to use text-properties > on property drawers only. Ignoring isearch inside those sounds > tolerable, at least. Hope the patch below is a reasonable solution to isearch problem with 'invisible text properties. Best, Ihor [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: org-mode-drawertextprop.patch --] [-- Type: text/x-diff, Size: 13892 bytes --] diff --git a/lisp/org-clock.el b/lisp/org-clock.el index 34179096d..463b28f47 100644 --- a/lisp/org-clock.el +++ b/lisp/org-clock.el @@ -1359,14 +1359,14 @@ the default behavior." (sit-for 2) (throw 'abort nil)) (t - (insert-before-markers "\n") + (insert-before-markers-and-inherit "\n") (backward-char 1) (when (and (save-excursion (end-of-line 0) (org-in-item-p))) (beginning-of-line 1) (indent-line-to (- (current-indentation) 2))) - (insert org-clock-string " ") + (insert-and-inherit org-clock-string " ") (setq org-clock-effort (org-entry-get (point) org-effort-property)) (setq org-clock-total-time (org-clock-sum-current-item (org-clock-get-sum-start))) @@ -1658,7 +1658,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'." (if fail-quietly (throw 'exit nil) (error "Clock start time is gone"))) (goto-char (match-end 0)) (delete-region (point) (point-at-eol)) - (insert "--") + (insert-and-inherit "--") (setq te (org-insert-time-stamp (or at-time now) 'with-hm 'inactive)) (setq s (org-time-convert-to-integer (time-subtract @@ -1666,7 +1666,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'." (org-time-string-to-time ts))) h (floor s 3600) m (floor (mod s 3600) 60)) - (insert " => " (format "%2d:%02d" h m)) + (insert-and-inherit " => " (format "%2d:%02d" h m)) (move-marker org-clock-marker nil) (move-marker org-clock-hd-marker nil) ;; Possibly remove zero time clocks. However, do not add diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..4b0e23f6a 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." \f -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + (pcase spec + ('outline + (remove-overlays from to 'invisible spec) + ;; Use `front-advance' since text right before to the beginning of + ;; the overlay belongs to the visible line than to the contents. + (when flag + (let ((o (make-overlay from to nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) + (_ + ;; Use text properties instead of overlays for speed. + ;; Overlays are too slow (Emacs Bug#35453). + (with-silent-modifications + (remove-text-properties from to '(invisible nil)) + (when flag + (put-text-property from to 'rear-non-sticky nil) + (put-text-property from to 'front-sticky t) + (put-text-property from to 'invisible spec)))))) \f ;;; Regexp matching diff --git a/lisp/org.el b/lisp/org.el index 287fe30e8..335f68a85 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -4869,6 +4870,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5859,9 +5864,26 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) + ;; do not remove invisible text properties specified by + ;; 'org-hide-block and 'org-hide-drawer (but remove 'org-link) + ;; this is needed to keep the drawers and blocks hidden unless + ;; they are toggled by user + ;; Note: The below may be too specific and create troubles + ;; if more invisibility specs are added to org in future + (let ((pos beg) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer)) + (remove-text-properties pos next '(invisible t))) + (setq pos next))) (remove-text-properties beg end '(mouse-face t keymap t org-linked-text t - invisible t intangible t + ;; Do not remove all invisible during fontification + ;; invisible t + intangible t org-emphasis t)) (org-remove-font-lock-display-properties beg end))) @@ -6677,8 +6699,13 @@ information." ;; expose it. (dolist (o (overlays-at (point))) (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) + '(outline)) (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -10849,8 +10876,8 @@ EXTRA is additional text that will be inserted into the notes buffer." (unless (eq org-log-note-purpose 'clock-out) (goto-char (org-log-beginning t))) ;; Make sure point is at the beginning of an empty line. - (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert "\n"))) - ((looking-at "[ \t]*\\S-") (save-excursion (insert "\n")))) + (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert-and-inherit "\n"))) + ((looking-at "[ \t]*\\S-") (save-excursion (insert-and-inherit "\n")))) ;; In an existing list, add a new item at the top level. ;; Otherwise, indent line like a regular one. (let ((itemp (org-in-item-p))) @@ -10860,12 +10887,12 @@ EXTRA is additional text that will be inserted into the notes buffer." (goto-char itemp) (org-list-struct)))) (org-list-get-ind (org-list-get-top-point struct) struct))) (org-indent-line))) - (insert (org-list-bullet-string "-") (pop lines)) + (insert-and-inherit (org-list-bullet-string "-") (pop lines)) (let ((ind (org-list-item-body-column (line-beginning-position)))) (dolist (line lines) - (insert "\n") + (insert-and-inherit "\n") (indent-line-to ind) - (insert line))) + (insert-and-inherit line))) (message "Note stored") (org-back-to-heading t)) ;; Fix `buffer-undo-list' when `org-store-log-note' is called @@ -13036,10 +13063,10 @@ decreases scheduled or deadline date by one day." (progn (delete-region (match-beginning 0) (match-end 0)) (goto-char (match-beginning 0))) (goto-char end) - (insert "\n") + (insert-and-inherit "\n") (backward-char)) - (insert ":" property ":") - (when value (insert " " value)) + (insert-and-inherit ":" property ":") + (when value (insert-and-inherit " " value)) (org-indent-line))))) (run-hook-with-args 'org-property-changed-functions property value))) @@ -14177,7 +14204,7 @@ The command returns the inserted time stamp." (let ((fmt (funcall (if with-hm 'cdr 'car) org-time-stamp-formats)) stamp) (when inactive (setq fmt (concat "[" (substring fmt 1 -1) "]"))) - (insert-before-markers (or pre "")) + (insert-before-markers-and-inherit (or pre "")) (when (listp extra) (setq extra (car extra)) (if (and (stringp extra) @@ -14188,8 +14215,8 @@ The command returns the inserted time stamp." (setq extra nil))) (when extra (setq fmt (concat (substring fmt 0 -1) extra (substring fmt -1)))) - (insert-before-markers (setq stamp (format-time-string fmt time))) - (insert-before-markers (or post "")) + (insert-before-markers-and-inherit (setq stamp (format-time-string fmt time))) + (insert-before-markers-and-inherit (or post "")) (setq org-last-inserted-timestamp stamp))) (defun org-toggle-time-stamp-overlays () @@ -20913,6 +20940,79 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) \f + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (remove-text-properties (car region) (cdr region) '(invisible nil)))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + +\f + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode [-- Attachment #3: Type: text/plain, Size: 3286 bytes --] Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we >> ;; cannot even use cursor-sensor-functions as a workaround >> ;; I used a less ideas approach with advice to isearch-search-string as >> ;; a workaround > > OK. > >> (defun org-find-text-property-region (pos prop) >> "Find a region containing PROP text property around point POS." >> (require 'org-macs) ;; org-with-point-at >> (org-with-point-at pos > > Do we really need that since every function has a POS argument anyway? > Is it for the `widen' part? > >> (let* ((beg (and (get-text-property pos prop) pos)) >> (end beg)) >> (when beg >> (setq beg (or (previous-single-property-change pos prop) >> beg)) > > Shouldn't fall-back be (point-min)? > >> (setq end (or (next-single-property-change pos prop) >> end)) > > And (point-max) here? > >> (unless (equal beg end) > > Nitpick: `equal' -> = > >> (cons beg end)))))) > >> ;; :FIXME: re-hide properties when point moves away >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> "Reveal hidden text at point." >> (when-let ((region (org-find-text-property-region (point) 'invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible))) >> (remove-text-properties (car region) (cdr region) '(invisible nil)))) > > Could we use `isearch-update-post-hook' here? > > Or, it seems nicer to `add-function' around `isearch-filter-predicate' > and extend isearch-filter-visible to support (i.e., stop at, and > display) invisible text through text properties. > >> ;; this seems to be unstable, but I cannot figure out why >> (defun org-restore-invisibility-specs (&rest _) >> "" >> (let ((pos (point-min))) >> (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max)) >> (when-let ((region (org-find-text-property-region pos 'org-invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible)) >> (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) > > Could you use the hook above to store all visited invisible texts, and > re-hide them at the end of the search, e.g., using > `isearch-mode-end-hook'? > >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) > > Ouch. I hope we can avoid that. > > I wonder how it compares to drawers using the same invisible spec as > headlines, as it was the case before. Could you give it a try? > > I think hiding all property drawers right after opening a subtree is > fast enough. > > Another option, as I already suggested, would be to use text-properties > on property drawers only. Ignoring isearch inside those sounds > tolerable, at least. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply related [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 15:40 ` Ihor Radchenko @ 2020-05-09 16:30 ` Ihor Radchenko 2020-05-09 17:32 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 16:30 UTC (permalink / raw) To: emacs-orgmode, Nicolas Goaziou Note that the following commits seems to break my patch: 074ea1629 origin/master master Deprecate `org-cycle-hide-drawers' 1027e0256 Implement `org-cycle-hide-property-drawers' 8b05c06d4 Use `outline' invisibility spec for property drawers The patch should work for commit ed0e75d24 in master. Best, Ihor Ihor Radchenko <yantar92@gmail.com> writes: > I have prepared a patch taking into account your comments and fixing > other issues, reported by Karl Voit and found by myself. > > Summary of what is done in the patch: > > 1. iSearching in drawers is rewritten using using > isearch-filter-predicate and isearch-mode-end-hook. > The idea is to create temporary overlays in place of drawers to make > isearch work as usual. > > 2. Change org-show-set-visibility to consider text properties. This > makes helm-occur open drawers. > > 3. Make sure (partially) that text inserted into hidden drawers is also > hidden (to avoid glitches reported by Karl Voit). > The reason why it was happening was because `insert' does not inherit > text properties by default, which means that all the inserted text is > visible by default. I have changes some instances of insert and > insert-before-markers to thair *-and-inherit versions. Still looking > into where else I need to do the replacement. > > Note that "glitch" might appear in many external packages writing into > org drawers. I do not think that insert-and-inherit is often used or > even known. > > Remaining problems: > > 1. insert-* -> insert-*-and-inherit replacement will at least need to be > done in org-table.el and probably other places > > 2. I found hi-lock re-opening drawers after exiting isearch for some > reason. This happens when hi-lock tries to highlight isearch matches. > Not sure about the cause. > > 3. There is still some visual glitch when unnecessary org-ellipsis is > shown when text was inserted into hidden property drawer, though the > inserted text itself is hidden. > >>> (defun org-find-text-property-region (pos prop) >>> "Find a region containing PROP text property around point POS." >>> (require 'org-macs) ;; org-with-point-at >>> (org-with-point-at pos >> >> Do we really need that since every function has a POS argument anyway? >> Is it for the `widen' part? > > Yes, it is not needed. Fixed. > >>> (let* ((beg (and (get-text-property pos prop) pos)) >>> (end beg)) >>> (when beg >>> (setq beg (or (previous-single-property-change pos prop) >>> beg)) >> >> Shouldn't fall-back be (point-min)? >> >>> (setq end (or (next-single-property-change pos prop) >>> end)) >> >> And (point-max) here? > > No, (point-min) and (point-max) may cause problems there. > previous/next-single-property-change returns nil when called at the > beginning/end of the region with given text property. Falling back to > (point-min/max) may wrongly return too large region. > >> Nitpick: `equal' -> = > > Fixed. > >> Or, it seems nicer to `add-function' around `isearch-filter-predicate' >> and extend isearch-filter-visible to support (i.e., stop at, and >> display) invisible text through text properties. > > Done. I used > (setq-local isearch-filter-predicate #'org--isearch-filter-predicate), > which should be even cleaner. > >> I wonder how it compares to drawers using the same invisible spec as >> headlines, as it was the case before. Could you give it a try? > >> I think hiding all property drawers right after opening a subtree is >> fast enough. > > I am not sure what you refer to. Just saw your relevant commit. Will > test ASAP. > > Without testing, the code does not seem to change the number of > overlays. A new overlay is still created for each property drawer. > As I mentioned in the first email, the large number of overlays is what > makes Emacs slow. Citing Eli Zaretskii's reply to my Bug#354553, > explaining why Emacs becomes slow on large org file: > > "... When C-n calls vertical-motion, the latter needs to find the > buffer position displayed directly below the place where you typed > C-n. Since much of the text between these places, vertical-motion > needs to skip the invisible text as quickly as possible, because from > the user's POV that text "doesn't exist": it isn't on the screen. > However, Org makes this skipping exceedingly hard, because (1) it uses > overlays (as opposed to text properties) to hide text, and (2) it puts > an awful lot of overlays on the hidden text: there are 18400 overlays > in this file's buffer, 17500 of them between the 3rd and the 4th > heading. Because of this, vertical-motion must examine each and every > overlay as it moves through the text, because each overlay can > potentially change the 'invisible' property of text, or it might have > a display string that needs to be displayed. So instead of skipping > all that hidden text in one go, vertical-motion loops over those 17.5K > overlays examining the properties of each one of them. And that takes > time." > > I imagine that opening subtree will also require cycling over the > [many] overlays in the subtree. > >> Another option, as I already suggested, would be to use text-properties >> on property drawers only. Ignoring isearch inside those sounds >> tolerable, at least. > > Hope the patch below is a reasonable solution to isearch problem with > 'invisible text properties. > > Best, > Ihor > > diff --git a/lisp/org-clock.el b/lisp/org-clock.el > index 34179096d..463b28f47 100644 > --- a/lisp/org-clock.el > +++ b/lisp/org-clock.el > @@ -1359,14 +1359,14 @@ the default behavior." > (sit-for 2) > (throw 'abort nil)) > (t > - (insert-before-markers "\n") > + (insert-before-markers-and-inherit "\n") > (backward-char 1) > (when (and (save-excursion > (end-of-line 0) > (org-in-item-p))) > (beginning-of-line 1) > (indent-line-to (- (current-indentation) 2))) > - (insert org-clock-string " ") > + (insert-and-inherit org-clock-string " ") > (setq org-clock-effort (org-entry-get (point) org-effort-property)) > (setq org-clock-total-time (org-clock-sum-current-item > (org-clock-get-sum-start))) > @@ -1658,7 +1658,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'." > (if fail-quietly (throw 'exit nil) (error "Clock start time is gone"))) > (goto-char (match-end 0)) > (delete-region (point) (point-at-eol)) > - (insert "--") > + (insert-and-inherit "--") > (setq te (org-insert-time-stamp (or at-time now) 'with-hm 'inactive)) > (setq s (org-time-convert-to-integer > (time-subtract > @@ -1666,7 +1666,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'." > (org-time-string-to-time ts))) > h (floor s 3600) > m (floor (mod s 3600) 60)) > - (insert " => " (format "%2d:%02d" h m)) > + (insert-and-inherit " => " (format "%2d:%02d" h m)) > (move-marker org-clock-marker nil) > (move-marker org-clock-hd-marker nil) > ;; Possibly remove zero time clocks. However, do not add > diff --git a/lisp/org-macs.el b/lisp/org-macs.el > index a02f713ca..4b0e23f6a 100644 > --- a/lisp/org-macs.el > +++ b/lisp/org-macs.el > @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." > > > \f > -;;; Overlays > +;;; Overlays and text properties > > (defun org-overlay-display (ovl text &optional face evap) > "Make overlay OVL display TEXT with face FACE." > @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays." > (delete (delete-overlay ov)) > (t (push ov found)))))) > > +(defun org--find-text-property-region (pos prop) > + "Find a region containing PROP text property around point POS." > + (let* ((beg (and (get-text-property pos prop) pos)) > + (end beg)) > + (when beg > + ;; when beg is the first point in the region, `previous-single-property-change' > + ;; will return nil. > + (setq beg (or (previous-single-property-change pos prop) > + beg)) > + ;; when end is the last point in the region, `next-single-property-change' > + ;; will return nil. > + (setq end (or (next-single-property-change pos prop) > + end)) > + (unless (= beg end) ; this should not happen > + (cons beg end))))) > + > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > - (remove-overlays from to 'invisible spec) > - ;; Use `front-advance' since text right before to the beginning of > - ;; the overlay belongs to the visible line than to the contents. > - (when flag > - (let ((o (make-overlay from to nil 'front-advance))) > - (overlay-put o 'evaporate t) > - (overlay-put o 'invisible spec) > - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > - > + (pcase spec > + ('outline > + (remove-overlays from to 'invisible spec) > + ;; Use `front-advance' since text right before to the beginning of > + ;; the overlay belongs to the visible line than to the contents. > + (when flag > + (let ((o (make-overlay from to nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > + (_ > + ;; Use text properties instead of overlays for speed. > + ;; Overlays are too slow (Emacs Bug#35453). > + (with-silent-modifications > + (remove-text-properties from to '(invisible nil)) > + (when flag > + (put-text-property from to 'rear-non-sticky nil) > + (put-text-property from to 'front-sticky t) > + (put-text-property from to 'invisible spec)))))) > > \f > ;;; Regexp matching > diff --git a/lisp/org.el b/lisp/org.el > index 287fe30e8..335f68a85 100644 > --- a/lisp/org.el > +++ b/lisp/org.el > @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") > (declare-function cdlatex-math-symbol "ext:cdlatex") > (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) > (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) > +(declare-function isearch-filter-visible "isearch" (beg end)) > (declare-function org-add-archive-files "org-archive" (files)) > (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) > (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) > @@ -4869,6 +4870,10 @@ The following commands are available: > (setq-local outline-isearch-open-invisible-function > (lambda (&rest _) (org-show-context 'isearch))) > > + ;; Make isearch search in blocks hidden via text properties > + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) > + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) > + > ;; Setup the pcomplete hooks > (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) > (setq-local pcomplete-command-name-function #'org-command-at-point) > @@ -5859,9 +5864,26 @@ If TAG is a number, get the corresponding match group." > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > + ;; do not remove invisible text properties specified by > + ;; 'org-hide-block and 'org-hide-drawer (but remove 'org-link) > + ;; this is needed to keep the drawers and blocks hidden unless > + ;; they are toggled by user > + ;; Note: The below may be too specific and create troubles > + ;; if more invisibility specs are added to org in future > + (let ((pos beg) > + next spec) > + (while (< pos end) > + (setq next (next-single-property-change pos 'invisible nil end) > + spec (get-text-property pos 'invisible)) > + (unless (memq spec (list 'org-hide-block > + 'org-hide-drawer)) > + (remove-text-properties pos next '(invisible t))) > + (setq pos next))) > (remove-text-properties beg end > '(mouse-face t keymap t org-linked-text t > - invisible t intangible t > + ;; Do not remove all invisible during fontification > + ;; invisible t > + intangible t > org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) > > @@ -6677,8 +6699,13 @@ information." > ;; expose it. > (dolist (o (overlays-at (point))) > (when (memq (overlay-get o 'invisible) > - '(org-hide-block org-hide-drawer outline)) > + '(outline)) > (delete-overlay o))) > + (when (memq (get-text-property (point) 'invisible) > + '(org-hide-block org-hide-drawer)) > + (let ((spec (get-text-property (point) 'invisible)) > + (region (org--find-text-property-region (point) 'invisible))) > + (org-flag-region (car region) (cdr region) nil spec))) > (unless (org-before-first-heading-p) > (org-with-limited-levels > (cl-case detail > @@ -10849,8 +10876,8 @@ EXTRA is additional text that will be inserted into the notes buffer." > (unless (eq org-log-note-purpose 'clock-out) > (goto-char (org-log-beginning t))) > ;; Make sure point is at the beginning of an empty line. > - (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert "\n"))) > - ((looking-at "[ \t]*\\S-") (save-excursion (insert "\n")))) > + (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert-and-inherit "\n"))) > + ((looking-at "[ \t]*\\S-") (save-excursion (insert-and-inherit "\n")))) > ;; In an existing list, add a new item at the top level. > ;; Otherwise, indent line like a regular one. > (let ((itemp (org-in-item-p))) > @@ -10860,12 +10887,12 @@ EXTRA is additional text that will be inserted into the notes buffer." > (goto-char itemp) (org-list-struct)))) > (org-list-get-ind (org-list-get-top-point struct) struct))) > (org-indent-line))) > - (insert (org-list-bullet-string "-") (pop lines)) > + (insert-and-inherit (org-list-bullet-string "-") (pop lines)) > (let ((ind (org-list-item-body-column (line-beginning-position)))) > (dolist (line lines) > - (insert "\n") > + (insert-and-inherit "\n") > (indent-line-to ind) > - (insert line))) > + (insert-and-inherit line))) > (message "Note stored") > (org-back-to-heading t)) > ;; Fix `buffer-undo-list' when `org-store-log-note' is called > @@ -13036,10 +13063,10 @@ decreases scheduled or deadline date by one day." > (progn (delete-region (match-beginning 0) (match-end 0)) > (goto-char (match-beginning 0))) > (goto-char end) > - (insert "\n") > + (insert-and-inherit "\n") > (backward-char)) > - (insert ":" property ":") > - (when value (insert " " value)) > + (insert-and-inherit ":" property ":") > + (when value (insert-and-inherit " " value)) > (org-indent-line))))) > (run-hook-with-args 'org-property-changed-functions property value))) > > @@ -14177,7 +14204,7 @@ The command returns the inserted time stamp." > (let ((fmt (funcall (if with-hm 'cdr 'car) org-time-stamp-formats)) > stamp) > (when inactive (setq fmt (concat "[" (substring fmt 1 -1) "]"))) > - (insert-before-markers (or pre "")) > + (insert-before-markers-and-inherit (or pre "")) > (when (listp extra) > (setq extra (car extra)) > (if (and (stringp extra) > @@ -14188,8 +14215,8 @@ The command returns the inserted time stamp." > (setq extra nil))) > (when extra > (setq fmt (concat (substring fmt 0 -1) extra (substring fmt -1)))) > - (insert-before-markers (setq stamp (format-time-string fmt time))) > - (insert-before-markers (or post "")) > + (insert-before-markers-and-inherit (setq stamp (format-time-string fmt time))) > + (insert-before-markers-and-inherit (or post "")) > (setq org-last-inserted-timestamp stamp))) > > (defun org-toggle-time-stamp-overlays () > @@ -20913,6 +20940,79 @@ Started from `gnus-info-find-node'." > (t default-org-info-node)))))) > > \f > + > +;;; Make isearch search in some text hidden via text propertoes > + > +(defvar org--isearch-overlays nil > + "List of overlays temporarily created during isearch. > +This is used to allow searching in regions hidden via text properties. > +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. > +Any text hidden via text properties is not revealed even if `search-invisible' > +is set to 't.") > + > +;; Not sure if it needs to be a user option > +;; One might want to reveal hidden text in, for example, hidden parts of the links. > +;; Currently, hidden text in links is never revealed by isearch. > +(defvar org-isearch-specs '(org-hide-block > + org-hide-drawer) > + "List of text invisibility specs to be searched by isearch. > +By default ([2020-05-09 Sat]), isearch does not search in hidden text, > +which was made invisible using text properties. Isearch will be forced > +to search in hidden text with any of the listed 'invisible property value.") > + > +(defun org--create-isearch-overlays (beg end) > + "Replace text property invisibility spec by overlays between BEG and END. > +All the regions with invisibility text property spec from > +`org-isearch-specs' will be changed to use overlays instead > +of text properties. The created overlays will be stored in > +`org--isearch-overlays'." > + (let ((pos beg)) > + (while (< pos end) > + (when-let* ((spec (get-text-property pos 'invisible)) > + (spec (memq spec org-isearch-specs)) > + (region (org--find-text-property-region pos 'invisible))) > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] > + ;; overlay for 'outline blocks. > + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + ;; `delete-overlay' here means that spec information will be lost > + ;; for the region. The region will remain visible. > + (overlay-put o 'isearch-open-invisible #'delete-overlay) > + (push o org--isearch-overlays)) > + (remove-text-properties (car region) (cdr region) '(invisible nil)))) > + (setq pos (next-single-property-change pos 'invisible nil end))))) > + > +(defun org--isearch-filter-predicate (beg end) > + "Return non-nil if text between BEG and END is deemed visible by Isearch. > +This function is intended to be used as `isearch-filter-predicate'. > +Unlike `isearch-filter-visible', make text with 'invisible text property > +value listed in `org-isearch-specs' visible to Isearch." > + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text > + (isearch-filter-visible beg end)) > + > +(defun org--clear-isearch-overlay (ov) > + "Convert OV region back into using text properties." > + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec))) > + (when (member ov isearch-opened-overlays) > + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) > + (delete-overlay ov)) > + > +(defun org--clear-isearch-overlays () > + "Convert overlays from `org--isearch-overlays' back into using text properties." > + (when org--isearch-overlays > + (mapc #'org--clear-isearch-overlay org--isearch-overlays) > + (setq org--isearch-overlays nil))) > + > +\f > + > ;;; Finish up > > (add-hook 'org-mode-hook ;remove overlays when changing major mode > > > > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Hello, >> >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we >>> ;; cannot even use cursor-sensor-functions as a workaround >>> ;; I used a less ideas approach with advice to isearch-search-string as >>> ;; a workaround >> >> OK. >> >>> (defun org-find-text-property-region (pos prop) >>> "Find a region containing PROP text property around point POS." >>> (require 'org-macs) ;; org-with-point-at >>> (org-with-point-at pos >> >> Do we really need that since every function has a POS argument anyway? >> Is it for the `widen' part? >> >>> (let* ((beg (and (get-text-property pos prop) pos)) >>> (end beg)) >>> (when beg >>> (setq beg (or (previous-single-property-change pos prop) >>> beg)) >> >> Shouldn't fall-back be (point-min)? >> >>> (setq end (or (next-single-property-change pos prop) >>> end)) >> >> And (point-max) here? >> >>> (unless (equal beg end) >> >> Nitpick: `equal' -> = >> >>> (cons beg end)))))) >> >>> ;; :FIXME: re-hide properties when point moves away >>> (define-advice isearch-search-string (:after (&rest _) put-overlay) >>> "Reveal hidden text at point." >>> (when-let ((region (org-find-text-property-region (point) 'invisible))) >>> (with-silent-modifications >>> (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible))) >>> (remove-text-properties (car region) (cdr region) '(invisible nil)))) >> >> Could we use `isearch-update-post-hook' here? >> >> Or, it seems nicer to `add-function' around `isearch-filter-predicate' >> and extend isearch-filter-visible to support (i.e., stop at, and >> display) invisible text through text properties. >> >>> ;; this seems to be unstable, but I cannot figure out why >>> (defun org-restore-invisibility-specs (&rest _) >>> "" >>> (let ((pos (point-min))) >>> (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max)) >>> (when-let ((region (org-find-text-property-region pos 'org-invisible))) >>> (with-silent-modifications >>> (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible)) >>> (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) >> >> Could you use the hook above to store all visited invisible texts, and >> re-hide them at the end of the search, e.g., using >> `isearch-mode-end-hook'? >> >>> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> >> Ouch. I hope we can avoid that. >> >> I wonder how it compares to drawers using the same invisible spec as >> headlines, as it was the case before. Could you give it a try? >> >> I think hiding all property drawers right after opening a subtree is >> fast enough. >> >> Another option, as I already suggested, would be to use text-properties >> on property drawers only. Ignoring isearch inside those sounds >> tolerable, at least. >> >> Regards, >> >> -- >> Nicolas Goaziou > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 16:30 ` Ihor Radchenko @ 2020-05-09 17:32 ` Nicolas Goaziou 2020-05-09 18:06 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-09 17:32 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Note that the following commits seems to break my patch: Unfortunately, I don't see your patch. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 17:32 ` Nicolas Goaziou @ 2020-05-09 18:06 ` Ihor Radchenko 2020-05-10 14:59 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-09 18:06 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > Unfortunately, I don't see your patch. My response to you was blocked by your mail server: > 550 5.7.1 Reject for policy reason RULE3_2. See > http://postmaster.gandi.net The message landed on the mail list though: https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> Note that the following commits seems to break my patch: > > Unfortunately, I don't see your patch. -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-09 18:06 ` Ihor Radchenko @ 2020-05-10 14:59 ` Nicolas Goaziou 2020-05-10 15:15 ` Kyle Meyer 2020-05-10 16:30 ` Ihor Radchenko 0 siblings, 2 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 14:59 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > My response to you was blocked by your mail server: > >> 550 5.7.1 Reject for policy reason RULE3_2. See >> http://postmaster.gandi.net Aka "spam detected". Bah. > The message landed on the mail list though: > https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html Unfortunately, reviewing this way is not nice. The `insert-and-inherit' issue sounds serious. We cannot reasonably expect any library outside Org to use it. We could automatically extend invisible area with `after-change-functions', i.e., if we're inserting something and both side have the same `invisible' property, extend it. Using `after-change-functions' sounds bad, but this kind of check shouldn't cost much. WDYT? Regards, ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 14:59 ` Nicolas Goaziou @ 2020-05-10 15:15 ` Kyle Meyer 2020-05-10 16:30 ` Ihor Radchenko 1 sibling, 0 replies; 192+ messages in thread From: Kyle Meyer @ 2020-05-10 15:15 UTC (permalink / raw) To: emacs-orgmode; +Cc: Ihor Radchenko Nicolas Goaziou writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> My response to you was blocked by your mail server: >> >>> 550 5.7.1 Reject for policy reason RULE3_2. See >>> http://postmaster.gandi.net > > Aka "spam detected". Bah. > >> The message landed on the mail list though: >> https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html > > Unfortunately, reviewing this way is not nice. It's probably not helpful at this point, but just in case: you can get that message's mbox with curl -fSs https://yhetil.org/orgmode/87imh5w1zt.fsf@localhost/raw >mbox ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 14:59 ` Nicolas Goaziou 2020-05-10 15:15 ` Kyle Meyer @ 2020-05-10 16:30 ` Ihor Radchenko 2020-05-10 19:32 ` Nicolas Goaziou 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-10 16:30 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > Unfortunately, reviewing this way is not nice. This should be better: https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930 > The `insert-and-inherit' issue sounds serious. We cannot reasonably > expect any library outside Org to use it. > > We could automatically extend invisible area with > `after-change-functions', i.e., if we're inserting something and both > side have the same `invisible' property, extend it. Using > `after-change-functions' sounds bad, but this kind of check shouldn't > cost much. > > WDYT? This might get tricky in the following case: :PROPERTIES: :CREATED: [2020-04-13 Mon 22:31] <region-beginning> :SHOWFROMDATE: 2020-05-11 :ID: e05e3b33-71ba-4bbc-abba-8a92c565ad34 :END: <many subtrees in between> :PROPERTIES: :CREATED: [2020-04-27 Mon 13:50] <region-end> :ID: b2eef49f-1c5c-4ff1-8e10-80423c8d8532 :ATTACH_DIR_INHERIT: t :END: If the text in the region is replaced by something else, <many subtrees in between> should not be fully hidden. We cannot simply look at the 'invisible property before and after the changed region. I think that using fontification (something similar to org-fontify-drawers) instead of after-change-functions should be faster. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> My response to you was blocked by your mail server: >> >>> 550 5.7.1 Reject for policy reason RULE3_2. See >>> http://postmaster.gandi.net > > Aka "spam detected". Bah. > >> The message landed on the mail list though: >> https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html > > Unfortunately, reviewing this way is not nice. > > The `insert-and-inherit' issue sounds serious. We cannot reasonably > expect any library outside Org to use it. > > We could automatically extend invisible area with > `after-change-functions', i.e., if we're inserting something and both > side have the same `invisible' property, extend it. Using > `after-change-functions' sounds bad, but this kind of check shouldn't > cost much. > > WDYT? > > Regards, -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 16:30 ` Ihor Radchenko @ 2020-05-10 19:32 ` Nicolas Goaziou 2020-05-12 10:03 ` Nicolas Goaziou 2020-05-17 15:00 ` Ihor Radchenko 0 siblings, 2 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-10 19:32 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > This should be better: > https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930 Thank you. > This might get tricky in the following case: > > :PROPERTIES: > :CREATED: [2020-04-13 Mon 22:31] > <region-beginning> > :SHOWFROMDATE: 2020-05-11 > :ID: e05e3b33-71ba-4bbc-abba-8a92c565ad34 > :END: > > <many subtrees in between> > > :PROPERTIES: > :CREATED: [2020-04-27 Mon 13:50] > <region-end> > :ID: b2eef49f-1c5c-4ff1-8e10-80423c8d8532 > :ATTACH_DIR_INHERIT: t > :END: > > If the text in the region is replaced by something else, <many subtrees > in between> should not be fully hidden. We cannot simply look at the > 'invisible property before and after the changed region. Be careful: "replaced by something else" is ambiguous. "Replacing" is an unlikely change: you would need to do (setf (buffer-substring x y) "foo") We can safely assume this will not happen. If it does, we can accept the subsequent glitch. Anyway it is less confusing to think in terms of deletion and insertion. In the case above, you probably mean "the region is deleted then something else is inserted", or the other way. So there are two actions going on, i.e., `after-change-functions' are called twice. In particular the situation you foresee /cannot happen/ with an insertion. Text is inserted at a single point. Let's assume this is in the first drawer. Once inserted, both text before and after the new text were part of the same drawer. Insertion introduces other problems, though. More on this below. It is true the deletion can produce the situation above. But in this case, there is nothing to do, you just merged two drawers into a single one, which stays invisible. Problem solved. IOW, big changes like the one you describe are not an issue. I think the "check if previous and next parts match" trick gives us roughly the same functionality, and the same glitches, as overlays. However, I think we can do better than that, and also fix the glitches from overlays. Here are two of them. Write the following drawer: :FOO: bar :END: fold it and delete the ":f". The overlay is still there, and you cannot remove it with TAB any more. Or, with the same initial drawer, from beginning of buffer, evaluate: (progn (re-search-forward ":END:") (replace-match "")) This is no longer a drawer: you just removed its closing line. Yet, the overlay is still there, and TAB is ineffective. Here's an idea to develop that would make folding more robust, and still fast. Each syntactical element has a "sensitive part", a particular area that can change the nature of the element when it is altered. Luckily drawers (and blocks) are sturdy. For a drawer, there are three things to check: 1. the opening line must match org-drawer-regexp 2. the closing line must match org-property-end-re (case ignored) 3. between those, you must not insert text match org-property-end-re or org-outline-regexp-bol Obviously, point 3 needs not be checked during deletion. Instead of `after-change-functions', we may use `modification-hooks' for deletions, and `insert-behind-hooks' for insertions. For example, we might add modification-hooks property to both opening and closing line, and `insert-behind-hooks' on all the drawer. If any of the 3 points above is verified, we remove all properties. Note that if we can implement something robust with text properties, we might use them for headlines too, for another significant speed-up. WDYT? > I think that using fontification (something similar to > org-fontify-drawers) instead of after-change-functions should be > faster. I don't think it would be faster. With `after-change-functions', `modification-hooks' or `insert-behind-hook', we know exactly where the change happened. Fontification is fuzzier. It is not instantaneous either. It is an option only if we cannot do something fast and accurate with `after-change-functions', IMO. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 19:32 ` Nicolas Goaziou @ 2020-05-12 10:03 ` Nicolas Goaziou 2020-05-17 15:00 ` Ihor Radchenko 1 sibling, 0 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-12 10:03 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Completing myself, Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Each syntactical element has a "sensitive part", a particular area that > can change the nature of the element when it is altered. Luckily drawers > (and blocks) are sturdy. For a drawer, there are three things to check: > > 1. the opening line must match org-drawer-regexp > 2. the closing line must match org-property-end-re (case ignored) > 3. between those, you must not insert text match org-property-end-re or > org-outline-regexp-bol > > Obviously, point 3 needs not be checked during deletion. Point 3 above is inaccurate, one also needs to check that "^[ \t]#\\+end[:_]" doesn't match the body, either. > Instead of `after-change-functions', we may use `modification-hooks' for > deletions, and `insert-behind-hooks' for insertions. For example, we > might add modification-hooks property to both opening and closing line, > and `insert-behind-hooks' on all the drawer. If any of the 3 points > above is verified, we remove all properties. > > Note that if we can implement something robust with text properties, we > might use them for headlines too, for another significant speed-up. Another, less ambitious, possibility is to expand the drawer as soon as text is inserted or removed in the invisible part. Callers (e.g., `org-entry-put') are then responsible to fold it again, if necessary. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-10 19:32 ` Nicolas Goaziou 2020-05-12 10:03 ` Nicolas Goaziou @ 2020-05-17 15:00 ` Ihor Radchenko 2020-05-17 15:40 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-17 15:00 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 6013 bytes --] Hi, [All the changes below are relative to commit ed0e75d24. Later commits make it hard to distinguish between hidden headlines and drawers. I will need to figure out a way to merge this branch with master. It does not seem to be trivial.] I have finished a seemingly stable implementation of handling changes inside drawer and block elements. For now, I did not bother with 'modification-hooks and 'insert-in-font/behind-hooks, but simply used before/after-change-functions. The basic idea is saving parsed org-elements before the modification (with :begin and :end replaced by markers) and comparing them with the versions of the same elements after the modification. Any valid org element can be examined in such way by an arbitrary function (see org-track-modification-elements) [1]. For now, I have implemented tracking changes in all the drawer and block elements. If the contents of an element is changed and the element is hidden, the contents remains hidden unless the change was done with self-insert-command. If the begin/end line of the element was changed in the way that the element changes the type or extends/shrinks, the element contents is revealed. To illustrate: Case #1 (the element content is hidden): :PROPERTIES: :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 :END: is changed to :ROPERTIES: :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 :END: Text is revealed, because we have drawer in place of property-drawer Case #2 (the element content is hidden): :ROPERTIES: :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 :END: is changed to :OPERTIES: :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 :END: The text remains hidden since it is still a drawer. Case #3: (the element content is hidden): :FOO: bar tmp :END: is changed to :FOO: bar :END: tmp :END: The text is revealed because the drawer contents shrank. Case #4: (the element content is hidden in both the drawers): :FOO: bar tmp :END: :BAR: jjd :END: is changed to :FOO: bar tmp :BAR: jjd :END: The text is revealed in both the drawers because the drawers are merged into a new drawer. > However, I think we can do better than that, and also fix the glitches > from overlays. Here are two of them. Write the following drawer: > > :FOO: > bar > :END: > > fold it and delete the ":f". The overlay is still there, and you cannot > remove it with TAB any more. Or, with the same initial drawer, from > beginning of buffer, evaluate: > > (progn (re-search-forward ":END:") (replace-match "")) > > This is no longer a drawer: you just removed its closing line. Yet, the > overlay is still there, and TAB is ineffective. I think the above examples cover what you described. Case #5 (the element content is hidden, point at <!>): :FOO:<!> bar tmp :END: is changed (via self-insert-command) to :FOO:a<!> bar tmp :END: The text is revealed. This last case sounds logical and might potentially replace org-catch-invisible-edits. ------------------------------------------------------------------------ Some potential issues with the implementation: 1. org--after-element-change-function can called many times even for trivial operations. For example (insert "\n" ":TEST:") seems to call it two times already. This has two implications: (1) potential performance degradation; (2) org-element library may not be able to parse the changed element because its intermediate modified state may not match the element syntax. Specifically, inserting new property into :PROPERTIES: drawer inserts a newline at some point, which makes org-element-at-point think that it is not a 'property-drawer, but just 'drawer. For (1), I did not really do any workaround for now. One potential way is making use of combine-after-change-calls (info:elisp#Change Hooks). At least, within org source code. For (2), I have introduced org--property-drawer-modified-re to override org-property-drawer-re in relevant *-change-function. This seems to work for property drawers. However, I am not sure if similar problem may happen in some border cases with ordinary drawers or blocks. 2. I have noticed that results of org-element-at-point and org-element-parse-buffer are not always consistent. In my tests, they returned different number of empty lines after drawers (:post-blank and :end properties). I am not sure here if I did something wrong in the code or if it is a real issue in org-element. For now, I simply called org-element-at-point with point at :begin property of all the elements returned by org-element-parse buffer to make things consistent. This indeed introduced overhead, but I do not see other way to solve the inconsistency. 3. This implementation did not directly solve the previously observed issue with two ellipsis displayed in folded drawer after adding hidden text inside: :PROPERTY: ... --> :PROPERTY: ... ... For now, I just did (org-hide-drawer-toggle 'off) (org-hide-drawer-toggle 'hide) to hide the second ellipsis, but I still don't understand why it is happening. Is it some Emacs bug? I am not sure. 4. For some reason, before/after-change-functions do not seem to trigger when adding note after todo state change. ------------------------------------------------------------------------ Further plans: 1. Investigate the issue with log notes. 2. Try to change headings to use text properties as well. The current version of the patch (relative to commit ed0e75d24) is attached. ------------------------------------------------------------------------ P.S. I have noticed an issue with hidden text on master (9bc0cc7fb) with my personal config: For the following .org file: * TODO b :PROPERTIES: :CREATED: [2020-05-17 Sun 22:37] :END: folded to * TODO b... Changing todo to DONE will be shown as * DONE b CLOSED: [2020-05-17 Sun 22:54]...:LOGBOOK:... ------------------------------------------------------------------------ [1] If one wants to track changes in two elements types, where one is always inside the other, it is not possible now. Best, Ihor [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: featuredrawertextprop.patch --] [-- Type: text/x-diff, Size: 17817 bytes --] diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..4b0e23f6a 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." \f -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + (pcase spec + ('outline + (remove-overlays from to 'invisible spec) + ;; Use `front-advance' since text right before to the beginning of + ;; the overlay belongs to the visible line than to the contents. + (when flag + (let ((o (make-overlay from to nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) + (_ + ;; Use text properties instead of overlays for speed. + ;; Overlays are too slow (Emacs Bug#35453). + (with-silent-modifications + (remove-text-properties from to '(invisible nil)) + (when flag + (put-text-property from to 'rear-non-sticky nil) + (put-text-property from to 'front-sticky t) + (put-text-property from to 'invisible spec)))))) \f ;;; Regexp matching diff --git a/lisp/org.el b/lisp/org.el index 96e7384f3..1bf90edae 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") (defvar ffap-url-regexp) (defvar org-element-paragraph-separate) +(defvar org-element-all-objects) +(defvar org-element-all-elements) +(defvar org-element-greater-elements) (defvar org-indent-indentation-per-level) (defvar org-radio-target-regexp) (defvar org-target-link-regexp) @@ -4737,6 +4741,153 @@ This is for getting out of special buffers like capture.") (defun org-before-change-function (_beg _end) "Every change indicates that a table might need an update." (setq org-table-may-need-update t)) + +(defvar-local org--modified-elements nil + "List of unmodified versions of recently modified elements. + +The :begin and :end element properties contain markers instead of positions.") + +(defvar org--property-drawer-modified-re (concat (replace-regexp-in-string "\\$$" "\n" org-property-start-re) + "\\(?:.*\n\\)*?" + (replace-regexp-in-string "^\\^" "" org-property-end-re)) + "Matches entire property drawer, including its state during modification. + +This should be different from `org-property-drawer-re' because +property drawer may contain empty or incomplete lines in the middle of +modification.") + +(defun org--drawer-or-block-change-function (el) + "Update visibility of changed drawer/block EL. + +If text was added to hidden drawer/block, +make sure that the text is also hidden, unless +the change was done by `self-insert-command'. +If the modification destroyed the drawer/block, +reveal the hidden text in former drawer/block." + (save-match-data + (save-excursion + (save-restriction + (goto-char (org-element-property :begin el)) + (let* ((newel (org-element-at-point)) + (spec (if (string-match-p "block" (symbol-name (org-element-type el))) + 'org-hide-block + (if (string-match-p "drawer" (symbol-name (org-element-type el))) + 'org-hide-drawer + t)))) + (if (and (equal (org-element-type el) (org-element-type newel)) + (equal (marker-position (org-element-property :begin el)) + (org-element-property :begin newel)) + (equal (marker-position (org-element-property :end el)) + (org-element-property :end newel))) + (when (text-property-any (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + 'invisible spec) + (if (memq this-command '(self-insert-command)) + ;; reveal if change was made by typing + (org-hide-drawer-toggle 'off) + ;; re-hide the inserted text + ;; FIXME: opening the drawer before hiding should not be needed here + (org-hide-drawer-toggle 'off) ; this is needed to avoid showing double ellipsis + (org-hide-drawer-toggle 'hide))) + ;; The element was destroyed. Reveal everything. + (org-flag-region (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + nil spec) + (org-flag-region (org-element-property :begin newel) + (org-element-property :end newel) + nil spec))))))) + +(defvar org-track-modification-elements (list (cons 'center-block #'org--drawer-or-block-change-function) + (cons 'drawer #'org--drawer-or-block-change-function) + (cons 'dynamic-block #'org--drawer-or-block-change-function) + (cons 'property-drawer #'org--drawer-or-block-change-function) + (cons 'quote-block #'org--drawer-or-block-change-function) + (cons 'special-block #'org--drawer-or-block-change-function)) + "Alist of elements to be tracked for modifications. +Each element of the alist is a cons of an element from +`org-element-all-elements' and the function used to handle the +modification. +The function must accept a single argument - parsed element before +modificatin with :begin and :end properties containing markers.") + +(defun org--find-elements-in-region (beg end elements &optional include-partial) + "Find all elements from ELEMENTS list in region BEG . END. +All the listed elements must be resolvable by `org-element-at-point'. +Include elements if they are partially inside region when INCLUDE-PARTIAL is non-nil." + (when include-partial + (org-with-point-at beg + (when-let ((new-beg (org-element-property :begin + (org-element-lineage (org-element-at-point) + elements + 'with-self)))) + (setq beg new-beg)) + (when (memq 'headline elements) + (when-let ((new-beg (ignore-error user-error (org-back-to-heading 'include-invisible)))) + (setq beg new-beg)))) + (org-with-point-at end + (when-let ((new-end (org-element-property :end + (org-element-lineage (org-element-at-point) + elements + 'with-self)))) + (setq end new-end)) + (when (memq 'headline elements) + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) + (setq end (1- new-end)))))) + (save-excursion + (save-restriction + (narrow-to-region beg end) + (let (has-object has-element has-greater-element granularity) + (dolist (el elements) + (when (memq el org-element-all-objects) (setq has-object t)) + (when (memq el org-element-all-elements) (setq has-element t)) + (when (memq el org-element-greater-elements) (setq has-greater-element t))) + (if has-object + (setq granularity 'object) + (if has-greater-element + (setq granularity 'greater-element) + (if has-element + (setq granularity 'element) + (setq granularity 'headline)))) + (org-element-map (org-element-parse-buffer granularity) elements #'identity))))) + +(defun org--before-element-change-function (beg end) + "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END." + (let ((org-property-drawer-re org--property-drawer-modified-re)) + (save-match-data + (save-excursion + (save-restriction + (dolist (el (org--find-elements-in-region beg + end + (mapcar #'car org-track-modification-elements) + 'include-partial)) + ;; `org-element-at-point' is not consistent with results + ;; of `org-element-parse-buffer' for :post-blank and :end + ;; Using `org-element-at-point to keep consistent + ;; parse results with `org--after-element-change-function' + (let* ((el (org-with-point-at (org-element-property :begin el) + (org-element-at-point))) + (beg-marker (copy-marker (org-element-property :begin el) 't)) + (end-marker (copy-marker (org-element-property :end el) 't))) + (when (and (marker-position beg-marker) (marker-position end-marker)) + (org-element-put-property el :begin beg-marker) + (org-element-put-property el :end end-marker) + (add-to-list 'org--modified-elements el))))))))) + +;; FIXME: this function may be called many times during routine modifications +;; The normal way to avoid this is `combine-after-change-calls' - not +;; the case in most org primitives. +(defun org--after-element-change-function (&rest _) + "Handle changed elements from `org--modified-elements'." + (let ((org-property-drawer-re org--property-drawer-modified-re)) + (dolist (el org--modified-elements) + (save-match-data + (save-excursion + (save-restriction + (let* ((type (org-element-type el)) + (change-func (alist-get type org-track-modification-elements))) + (funcall (symbol-function change-func) el))))))) + (setq org--modified-elements nil)) + (defvar org-mode-map) (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. @@ -4818,6 +4969,9 @@ The following commands are available: ;; Activate before-change-function (setq-local org-table-may-need-update t) (add-hook 'before-change-functions 'org-before-change-function nil 'local) + (add-hook 'before-change-functions 'org--before-element-change-function nil 'local) + ;; Activate after-change-function + (add-hook 'after-change-functions 'org--after-element-change-function nil 'local) ;; Check for running clock before killing a buffer (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) ;; Initialize macros templates. @@ -4869,6 +5023,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5859,9 +6017,26 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) + ;; do not remove invisible text properties specified by + ;; 'org-hide-block and 'org-hide-drawer (but remove 'org-link) + ;; this is needed to keep the drawers and blocks hidden unless + ;; they are toggled by user + ;; Note: The below may be too specific and create troubles + ;; if more invisibility specs are added to org in future + (let ((pos beg) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer)) + (remove-text-properties pos next '(invisible t))) + (setq pos next))) (remove-text-properties beg end '(mouse-face t keymap t org-linked-text t - invisible t intangible t + ;; Do not remove all invisible during fontification + ;; invisible t + intangible t org-emphasis t)) (org-remove-font-lock-display-properties beg end))) @@ -6666,8 +6841,13 @@ information." ;; expose it. (dolist (o (overlays-at (point))) (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) + '(outline)) (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -20902,6 +21082,79 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) \f + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (remove-text-properties (car region) (cdr region) '(invisible nil)))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + +\f + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode [-- Attachment #3: Type: text/plain, Size: 4435 bytes --] Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> This should be better: >> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930 > > Thank you. > >> This might get tricky in the following case: >> >> :PROPERTIES: >> :CREATED: [2020-04-13 Mon 22:31] >> <region-beginning> >> :SHOWFROMDATE: 2020-05-11 >> :ID: e05e3b33-71ba-4bbc-abba-8a92c565ad34 >> :END: >> >> <many subtrees in between> >> >> :PROPERTIES: >> :CREATED: [2020-04-27 Mon 13:50] >> <region-end> >> :ID: b2eef49f-1c5c-4ff1-8e10-80423c8d8532 >> :ATTACH_DIR_INHERIT: t >> :END: >> >> If the text in the region is replaced by something else, <many subtrees >> in between> should not be fully hidden. We cannot simply look at the >> 'invisible property before and after the changed region. > > Be careful: "replaced by something else" is ambiguous. "Replacing" is an > unlikely change: you would need to do > > (setf (buffer-substring x y) "foo") > > We can safely assume this will not happen. If it does, we can accept the > subsequent glitch. > > Anyway it is less confusing to think in terms of deletion and insertion. > In the case above, you probably mean "the region is deleted then > something else is inserted", or the other way. So there are two actions > going on, i.e., `after-change-functions' are called twice. > > In particular the situation you foresee /cannot happen/ with an > insertion. Text is inserted at a single point. Let's assume this is in > the first drawer. Once inserted, both text before and after the new text > were part of the same drawer. Insertion introduces other problems, > though. More on this below. > > It is true the deletion can produce the situation above. But in this > case, there is nothing to do, you just merged two drawers into a single > one, which stays invisible. Problem solved. > > IOW, big changes like the one you describe are not an issue. I think the > "check if previous and next parts match" trick gives us roughly the same > functionality, and the same glitches, as overlays. > > However, I think we can do better than that, and also fix the glitches > from overlays. Here are two of them. Write the following drawer: > > :FOO: > bar > :END: > > fold it and delete the ":f". The overlay is still there, and you cannot > remove it with TAB any more. Or, with the same initial drawer, from > beginning of buffer, evaluate: > > (progn (re-search-forward ":END:") (replace-match "")) > > This is no longer a drawer: you just removed its closing line. Yet, the > overlay is still there, and TAB is ineffective. > > Here's an idea to develop that would make folding more robust, and still > fast. > > Each syntactical element has a "sensitive part", a particular area that > can change the nature of the element when it is altered. Luckily drawers > (and blocks) are sturdy. For a drawer, there are three things to check: > > 1. the opening line must match org-drawer-regexp > 2. the closing line must match org-property-end-re (case ignored) > 3. between those, you must not insert text match org-property-end-re or > org-outline-regexp-bol > > Obviously, point 3 needs not be checked during deletion. > > Instead of `after-change-functions', we may use `modification-hooks' for > deletions, and `insert-behind-hooks' for insertions. For example, we > might add modification-hooks property to both opening and closing line, > and `insert-behind-hooks' on all the drawer. If any of the 3 points > above is verified, we remove all properties. > > Note that if we can implement something robust with text properties, we > might use them for headlines too, for another significant speed-up. > > WDYT? > >> I think that using fontification (something similar to >> org-fontify-drawers) instead of after-change-functions should be >> faster. > > I don't think it would be faster. With `after-change-functions', > `modification-hooks' or `insert-behind-hook', we know exactly where the > change happened. Fontification is fuzzier. It is not instantaneous > either. > > It is an option only if we cannot do something fast and accurate with > `after-change-functions', IMO. > -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply related [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-17 15:00 ` Ihor Radchenko @ 2020-05-17 15:40 ` Ihor Radchenko 2020-05-18 14:35 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-17 15:40 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Dear Nicolas Goaziou, Apparently my previous email was again refused by your mail server (I tried to add patch as attachment this time). The patch is in https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef This patch is actually one commit ahead of the patch in the email, fixing an issue when change function throws an error. I wrapped the call into with-demoted-errors to avoid potential data loss on error in future. Best, Ihor Ihor Radchenko <yantar92@gmail.com> writes: > Hi, > > [All the changes below are relative to commit ed0e75d24. Later commits > make it hard to distinguish between hidden headlines and drawers. I will > need to figure out a way to merge this branch with master. It does not > seem to be trivial.] > > I have finished a seemingly stable implementation of handling changes > inside drawer and block elements. For now, I did not bother with > 'modification-hooks and 'insert-in-font/behind-hooks, but simply used > before/after-change-functions. > > The basic idea is saving parsed org-elements before the modification > (with :begin and :end replaced by markers) and comparing them with the > versions of the same elements after the modification. > Any valid org element can be examined in such way by an arbitrary > function (see org-track-modification-elements) [1]. > > For now, I have implemented tracking changes in all the drawer and block > elements. If the contents of an element is changed and the element is > hidden, the contents remains hidden unless the change was done with > self-insert-command. If the begin/end line of the element was changed in > the way that the element changes the type or extends/shrinks, the > element contents is revealed. To illustrate: > > Case #1 (the element content is hidden): > > :PROPERTIES: > :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > :END: > > is changed to > > :ROPERTIES: > :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > :END: > > Text is revealed, because we have drawer in place of property-drawer > > Case #2 (the element content is hidden): > > :ROPERTIES: > :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > :END: > > is changed to > > :OPERTIES: > :ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > :END: > > The text remains hidden since it is still a drawer. > > Case #3: (the element content is hidden): > > :FOO: > bar > tmp > :END: > > is changed to > > :FOO: > bar > :END: > tmp > :END: > > The text is revealed because the drawer contents shrank. > > Case #4: (the element content is hidden in both the drawers): > > :FOO: > bar > tmp > :END: > :BAR: > jjd > :END: > > is changed to > > :FOO: > bar > tmp > :BAR: > jjd > :END: > > The text is revealed in both the drawers because the drawers are merged > into a new drawer. > >> However, I think we can do better than that, and also fix the glitches >> from overlays. Here are two of them. Write the following drawer: >> >> :FOO: >> bar >> :END: >> >> fold it and delete the ":f". The overlay is still there, and you cannot >> remove it with TAB any more. Or, with the same initial drawer, from >> beginning of buffer, evaluate: >> >> (progn (re-search-forward ":END:") (replace-match "")) >> >> This is no longer a drawer: you just removed its closing line. Yet, the >> overlay is still there, and TAB is ineffective. > > I think the above examples cover what you described. > > Case #5 (the element content is hidden, point at <!>): > > :FOO:<!> > bar > tmp > :END: > > is changed (via self-insert-command) to > > :FOO:a<!> > bar > tmp > :END: > > The text is revealed. > > This last case sounds logical and might potentially replace > org-catch-invisible-edits. > > ------------------------------------------------------------------------ > > Some potential issues with the implementation: > > 1. org--after-element-change-function can called many times even for > trivial operations. For example (insert "\n" ":TEST:") seems to call it > two times already. This has two implications: (1) potential performance > degradation; (2) org-element library may not be able to parse the > changed element because its intermediate modified state may not match > the element syntax. Specifically, inserting new property into > :PROPERTIES: drawer inserts a newline at some point, which makes > org-element-at-point think that it is not a 'property-drawer, but just > 'drawer. > > For (1), I did not really do any workaround for now. One potential way > is making use of combine-after-change-calls (info:elisp#Change Hooks). > At least, within org source code. > > For (2), I have introduced org--property-drawer-modified-re to override > org-property-drawer-re in relevant *-change-function. This seems to work > for property drawers. However, I am not sure if similar problem may > happen in some border cases with ordinary drawers or blocks. > > 2. I have noticed that results of org-element-at-point and > org-element-parse-buffer are not always consistent. > In my tests, they returned different number of empty lines after drawers > (:post-blank and :end properties). I am not sure here if I did something > wrong in the code or if it is a real issue in org-element. > > For now, I simply called org-element-at-point with point at :begin > property of all the elements returned by org-element-parse buffer to > make things consistent. This indeed introduced overhead, but I do not > see other way to solve the inconsistency. > > 3. This implementation did not directly solve the previously observed > issue with two ellipsis displayed in folded drawer after adding hidden > text inside: > > :PROPERTY: ... --> :PROPERTY: ... ... > > For now, I just did > > (org-hide-drawer-toggle 'off) > (org-hide-drawer-toggle 'hide) > > to hide the second ellipsis, but I still don't understand why it is > happening. Is it some Emacs bug? I am not sure. > > 4. For some reason, before/after-change-functions do not seem to trigger > when adding note after todo state change. > > ------------------------------------------------------------------------ > > Further plans: > > 1. Investigate the issue with log notes. > 2. Try to change headings to use text properties as well. > > The current version of the patch (relative to commit ed0e75d24) is > attached. > > ------------------------------------------------------------------------ > > P.S. I have noticed an issue with hidden text on master (9bc0cc7fb) with > my personal config: > > For the following .org file: > > * TODO b > :PROPERTIES: > :CREATED: [2020-05-17 Sun 22:37] > :END: > > folded to > > * TODO b... > > Changing todo to DONE will be shown as > > * DONE b > CLOSED: [2020-05-17 Sun 22:54]...:LOGBOOK:... > > ------------------------------------------------------------------------ > > [1] If one wants to track changes in two elements types, where one is > always inside the other, it is not possible now. > > Best, > Ihor > > diff --git a/lisp/org-macs.el b/lisp/org-macs.el > index a02f713ca..4b0e23f6a 100644 > --- a/lisp/org-macs.el > +++ b/lisp/org-macs.el > @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." > > > \f > -;;; Overlays > +;;; Overlays and text properties > > (defun org-overlay-display (ovl text &optional face evap) > "Make overlay OVL display TEXT with face FACE." > @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays." > (delete (delete-overlay ov)) > (t (push ov found)))))) > > +(defun org--find-text-property-region (pos prop) > + "Find a region containing PROP text property around point POS." > + (let* ((beg (and (get-text-property pos prop) pos)) > + (end beg)) > + (when beg > + ;; when beg is the first point in the region, `previous-single-property-change' > + ;; will return nil. > + (setq beg (or (previous-single-property-change pos prop) > + beg)) > + ;; when end is the last point in the region, `next-single-property-change' > + ;; will return nil. > + (setq end (or (next-single-property-change pos prop) > + end)) > + (unless (= beg end) ; this should not happen > + (cons beg end))))) > + > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > - (remove-overlays from to 'invisible spec) > - ;; Use `front-advance' since text right before to the beginning of > - ;; the overlay belongs to the visible line than to the contents. > - (when flag > - (let ((o (make-overlay from to nil 'front-advance))) > - (overlay-put o 'evaporate t) > - (overlay-put o 'invisible spec) > - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > - > + (pcase spec > + ('outline > + (remove-overlays from to 'invisible spec) > + ;; Use `front-advance' since text right before to the beginning of > + ;; the overlay belongs to the visible line than to the contents. > + (when flag > + (let ((o (make-overlay from to nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > + (_ > + ;; Use text properties instead of overlays for speed. > + ;; Overlays are too slow (Emacs Bug#35453). > + (with-silent-modifications > + (remove-text-properties from to '(invisible nil)) > + (when flag > + (put-text-property from to 'rear-non-sticky nil) > + (put-text-property from to 'front-sticky t) > + (put-text-property from to 'invisible spec)))))) > > \f > ;;; Regexp matching > diff --git a/lisp/org.el b/lisp/org.el > index 96e7384f3..1bf90edae 100644 > --- a/lisp/org.el > +++ b/lisp/org.el > @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") > (declare-function cdlatex-math-symbol "ext:cdlatex") > (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) > (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) > +(declare-function isearch-filter-visible "isearch" (beg end)) > (declare-function org-add-archive-files "org-archive" (files)) > (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) > (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) > @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") > > (defvar ffap-url-regexp) > (defvar org-element-paragraph-separate) > +(defvar org-element-all-objects) > +(defvar org-element-all-elements) > +(defvar org-element-greater-elements) > (defvar org-indent-indentation-per-level) > (defvar org-radio-target-regexp) > (defvar org-target-link-regexp) > @@ -4737,6 +4741,153 @@ This is for getting out of special buffers like capture.") > (defun org-before-change-function (_beg _end) > "Every change indicates that a table might need an update." > (setq org-table-may-need-update t)) > + > +(defvar-local org--modified-elements nil > + "List of unmodified versions of recently modified elements. > + > +The :begin and :end element properties contain markers instead of positions.") > + > +(defvar org--property-drawer-modified-re (concat (replace-regexp-in-string "\\$$" "\n" org-property-start-re) > + "\\(?:.*\n\\)*?" > + (replace-regexp-in-string "^\\^" "" org-property-end-re)) > + "Matches entire property drawer, including its state during modification. > + > +This should be different from `org-property-drawer-re' because > +property drawer may contain empty or incomplete lines in the middle of > +modification.") > + > +(defun org--drawer-or-block-change-function (el) > + "Update visibility of changed drawer/block EL. > + > +If text was added to hidden drawer/block, > +make sure that the text is also hidden, unless > +the change was done by `self-insert-command'. > +If the modification destroyed the drawer/block, > +reveal the hidden text in former drawer/block." > + (save-match-data > + (save-excursion > + (save-restriction > + (goto-char (org-element-property :begin el)) > + (let* ((newel (org-element-at-point)) > + (spec (if (string-match-p "block" (symbol-name (org-element-type el))) > + 'org-hide-block > + (if (string-match-p "drawer" (symbol-name (org-element-type el))) > + 'org-hide-drawer > + t)))) > + (if (and (equal (org-element-type el) (org-element-type newel)) > + (equal (marker-position (org-element-property :begin el)) > + (org-element-property :begin newel)) > + (equal (marker-position (org-element-property :end el)) > + (org-element-property :end newel))) > + (when (text-property-any (marker-position (org-element-property :begin el)) > + (marker-position (org-element-property :end el)) > + 'invisible spec) > + (if (memq this-command '(self-insert-command)) > + ;; reveal if change was made by typing > + (org-hide-drawer-toggle 'off) > + ;; re-hide the inserted text > + ;; FIXME: opening the drawer before hiding should not be needed here > + (org-hide-drawer-toggle 'off) ; this is needed to avoid showing double ellipsis > + (org-hide-drawer-toggle 'hide))) > + ;; The element was destroyed. Reveal everything. > + (org-flag-region (marker-position (org-element-property :begin el)) > + (marker-position (org-element-property :end el)) > + nil spec) > + (org-flag-region (org-element-property :begin newel) > + (org-element-property :end newel) > + nil spec))))))) > + > +(defvar org-track-modification-elements (list (cons 'center-block #'org--drawer-or-block-change-function) > + (cons 'drawer #'org--drawer-or-block-change-function) > + (cons 'dynamic-block #'org--drawer-or-block-change-function) > + (cons 'property-drawer #'org--drawer-or-block-change-function) > + (cons 'quote-block #'org--drawer-or-block-change-function) > + (cons 'special-block #'org--drawer-or-block-change-function)) > + "Alist of elements to be tracked for modifications. > +Each element of the alist is a cons of an element from > +`org-element-all-elements' and the function used to handle the > +modification. > +The function must accept a single argument - parsed element before > +modificatin with :begin and :end properties containing markers.") > + > +(defun org--find-elements-in-region (beg end elements &optional include-partial) > + "Find all elements from ELEMENTS list in region BEG . END. > +All the listed elements must be resolvable by `org-element-at-point'. > +Include elements if they are partially inside region when INCLUDE-PARTIAL is non-nil." > + (when include-partial > + (org-with-point-at beg > + (when-let ((new-beg (org-element-property :begin > + (org-element-lineage (org-element-at-point) > + elements > + 'with-self)))) > + (setq beg new-beg)) > + (when (memq 'headline elements) > + (when-let ((new-beg (ignore-error user-error (org-back-to-heading 'include-invisible)))) > + (setq beg new-beg)))) > + (org-with-point-at end > + (when-let ((new-end (org-element-property :end > + (org-element-lineage (org-element-at-point) > + elements > + 'with-self)))) > + (setq end new-end)) > + (when (memq 'headline elements) > + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) > + (setq end (1- new-end)))))) > + (save-excursion > + (save-restriction > + (narrow-to-region beg end) > + (let (has-object has-element has-greater-element granularity) > + (dolist (el elements) > + (when (memq el org-element-all-objects) (setq has-object t)) > + (when (memq el org-element-all-elements) (setq has-element t)) > + (when (memq el org-element-greater-elements) (setq has-greater-element t))) > + (if has-object > + (setq granularity 'object) > + (if has-greater-element > + (setq granularity 'greater-element) > + (if has-element > + (setq granularity 'element) > + (setq granularity 'headline)))) > + (org-element-map (org-element-parse-buffer granularity) elements #'identity))))) > + > +(defun org--before-element-change-function (beg end) > + "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END." > + (let ((org-property-drawer-re org--property-drawer-modified-re)) > + (save-match-data > + (save-excursion > + (save-restriction > + (dolist (el (org--find-elements-in-region beg > + end > + (mapcar #'car org-track-modification-elements) > + 'include-partial)) > + ;; `org-element-at-point' is not consistent with results > + ;; of `org-element-parse-buffer' for :post-blank and :end > + ;; Using `org-element-at-point to keep consistent > + ;; parse results with `org--after-element-change-function' > + (let* ((el (org-with-point-at (org-element-property :begin el) > + (org-element-at-point))) > + (beg-marker (copy-marker (org-element-property :begin el) 't)) > + (end-marker (copy-marker (org-element-property :end el) 't))) > + (when (and (marker-position beg-marker) (marker-position end-marker)) > + (org-element-put-property el :begin beg-marker) > + (org-element-put-property el :end end-marker) > + (add-to-list 'org--modified-elements el))))))))) > + > +;; FIXME: this function may be called many times during routine modifications > +;; The normal way to avoid this is `combine-after-change-calls' - not > +;; the case in most org primitives. > +(defun org--after-element-change-function (&rest _) > + "Handle changed elements from `org--modified-elements'." > + (let ((org-property-drawer-re org--property-drawer-modified-re)) > + (dolist (el org--modified-elements) > + (save-match-data > + (save-excursion > + (save-restriction > + (let* ((type (org-element-type el)) > + (change-func (alist-get type org-track-modification-elements))) > + (funcall (symbol-function change-func) el))))))) > + (setq org--modified-elements nil)) > + > (defvar org-mode-map) > (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. > (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. > @@ -4818,6 +4969,9 @@ The following commands are available: > ;; Activate before-change-function > (setq-local org-table-may-need-update t) > (add-hook 'before-change-functions 'org-before-change-function nil 'local) > + (add-hook 'before-change-functions 'org--before-element-change-function nil 'local) > + ;; Activate after-change-function > + (add-hook 'after-change-functions 'org--after-element-change-function nil 'local) > ;; Check for running clock before killing a buffer > (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) > ;; Initialize macros templates. > @@ -4869,6 +5023,10 @@ The following commands are available: > (setq-local outline-isearch-open-invisible-function > (lambda (&rest _) (org-show-context 'isearch))) > > + ;; Make isearch search in blocks hidden via text properties > + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) > + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) > + > ;; Setup the pcomplete hooks > (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) > (setq-local pcomplete-command-name-function #'org-command-at-point) > @@ -5859,9 +6017,26 @@ If TAG is a number, get the corresponding match group." > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > + ;; do not remove invisible text properties specified by > + ;; 'org-hide-block and 'org-hide-drawer (but remove 'org-link) > + ;; this is needed to keep the drawers and blocks hidden unless > + ;; they are toggled by user > + ;; Note: The below may be too specific and create troubles > + ;; if more invisibility specs are added to org in future > + (let ((pos beg) > + next spec) > + (while (< pos end) > + (setq next (next-single-property-change pos 'invisible nil end) > + spec (get-text-property pos 'invisible)) > + (unless (memq spec (list 'org-hide-block > + 'org-hide-drawer)) > + (remove-text-properties pos next '(invisible t))) > + (setq pos next))) > (remove-text-properties beg end > '(mouse-face t keymap t org-linked-text t > - invisible t intangible t > + ;; Do not remove all invisible during fontification > + ;; invisible t > + intangible t > org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) > > @@ -6666,8 +6841,13 @@ information." > ;; expose it. > (dolist (o (overlays-at (point))) > (when (memq (overlay-get o 'invisible) > - '(org-hide-block org-hide-drawer outline)) > + '(outline)) > (delete-overlay o))) > + (when (memq (get-text-property (point) 'invisible) > + '(org-hide-block org-hide-drawer)) > + (let ((spec (get-text-property (point) 'invisible)) > + (region (org--find-text-property-region (point) 'invisible))) > + (org-flag-region (car region) (cdr region) nil spec))) > (unless (org-before-first-heading-p) > (org-with-limited-levels > (cl-case detail > @@ -20902,6 +21082,79 @@ Started from `gnus-info-find-node'." > (t default-org-info-node)))))) > > \f > + > +;;; Make isearch search in some text hidden via text propertoes > + > +(defvar org--isearch-overlays nil > + "List of overlays temporarily created during isearch. > +This is used to allow searching in regions hidden via text properties. > +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. > +Any text hidden via text properties is not revealed even if `search-invisible' > +is set to 't.") > + > +;; Not sure if it needs to be a user option > +;; One might want to reveal hidden text in, for example, hidden parts of the links. > +;; Currently, hidden text in links is never revealed by isearch. > +(defvar org-isearch-specs '(org-hide-block > + org-hide-drawer) > + "List of text invisibility specs to be searched by isearch. > +By default ([2020-05-09 Sat]), isearch does not search in hidden text, > +which was made invisible using text properties. Isearch will be forced > +to search in hidden text with any of the listed 'invisible property value.") > + > +(defun org--create-isearch-overlays (beg end) > + "Replace text property invisibility spec by overlays between BEG and END. > +All the regions with invisibility text property spec from > +`org-isearch-specs' will be changed to use overlays instead > +of text properties. The created overlays will be stored in > +`org--isearch-overlays'." > + (let ((pos beg)) > + (while (< pos end) > + (when-let* ((spec (get-text-property pos 'invisible)) > + (spec (memq spec org-isearch-specs)) > + (region (org--find-text-property-region pos 'invisible))) > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] > + ;; overlay for 'outline blocks. > + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + ;; `delete-overlay' here means that spec information will be lost > + ;; for the region. The region will remain visible. > + (overlay-put o 'isearch-open-invisible #'delete-overlay) > + (push o org--isearch-overlays)) > + (remove-text-properties (car region) (cdr region) '(invisible nil)))) > + (setq pos (next-single-property-change pos 'invisible nil end))))) > + > +(defun org--isearch-filter-predicate (beg end) > + "Return non-nil if text between BEG and END is deemed visible by Isearch. > +This function is intended to be used as `isearch-filter-predicate'. > +Unlike `isearch-filter-visible', make text with 'invisible text property > +value listed in `org-isearch-specs' visible to Isearch." > + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text > + (isearch-filter-visible beg end)) > + > +(defun org--clear-isearch-overlay (ov) > + "Convert OV region back into using text properties." > + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec))) > + (when (member ov isearch-opened-overlays) > + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) > + (delete-overlay ov)) > + > +(defun org--clear-isearch-overlays () > + "Convert overlays from `org--isearch-overlays' back into using text properties." > + (when org--isearch-overlays > + (mapc #'org--clear-isearch-overlay org--isearch-overlays) > + (setq org--isearch-overlays nil))) > + > +\f > + > ;;; Finish up > > (add-hook 'org-mode-hook ;remove overlays when changing major mode > > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>> This should be better: >>> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930 >> >> Thank you. >> >>> This might get tricky in the following case: >>> >>> :PROPERTIES: >>> :CREATED: [2020-04-13 Mon 22:31] >>> <region-beginning> >>> :SHOWFROMDATE: 2020-05-11 >>> :ID: e05e3b33-71ba-4bbc-abba-8a92c565ad34 >>> :END: >>> >>> <many subtrees in between> >>> >>> :PROPERTIES: >>> :CREATED: [2020-04-27 Mon 13:50] >>> <region-end> >>> :ID: b2eef49f-1c5c-4ff1-8e10-80423c8d8532 >>> :ATTACH_DIR_INHERIT: t >>> :END: >>> >>> If the text in the region is replaced by something else, <many subtrees >>> in between> should not be fully hidden. We cannot simply look at the >>> 'invisible property before and after the changed region. >> >> Be careful: "replaced by something else" is ambiguous. "Replacing" is an >> unlikely change: you would need to do >> >> (setf (buffer-substring x y) "foo") >> >> We can safely assume this will not happen. If it does, we can accept the >> subsequent glitch. >> >> Anyway it is less confusing to think in terms of deletion and insertion. >> In the case above, you probably mean "the region is deleted then >> something else is inserted", or the other way. So there are two actions >> going on, i.e., `after-change-functions' are called twice. >> >> In particular the situation you foresee /cannot happen/ with an >> insertion. Text is inserted at a single point. Let's assume this is in >> the first drawer. Once inserted, both text before and after the new text >> were part of the same drawer. Insertion introduces other problems, >> though. More on this below. >> >> It is true the deletion can produce the situation above. But in this >> case, there is nothing to do, you just merged two drawers into a single >> one, which stays invisible. Problem solved. >> >> IOW, big changes like the one you describe are not an issue. I think the >> "check if previous and next parts match" trick gives us roughly the same >> functionality, and the same glitches, as overlays. >> >> However, I think we can do better than that, and also fix the glitches >> from overlays. Here are two of them. Write the following drawer: >> >> :FOO: >> bar >> :END: >> >> fold it and delete the ":f". The overlay is still there, and you cannot >> remove it with TAB any more. Or, with the same initial drawer, from >> beginning of buffer, evaluate: >> >> (progn (re-search-forward ":END:") (replace-match "")) >> >> This is no longer a drawer: you just removed its closing line. Yet, the >> overlay is still there, and TAB is ineffective. >> >> Here's an idea to develop that would make folding more robust, and still >> fast. >> >> Each syntactical element has a "sensitive part", a particular area that >> can change the nature of the element when it is altered. Luckily drawers >> (and blocks) are sturdy. For a drawer, there are three things to check: >> >> 1. the opening line must match org-drawer-regexp >> 2. the closing line must match org-property-end-re (case ignored) >> 3. between those, you must not insert text match org-property-end-re or >> org-outline-regexp-bol >> >> Obviously, point 3 needs not be checked during deletion. >> >> Instead of `after-change-functions', we may use `modification-hooks' for >> deletions, and `insert-behind-hooks' for insertions. For example, we >> might add modification-hooks property to both opening and closing line, >> and `insert-behind-hooks' on all the drawer. If any of the 3 points >> above is verified, we remove all properties. >> >> Note that if we can implement something robust with text properties, we >> might use them for headlines too, for another significant speed-up. >> >> WDYT? >> >>> I think that using fontification (something similar to >>> org-fontify-drawers) instead of after-change-functions should be >>> faster. >> >> I don't think it would be faster. With `after-change-functions', >> `modification-hooks' or `insert-behind-hook', we know exactly where the >> change happened. Fontification is fuzzier. It is not instantaneous >> either. >> >> It is an option only if we cannot do something fast and accurate with >> `after-change-functions', IMO. >> > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-17 15:40 ` Ihor Radchenko @ 2020-05-18 14:35 ` Nicolas Goaziou 2020-05-18 16:52 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-18 14:35 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > Apparently my previous email was again refused by your mail server (I > tried to add patch as attachment this time). Ah. This is annoying, for you and for me. > The patch is in > https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef Thank you. >> I have finished a seemingly stable implementation of handling changes >> inside drawer and block elements. For now, I did not bother with >> 'modification-hooks and 'insert-in-font/behind-hooks, but simply used >> before/after-change-functions. >> >> The basic idea is saving parsed org-elements before the modification >> (with :begin and :end replaced by markers) and comparing them with the >> versions of the same elements after the modification. >> Any valid org element can be examined in such way by an arbitrary >> function (see org-track-modification-elements) [1]. As you noticed, using Org Element is a no-go, unfortunately. Parsing an element is a O(N) operation by the number of elements before it in a section. In particular, it is not bounded, and not mitigated by a cache. For large documents, it is going to be unbearably slow, too. I don't think the solution is to use combine-after-change-calls either, because even a single call to `org-element-at-point' can be noticeable in a very large section. Such low-level code should avoid using the Element library altogether, except for the initial folding part, which is interactive. If you use modification-hooks and al., you don't need to parse anything, because you can store information as text properties. Therefore, once the modification happens, you already know where you are (or, at least where you were before the change). The ideas I suggested about sensitive parts of elements are worth exploring, IMO. Do you have any issue with them? >> For (2), I have introduced org--property-drawer-modified-re to override >> org-property-drawer-re in relevant *-change-function. This seems to work >> for property drawers. However, I am not sure if similar problem may >> happen in some border cases with ordinary drawers or blocks. I already specified what parts were "sensitive" in a previous message. >> 2. I have noticed that results of org-element-at-point and >> org-element-parse-buffer are not always consistent. `org-element-at-point' is local, `org-element-parse-buffer' is global. They are not equivalent, but is it an issue? Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-18 14:35 ` Nicolas Goaziou @ 2020-05-18 16:52 ` Ihor Radchenko 2020-05-19 13:07 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-18 16:52 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > As you noticed, using Org Element is a no-go, unfortunately. Parsing an > element is a O(N) operation by the number of elements before it in > a section. In particular, it is not bounded, and not mitigated by > a cache. For large documents, it is going to be unbearably slow, too. Ouch. I thought it is faster. What do you mean by "not mitigated by a cache"? The reason I would like to utilise org-element parser to make tracking modifications more robust. Using details of the syntax would make the code fragile if any modifications are made to syntax in future. Debugging bugs in modification functions is not easy, according to my experience. One possible way to avoid performance issues during modification is running parser in advance. For example, folding an element may as well add information about the element to its text properties. This will not degrade performance of folding since we are already parsing the element during folding (at least, in org-hide-drawer-toggle). The problem with parsing an element during folding is that we cannot easily detect changes like below without re-parsing. :PROPERTIES: <folded> :CREATED: [2020-05-18 Mon] :END: <- added line :ID: test :END: or even :PROPERTIES: :CREATED: [2020-05-18 Mon] :ID: test :END: <- delete this line :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion> test :END: The re-parsing can be done via regexp, as you suggested, but I don't like this idea, because it will end up re-implementing org-element-*-parser. Would it be acceptable to run org-element-*-parser in after-change-functions? > If you use modification-hooks and al., you don't need to parse anything, > because you can store information as text properties. Therefore, once > the modification happens, you already know where you are (or, at least > where you were before the change). > The ideas I suggested about sensitive parts of elements are worth > exploring, IMO. Do you have any issue with them? If I understand correctly, it is not as easy. Consider the following example: :PROPERTIES: :CREATED: [2020-05-18 Mon] <region-beginning> :ID: example :END: <... a lot of text, maybe containing other drawers ...> Nullam rutrum. Pellentesque dapibus suscipit ligula. <region-end> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. If the region gets deleted, the modification hooks from chars inside drawer will be called as (hook-function <region-beginning> <region-end>). So, there is still a need to find the drawer somehow to mark it as about to be modified (modification hooks are ran before actual modification). The only difference between using modification hooks and before-change-functions is that modification hooks will trigger less frequently. Considering the performance of org-element-at-point, it is probably worth doing. Initially, I wanted to avoid it because setting a single before-change-functions hook sounded cleaner than setting modification-hooks, insert-behind-hooks, and insert-in-front-hooks. Moreover, these text properties would be copied by default if one uses buffer-substring. Then, the hooks will also trigger later in the yanked text, which may cause all kinds of bugs. > `org-element-at-point' is local, `org-element-parse-buffer' is global. > They are not equivalent, but is it an issue? It was mostly an annoyance, because they returned different results on the same element. Specifically, they returned different :post-blank and :end properties, which does not sound right. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> Apparently my previous email was again refused by your mail server (I >> tried to add patch as attachment this time). > > Ah. This is annoying, for you and for me. > >> The patch is in >> https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef > > Thank you. > >>> I have finished a seemingly stable implementation of handling changes >>> inside drawer and block elements. For now, I did not bother with >>> 'modification-hooks and 'insert-in-font/behind-hooks, but simply used >>> before/after-change-functions. >>> >>> The basic idea is saving parsed org-elements before the modification >>> (with :begin and :end replaced by markers) and comparing them with the >>> versions of the same elements after the modification. >>> Any valid org element can be examined in such way by an arbitrary >>> function (see org-track-modification-elements) [1]. > > As you noticed, using Org Element is a no-go, unfortunately. Parsing an > element is a O(N) operation by the number of elements before it in > a section. In particular, it is not bounded, and not mitigated by > a cache. For large documents, it is going to be unbearably slow, too. > > I don't think the solution is to use combine-after-change-calls either, > because even a single call to `org-element-at-point' can be noticeable > in a very large section. Such low-level code should avoid using the > Element library altogether, except for the initial folding part, which > is interactive. > > If you use modification-hooks and al., you don't need to parse anything, > because you can store information as text properties. Therefore, once > the modification happens, you already know where you are (or, at least > where you were before the change). > > The ideas I suggested about sensitive parts of elements are worth > exploring, IMO. Do you have any issue with them? > >>> For (2), I have introduced org--property-drawer-modified-re to override >>> org-property-drawer-re in relevant *-change-function. This seems to work >>> for property drawers. However, I am not sure if similar problem may >>> happen in some border cases with ordinary drawers or blocks. > > I already specified what parts were "sensitive" in a previous message. > >>> 2. I have noticed that results of org-element-at-point and >>> org-element-parse-buffer are not always consistent. > > `org-element-at-point' is local, `org-element-parse-buffer' is global. > They are not equivalent, but is it an issue? > > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-18 16:52 ` Ihor Radchenko @ 2020-05-19 13:07 ` Nicolas Goaziou 2020-05-23 13:52 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-19 13:07 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: >> As you noticed, using Org Element is a no-go, unfortunately. Parsing an >> element is a O(N) operation by the number of elements before it in >> a section. In particular, it is not bounded, and not mitigated by >> a cache. For large documents, it is going to be unbearably slow, too. > > Ouch. I thought it is faster. > What do you mean by "not mitigated by a cache"? Parsing starts from the closest headline, every time. So, if Org parses the Nth element in the entry two times, it really parses 2N elements. With a cache, assuming the buffer wasn't modified, Org would parse N elements only. With a smarter cache, with fine grained cache invalidation, it could also reduce the number of subsequent parsed elements. > The reason I would like to utilise org-element parser to make tracking > modifications more robust. Using details of the syntax would make the > code fragile if any modifications are made to syntax in future. I don't think the code would be more fragile. Also, the syntax we're talking about is not going to be modified anytime soon. Moreover, if folding breaks, it is usually visible, so the bug will not be unnoticed. This code is going to be as low-level as it can be. > Debugging bugs in modification functions is not easy, according to my > experience. No, it's not. But this is not really related to whether you use Element or not. > One possible way to avoid performance issues during modification is > running parser in advance. For example, folding an element may > as well add information about the element to its text properties. > This will not degrade performance of folding since we are already > parsing the element during folding (at least, in > org-hide-drawer-toggle). We can use this information stored at fold time. But I'm not even sure we need it. > The problem with parsing an element during folding is that we cannot > easily detect changes like below without re-parsing. Of course we can. It is only necessary to focus on changes that would break the structure of the element. This does not entail a full parsing. > :PROPERTIES: <folded> > :CREATED: [2020-05-18 Mon] > :END: <- added line > :ID: test > :END: > > or even > > :PROPERTIES: > :CREATED: [2020-05-18 Mon] > :ID: test > :END: <- delete this line > > :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion> > test > :END: Please have a look at the "sensitive parts" I wrote about. This takes care of this kind of breakage. > The re-parsing can be done via regexp, as you suggested, but I don't > like this idea, because it will end up re-implementing > org-element-*-parser. You may have misunderstood my suggestion. See below. > Would it be acceptable to run org-element-*-parser > in after-change-functions? I'd rather not do that. This is unnecessary consing, and matching, etc. > If I understand correctly, it is not as easy. > Consider the following example: > > :PROPERTIES: > :CREATED: [2020-05-18 Mon] > <region-beginning> > :ID: example > :END: > > <... a lot of text, maybe containing other drawers ...> > > Nullam rutrum. > Pellentesque dapibus suscipit ligula. > <region-end> > Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. > > If the region gets deleted, the modification hooks from chars inside > drawer will be called as (hook-function <region-beginning> > <region-end>). So, there is still a need to find the drawer somehow to > mark it as about to be modified (modification hooks are ran before > actual modification). If we can stick with `after-change-functions' (or local equivalent), that's better. It is more predictable than `before-change-functions' and alike. If it is a deletion, here is the kind of checks we could do, depending on when they are performed. Before actual changes : 1. The deletion is happening within a folded drawer (unnecessary step in local functions). 2. The change deleted the sensitive line ":END:". 3. Conclusion : unfold. Or, after actual changes : 1. The deletion involves a drawer. 2. Text properties indicate that the beginning of the propertized part of the buffer start with org-drawer-regexp, but doesn't end with `org-property-end-re'. A "sensitive part" disappeared! 3. Conclusion : unfold This is far away from parsing. IMO, a few checks cover all cases. Let me know if you have questions about it. Also, note that the kind of change you describe will happen perhaps 0.01% of the time. Most change are about one character, or a single line, long. > The only difference between using modification hooks and > before-change-functions is that modification hooks will trigger less > frequently. Exactly. Much less frequently. But extra care is required, as you noted already. > Considering the performance of org-element-at-point, it is > probably worth doing. Initially, I wanted to avoid it because setting a > single before-change-functions hook sounded cleaner than setting > modification-hooks, insert-behind-hooks, and insert-in-front-hooks. Well, `before-change-fuctions' and `after-change-functions' are not clean at all: you modify an unrelated part of the buffer, but still call those to check if a drawer needs to be unfolded somewhere. And, more importantly, they are not meant to be used together, i.e., you cannot assume that a single call to `before-change-functions' always happens before calling `after-change-functions'. This can be tricky if you want to use the former to pass information to the latter. But I understand that they are easier to use than their local counterparts. If you stick with (before|after)-change-functions, the function being called needs to drop the ball very quickly if the modification is not about folding changes. Also, I very much suggest to stick to only `after-change-functions', if feasible (I think it is), per above. > Moreover, these text properties would be copied by default if one uses > buffer-substring. Then, the hooks will also trigger later in the yanked > text, which may cause all kinds of bugs. Indeed, that would be something to handle specifically. I.e., destructive modifications (i.e., those that unfold) could clear such properties. It is more work. I don't know if it is worth the trouble if we can get out quickly of `after-change-functions' for unrelated changes. > It was mostly an annoyance, because they returned different results on > the same element. Specifically, they returned different :post-blank and > :end properties, which does not sound right. OK. If you have a reproducible recipe, I can look into it and see what can be done. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-19 13:07 ` Nicolas Goaziou @ 2020-05-23 13:52 ` Ihor Radchenko 2020-05-23 13:53 ` Ihor Radchenko 2020-05-26 8:33 ` Nicolas Goaziou 0 siblings, 2 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-05-23 13:52 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Hello, [The patch itself will be provided in the following email] I have five updates from the previous version of the patch: 1. I implemented a simplified version of element parsing to detect changes in folded drawers or blocks. No computationally expensive calls of org-element-at-point or org-element-parse-buffer are needed now. 2. The patch is now compatible with master (commit 2e96dc639). I reverted the earlier change in folding drawers and blocks. Now, they are back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would achieve nothing when we use text properties. 3. 'invisible text property can now be nested. This is important, for example, when text inside drawers contains fontified links (which also use 'invisible text property to hide parts of the link). Now, the old 'invisible spec is recovered after unfolding. 4. Some outline-* function calls in org referred to outline-flag-region implementation, which is not in sync with org-flag-region in this patch. I have implemented their org-* versions and replaced the calls throughout .el files. Actually, some org-* versions were already implemented in org, but not used for some reason (or not mentioned in the manual). I have updated the relevant sections of manual. These changes might be relevant to org independently of this feature branch. 5. I have managed to get a working version of outline folding via text properties. However, that approach has a big downside - folding state cannot be different in indirect buffer when we use text properties. I have seen packages relying on this feature of org and I do not see any obvious way to achieve different folding state in indirect buffer while using text properties for outline folding. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the new implementation for tracking changes: > Of course we can. It is only necessary to focus on changes that would > break the structure of the element. This does not entail a full parsing. I have limited parsing to matching beginning and end of a drawer/block. The basic functions are org--get-element-region-at-point, org--get-next-element-region-at-point, and org--find-elements-in-region. They are simplified versions of org-element-* parsers and do not require parsing everything from the beginning of the section. For now, I keep everything in org.el, but those simplified parsers probably belong to org-element.el. > If we can stick with `after-change-functions' (or local equivalent), > that's better. It is more predictable than `before-change-functions' and > alike. For now, I still used before/after-change-functions combination. I see the following problems with using only after-change-functions: 1. They are not guaranteed to be called after every single change: From (elisp) Change Hooks: "... some complex primitives call ‘before-change-functions’ once before making changes, and then call ‘after-change-functions’ zero or more times" The consequence of it is a possibility that region passed to the after-change-functions is quite big (including all the singular changes, even if they are distant). This region may contain changed drawers as well and unchanged drawers and needs to be parsed to determine which drawers need to be re-folded. > And, more importantly, they are not meant to be used together, i.e., you > cannot assume that a single call to `before-change-functions' always > happens before calling `after-change-functions'. This can be tricky if > you want to use the former to pass information to the latter. The fact that before-change-functions can be called multiple times before after-change-functions, is trivially solved by using buffer-local changes register (see org--modified-elements). The register is populated by before-change-functions and cleared by after-change-functions. > Well, `before-change-fuctions' and `after-change-functions' are not > clean at all: you modify an unrelated part of the buffer, but still call > those to check if a drawer needs to be unfolded somewhere. 2. As you pointed, instead of global before-change-functions, we can use modification-hooks text property on sensitive parts of the drawers/blocks. This would work, but I am concerned about one annoying special case: ------------------------------------------------------------------------- :BLAH: <inserted outside any of the existing drawers> <some text> :DRAWER: <folded> Donec at pede. :END: ------------------------------------------------------------------------- In this example, the user would not be able to unfold the folder DRAWER because it will technically become a part of a new giant BLAH drawer. This may be especially annoying if <some text> is more than one screen long and there is no easy way to identify why unfolding does not work (with point at :DRAWER:). Because of this scenario, limiting before-change-functions to folded drawers is not sufficient. Any change in text may need to trigger unfolding. In the patch, I always register possible modifications in the blocks/drawers intersecting with the modified region + a drawer/block right next to the region. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the nested 'invisible text property implementation. The idea is to keep 'invisible property stack push and popping from it as we add/remove 'invisible text property. All the work is done in org-flag-region. This was originally intended for folding outlines via text properties. Since using text properties for folding outlines is not a good idea, nested text properties have much less use. As I mentioned, they do preserve link fontification, but I am not sure if it worth it for the overhead to org-flag-region. Keeping this here mostly in the case if someone has any ideas how it can be useful. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on replaced outline-* -> org-* function calls. I have implemented org-* versions of the following functions: - outline-hide-entry - outline-hide-subtree - outline-hide-sublevels - outline-show-heading - outline-show-branches The org-* versions trivially use org-flag-region instead of outline-flag-region. Replaced outline-* calls where org- versions were already available: - outline-show-entry - outline-show-all - outline-show-subtree I reflected the new (including already available) functions in the manual and removed some defalias from org-compat.el where they are not needed. ----------------------------------------------------------------------- ----------------------------------------------------------------------- Further work: 1. after-change-functions use org-hide-drawer/block-toggle to fold/unfold after modification. However, I just found that they call org-element-at-point, which slows down modifications in folded drawers/blocks. For example, realigning a long table inside folded drawer takes >1sec, while it is instant in the unfolded drawer. 2. org-toggle-custom-properties is terribly slow on large org documents, similarly to folded drawers on master. It should be relatively easy to use text properties there instead of overlays. 3. Multiple calls to before/after-change-functions is still a problem. I am looking into following ways to reduce this number: - reduce the number of elements registered as potentially modified + do not add duplicates to org--modified-elements + do not add unfolded elements to org--modified-elements + register after-change-function as post-command hook and remove it from global after-change-functions. This way, it will be called twice per command only. - determine common region containing org--modified-elements. if change is happening within that region, there is no need to parse drawers/blocks there again. P.S. >> It was mostly an annoyance, because they returned different results on >> the same element. Specifically, they returned different :post-blank and >> :end properties, which does not sound right. > > OK. If you have a reproducible recipe, I can look into it and see what > can be done. Recipe to have different (org-element-at-point) and (org-element-parse-buffer 'element) ------------------------------------------------------------------------- <point-min> :PROPERTIES: :CREATED: [2020-05-23 Sat 02:32] :END: <point-max> ------------------------------------------------------------------------- Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an >>> element is a O(N) operation by the number of elements before it in >>> a section. In particular, it is not bounded, and not mitigated by >>> a cache. For large documents, it is going to be unbearably slow, too. >> >> Ouch. I thought it is faster. >> What do you mean by "not mitigated by a cache"? > > Parsing starts from the closest headline, every time. So, if Org parses > the Nth element in the entry two times, it really parses 2N elements. > > With a cache, assuming the buffer wasn't modified, Org would parse > N elements only. With a smarter cache, with fine grained cache > invalidation, it could also reduce the number of subsequent parsed > elements. > >> The reason I would like to utilise org-element parser to make tracking >> modifications more robust. Using details of the syntax would make the >> code fragile if any modifications are made to syntax in future. > > I don't think the code would be more fragile. Also, the syntax we're > talking about is not going to be modified anytime soon. Moreover, if > folding breaks, it is usually visible, so the bug will not be unnoticed. > > This code is going to be as low-level as it can be. > >> Debugging bugs in modification functions is not easy, according to my >> experience. > > No, it's not. > > But this is not really related to whether you use Element or not. > >> One possible way to avoid performance issues during modification is >> running parser in advance. For example, folding an element may >> as well add information about the element to its text properties. >> This will not degrade performance of folding since we are already >> parsing the element during folding (at least, in >> org-hide-drawer-toggle). > > We can use this information stored at fold time. But I'm not even sure > we need it. > >> The problem with parsing an element during folding is that we cannot >> easily detect changes like below without re-parsing. > > Of course we can. It is only necessary to focus on changes that would > break the structure of the element. This does not entail a full parsing. > >> :PROPERTIES: <folded> >> :CREATED: [2020-05-18 Mon] >> :END: <- added line >> :ID: test >> :END: >> >> or even >> >> :PROPERTIES: >> :CREATED: [2020-05-18 Mon] >> :ID: test >> :END: <- delete this line >> >> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion> >> test >> :END: > > Please have a look at the "sensitive parts" I wrote about. This takes > care of this kind of breakage. > >> The re-parsing can be done via regexp, as you suggested, but I don't >> like this idea, because it will end up re-implementing >> org-element-*-parser. > > You may have misunderstood my suggestion. See below. > >> Would it be acceptable to run org-element-*-parser >> in after-change-functions? > > I'd rather not do that. This is unnecessary consing, and matching, etc. > >> If I understand correctly, it is not as easy. >> Consider the following example: >> >> :PROPERTIES: >> :CREATED: [2020-05-18 Mon] >> <region-beginning> >> :ID: example >> :END: >> >> <... a lot of text, maybe containing other drawers ...> >> >> Nullam rutrum. >> Pellentesque dapibus suscipit ligula. >> <region-end> >> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. >> >> If the region gets deleted, the modification hooks from chars inside >> drawer will be called as (hook-function <region-beginning> >> <region-end>). So, there is still a need to find the drawer somehow to >> mark it as about to be modified (modification hooks are ran before >> actual modification). > > If we can stick with `after-change-functions' (or local equivalent), > that's better. It is more predictable than `before-change-functions' and > alike. > > If it is a deletion, here is the kind of checks we could do, depending > on when they are performed. > > Before actual changes : > > 1. The deletion is happening within a folded drawer (unnecessary step > in local functions). > 2. The change deleted the sensitive line ":END:". > 3. Conclusion : unfold. > > Or, after actual changes : > > 1. The deletion involves a drawer. > 2. Text properties indicate that the beginning of the propertized part > of the buffer start with org-drawer-regexp, but doesn't end with > `org-property-end-re'. A "sensitive part" disappeared! > 3. Conclusion : unfold > > This is far away from parsing. IMO, a few checks cover all cases. Let me > know if you have questions about it. > > Also, note that the kind of change you describe will happen perhaps > 0.01% of the time. Most change are about one character, or a single > line, long. > >> The only difference between using modification hooks and >> before-change-functions is that modification hooks will trigger less >> frequently. > > Exactly. Much less frequently. But extra care is required, as you noted > already. > >> Considering the performance of org-element-at-point, it is >> probably worth doing. Initially, I wanted to avoid it because setting a >> single before-change-functions hook sounded cleaner than setting >> modification-hooks, insert-behind-hooks, and insert-in-front-hooks. > > Well, `before-change-fuctions' and `after-change-functions' are not > clean at all: you modify an unrelated part of the buffer, but still call > those to check if a drawer needs to be unfolded somewhere. > > And, more importantly, they are not meant to be used together, i.e., you > cannot assume that a single call to `before-change-functions' always > happens before calling `after-change-functions'. This can be tricky if > you want to use the former to pass information to the latter. > > But I understand that they are easier to use than their local > counterparts. If you stick with (before|after)-change-functions, the > function being called needs to drop the ball very quickly if the > modification is not about folding changes. Also, I very much suggest to > stick to only `after-change-functions', if feasible (I think it is), per > above. > >> Moreover, these text properties would be copied by default if one uses >> buffer-substring. Then, the hooks will also trigger later in the yanked >> text, which may cause all kinds of bugs. > > Indeed, that would be something to handle specifically. I.e., > destructive modifications (i.e., those that unfold) could clear such > properties. > > It is more work. I don't know if it is worth the trouble if we can get > out quickly of `after-change-functions' for unrelated changes. > >> It was mostly an annoyance, because they returned different results on >> the same element. Specifically, they returned different :post-blank and >> :end properties, which does not sound right. > > OK. If you have a reproducible recipe, I can look into it and see what > can be done. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-23 13:52 ` Ihor Radchenko @ 2020-05-23 13:53 ` Ihor Radchenko 2020-05-23 15:26 ` Ihor Radchenko 2020-05-26 8:33 ` Nicolas Goaziou 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-05-23 13:53 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 23 bytes --] The patch is attached [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: featuredrawertextprop-20200523.patch --] [-- Type: text/x-diff, Size: 45706 bytes --] diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el index 9f8677871..ab470ea9b 100644 --- a/contrib/lisp/org-notify.el +++ b/contrib/lisp/org-notify.el @@ -246,7 +246,7 @@ seconds. The default value for SECS is 20." (switch-to-buffer (find-file-noselect file)) (org-with-wide-buffer (goto-char begin) - (outline-show-entry)) + (org-show-entry)) (goto-char begin) (search-forward "DEADLINE: <") (search-forward ":") diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el index bfc4d6c3e..2312b235c 100644 --- a/contrib/lisp/org-velocity.el +++ b/contrib/lisp/org-velocity.el @@ -325,7 +325,7 @@ use it." (save-excursion (when narrow (org-narrow-to-subtree)) - (outline-show-all))) + (org-show-all))) (defun org-velocity-edit-entry/inline (heading) "Edit entry at HEADING in the original buffer." diff --git a/doc/org-manual.org b/doc/org-manual.org index 96b160175..2ebe94538 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and Switch back to the startup visibility of the buffer (see [[*Initial visibility]]). -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) :: +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) :: #+cindex: show all, command #+kindex: C-u C-u C-u TAB - #+findex: outline-show-all + #+findex: org-show-all Show all, including drawers. - {{{kbd(C-c C-r)}}} (~org-reveal~) :: @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and headings. With a double prefix argument, also show the entire subtree of the parent. -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) :: +- {{{kbd(C-c C-k)}}} (~org-show-branches~) :: #+cindex: show branches, command #+kindex: C-c C-k - #+findex: outline-show-branches + #+findex: org-show-branches Expose all the headings of the subtree, but not their bodies. -- {{{kbd(C-c TAB)}}} (~outline-show-children~) :: +- {{{kbd(C-c TAB)}}} (~org-show-children~) :: #+cindex: show children, command #+kindex: C-c TAB - #+findex: outline-show-children + #+findex: org-show-children Expose all direct children of the subtree. With a numeric prefix argument {{{var(N)}}}, expose all children down to level {{{var(N)}}}. @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way: command (see [[*Visibility Cycling]]). You can force cycling archived subtrees with {{{kbd(C-TAB)}}}, or by setting the option ~org-cycle-open-archived-trees~. Also normal outline commands, like - ~outline-show-all~, open archived subtrees. + ~org-show-all~, open archived subtrees. - #+vindex: org-sparse-tree-open-archived-trees diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index ab13f926c..ad9244940 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'." (t " %-12:c%?-12t% s"))) (start 0) varform vars var e c f opt) - (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)" + (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)" s start) (setq var (or (cdr (assoc (match-string 4 s) '(("c" . category) ("t" . time) ("l" . level) ("s" . extra) @@ -9138,20 +9138,20 @@ if it was hidden in the outline." ((and (called-interactively-p 'any) (= more 1)) (message "Remote: show with default settings")) ((= more 2) - (outline-show-entry) + (org-show-entry) (org-show-children) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'children)) (message "Remote: CHILDREN")) ((= more 3) - (outline-show-subtree) + (org-show-subtree) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'subtree)) (message "Remote: SUBTREE")) ((> more 3) - (outline-show-subtree) + (org-show-subtree) (message "Remote: SUBTREE AND ALL DRAWERS"))) (select-window win))) diff --git a/lisp/org-archive.el b/lisp/org-archive.el index d3e12d17b..d864dad8a 100644 --- a/lisp/org-archive.el +++ b/lisp/org-archive.el @@ -330,7 +330,7 @@ direct children of this heading." (insert (if datetree-date "" "\n") heading "\n") (end-of-line 0)) ;; Make the subtree visible - (outline-show-subtree) + (org-show-subtree) (if org-archive-reversed-order (progn (org-back-to-heading t) diff --git a/lisp/org-colview.el b/lisp/org-colview.el index e50a4d7c8..e656df555 100644 --- a/lisp/org-colview.el +++ b/lisp/org-colview.el @@ -699,7 +699,7 @@ FUN is a function called with no argument." (move-beginning-of-line 2) (org-at-heading-p t))))) (unwind-protect (funcall fun) - (when hide-body (outline-hide-entry))))) + (when hide-body (org-hide-entry))))) (defun org-columns-previous-allowed-value () "Switch to the previous allowed value for this column." diff --git a/lisp/org-compat.el b/lisp/org-compat.el index 635a38dcd..8fe271896 100644 --- a/lisp/org-compat.el +++ b/lisp/org-compat.el @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer." ;;; Emacs < 25.1 compatibility (when (< emacs-major-version 25) - (defalias 'outline-hide-entry 'hide-entry) - (defalias 'outline-hide-sublevels 'hide-sublevels) - (defalias 'outline-hide-subtree 'hide-subtree) (defalias 'outline-show-branches 'show-branches) (defalias 'outline-show-children 'show-children) - (defalias 'outline-show-entry 'show-entry) (defalias 'outline-show-subtree 'show-subtree) (defalias 'xref-find-definitions 'find-tag) (defalias 'format-message 'format) diff --git a/lisp/org-element.el b/lisp/org-element.el index ac41b7650..2d5c8d771 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -4320,7 +4320,7 @@ element or object. Meaningful values are `first-section', TYPE is the type of the current element or object. If PARENT? is non-nil, assume the next element or object will be -located inside the current one. " +located inside the current one." (if parent? (pcase type (`headline 'section) diff --git a/lisp/org-keys.el b/lisp/org-keys.el index c006e9c12..deb5d7b90 100644 --- a/lisp/org-keys.el +++ b/lisp/org-keys.el @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names." #'org-next-visible-heading) (define-key org-mode-map [remap outline-previous-visible-heading] #'org-previous-visible-heading) -(define-key org-mode-map [remap show-children] #'org-show-children) +(define-key org-mode-map [remap outline-show-children] #'org-show-children) ;;;; Make `C-c C-x' a prefix key (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap)) diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..fa0a658f0 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." \f -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org-remove-text-properties (start end properties &optional object) + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. +Do not remove invisible text properties specified by 'outline, +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this +is needed to keep outlines, drawers, and blocks hidden unless they are +toggled by user. +Note: The below may be too specific and create troubles if more +invisibility specs are added to org in future" + (when (plist-member properties 'invisible) + (let ((pos start) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer + 'outline)) + (remove-text-properties pos next '(invisible nil) object)) + (setq pos next)))) + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) + (remove-text-properties start end properties-stripped object))) + +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + (pcase spec + ('outline + (remove-overlays from to 'invisible spec) + ;; Use `front-advance' since text right before to the beginning of + ;; the overlay belongs to the visible line than to the contents. + (when flag + (let ((o (make-overlay from to nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) + (_ + ;; Use text properties instead of overlays for speed. + ;; Overlays are too slow (Emacs Bug#35453). + (with-silent-modifications + ;; keep a backup stack of old text properties + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((old-spec (get-text-property (point) 'invisible)) + (end (next-single-property-change (point) 'invisible nil to))) + (when old-spec + (alter-text-property (point) end 'org-property-stack-invisible + (lambda (stack) + (if (or (eq old-spec (car stack)) + (eq spec old-spec) + (eq old-spec 'outline)) + stack + (cons old-spec stack))))) + (goto-char end)))) + + ;; cleanup everything + (remove-text-properties from to '(invisible nil)) + + ;; Recover properties from the backup stack + (unless flag + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((stack (get-text-property (point) 'org-property-stack-invisible)) + (end (next-single-property-change (point) 'org-property-stack-invisible nil to))) + (if (not stack) + (remove-text-properties (point) end '(org-property-stack-invisible nil)) + (put-text-property (point) end 'invisible (car stack)) + (alter-text-property (point) end 'org-property-stack-invisible + (lambda (stack) + (cdr stack)))) + (goto-char end))))) + + (when flag + (put-text-property from to 'rear-non-sticky nil) + (put-text-property from to 'front-sticky t) + (put-text-property from to 'invisible spec)))))) \f ;;; Regexp matching diff --git a/lisp/org-src.el b/lisp/org-src.el index c9eef744e..e89a1c580 100644 --- a/lisp/org-src.el +++ b/lisp/org-src.el @@ -523,8 +523,8 @@ Leave point in edit buffer." (org-src-switch-to-buffer buffer 'edit) ;; Insert contents. (insert contents) - (remove-text-properties (point-min) (point-max) - '(display nil invisible nil intangible nil)) + (org-remove-text-properties (point-min) (point-max) + '(display nil invisible nil intangible nil)) (unless preserve-ind (org-do-remove-indentation)) (set-buffer-modified-p nil) (setq buffer-file-name nil) diff --git a/lisp/org-table.el b/lisp/org-table.el index 6462b99c4..75801161b 100644 --- a/lisp/org-table.el +++ b/lisp/org-table.el @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'." (arg (let ((b (save-excursion (skip-chars-backward "^|") (point))) (e (save-excursion (skip-chars-forward "^|\r\n") (point)))) - (remove-text-properties b e '(invisible t intangible t)) + (org-remove-text-properties b e '(invisible t intangible t)) (if (and (boundp 'font-lock-mode) font-lock-mode) (font-lock-fontify-block)))) (t @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'." (setq word-wrap t) (goto-char (setq p (point-max))) (insert (org-trim field)) - (remove-text-properties p (point-max) '(invisible t intangible t)) + (org-remove-text-properties p (point-max) '(invisible t intangible t)) (goto-char p) (setq-local org-finish-function 'org-table-finish-edit-field) (setq-local org-window-configuration cw) diff --git a/lisp/org.el b/lisp/org.el index e577dc661..360974135 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") (defvar ffap-url-regexp) (defvar org-element-paragraph-separate) +(defvar org-element-all-objects) +(defvar org-element-all-elements) +(defvar org-element-greater-elements) (defvar org-indent-indentation-per-level) (defvar org-radio-target-regexp) (defvar org-target-link-regexp) @@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.") ;;;; Define the Org mode +;;; Handling buffer modifications + (defun org-before-change-function (_beg _end) "Every change indicates that a table might need an update." (setq org-table-may-need-update t)) + +(defvar-local org--modified-elements nil + "List of elements, marked as recently modified. +There is no guarantee that the elements in this list are fully parsed. +Only the element type, :begin and :end properties of the elements are +guaranteed to be available. The :begin and :end element properties +contain markers instead of positions.") + +(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command) + "List of commands triggerring element modifications unconditionally.") + +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$") + (property-drawer . ,org-property-start-re) + (drawer . ,org-drawer-regexp) + (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$") + (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$")) + "Alist of regexps matching beginning of elements. +Group 1 in the regexps (if any) contains the element type.") + +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$") + (property-drawer . ,org-property-end-re) + (drawer . ,org-property-end-re) + (quote-block . "^[ \t]*#\\+end_quote[ \t]*$") + (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$")) + "Alist of regexps matching end of elements. +Group 1 in the regexps (if any) contains the element type or END.") + +(defvar org-track-element-modifications + `((property-drawer . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (drawer . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (center-block . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (quote-block . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (special-block . (:after-change-function + org--drawer-or-block-unfold-maybe))) + "Alist of elements to be tracked for modifications. +The modification is only triggered according to :sensitive-re-list and +:sensitive-command-list (see below). +Each element of the alist is a cons of an element symbol and plist +defining how and when the modifications are handled. +In case of recursive elements/duplicates, the first element from the +list is considered. +The plist can have the following properties: +- :element-beginning-re :: regex matching beginning of the element + (default) :: (alist-get element org--element-beginning-re-alist) +- :element-end-re :: regex matching end of the element + (default) :: (alist-get element org--element-end-re-alist) +- :after-change-function :: function called after the modification +The function must accept a single argument - element from +`org--modified-elements'.") + +(defun org--get-element-region-at-point (types) + "Return TYPES element at point or nil. +If TYPES is a list, return first element at point from the list. The +returned value is partially parsed element only containing :begin and +:end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (begin-limit (save-excursion (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok)) + (point))) + (end-limit (or (save-excursion (outline-next-heading)) + (point-max))) + (point (point)) + begin end closest-begin) + (when (and begin-re end-re) + (save-excursion + (end-of-line) + (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point))) + (when (re-search-forward end-re end-limit 'noerror) (setq end (point))) + (setq closest-begin begin) + ;; slurp unmatched begin-re + (when (and begin end) + (goto-char begin) + (while (and (re-search-backward begin-re begin-limit 'noerror) + (= end (save-excursion (re-search-forward end-re end-limit 'noerror)))) + (setq begin (point))) + (when (and (>= point begin) (<= point end)) + (throw 'exit + (list type + (list + :begin begin + :end end))))))))))) + +(defun org--get-next-element-region-at-point (types &optional limit previous) + "Return TYPES element after point or nil. +If TYPES is a list, return first element after point from the list. +If PREVIOUS is non-nil, return first TYPES element before point. +Limit search by LIMIT or previous/next heading. +The returned value is partially parsed element only containing :begin +and :end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let* ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (limit (or limit (if previous + (save-excursion + (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok) + (point))) + (or (save-excursion (outline-next-heading)) + (point-max))))) + begin end) + (when (and begin-re end-re) + (save-excursion + (if previous + (when (re-search-backward begin-re limit 'noerror) + (when-let ((el (org--get-element-region-at-point type))) + (setq begin (org-element-property :begin el)) + (setq end (org-element-property :end el)))) + (when (re-search-forward begin-re limit 'noerror) + (when-let ((el (org--get-element-region-at-point type))) + (setq begin (org-element-property :begin el)) + (setq end (org-element-property :end el)))))) + (when (and begin end) + (throw 'exit + (list type + (list + :begin begin + :end end))))))))) + +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours) + "Find all elements from ELEMENTS in region BEG . END. +All the listed elements must be resolvable by +`org--get-element-region-at-point'. +Include elements if they are partially inside region when +INCLUDE-PARTIAL is non-nil. +Include preceding/subsequent neighbouring elements when no partial +element is found at the beginning/end of the region and +INCLUDE-NEIGHBOURS is non-nil." + (when include-partial + (org-with-point-at beg + (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements)))) + (if new-beg + (setq beg new-beg) + (when (and include-neighbours + (setq new-beg (org-element-property :begin + (org--get-next-element-region-at-point elements + (point-min) + 'previous)))) + (setq beg new-beg)))) + (when (memq 'headline elements) + (when-let ((new-beg (save-excursion + (org-with-limited-levels (outline-previous-heading))))) + (setq beg new-beg)))) + (org-with-point-at end + (let ((new-end (org-element-property :end (org--get-element-region-at-point elements)))) + (if new-end + (setq end new-end) + (when (and include-neighbours + (setq new-end (org-element-property :end + (org--get-next-element-region-at-point elements (point-max))))) + (setq end new-end)))) + (when (memq 'headline elements) + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) + (setq end (1- new-end)))))) + (save-excursion + (save-restriction + (narrow-to-region beg end) + (goto-char (point-min)) + (let (result el) + (while (setq el (org--get-next-element-region-at-point elements end)) + (push el result) + (goto-char (org-element-property :end el))) + result)))) + +(defun org--drawer-or-block-unfold-maybe (el) + "Update visibility of modified folded drawer/block EL. +If text was added to hidden drawer/block, make sure that the text is +also hidden, unless the change was done by a command listed in +`org-track-element-modification-default-sensitive-commands'. If the +modification destroyed the drawer/block, reveal the hidden text in +former drawer/block. If the modification shrinked/expanded the +drawer/block beyond the hidden text, reveal the affected +drawers/blocks as well. +Examples: +---------------------------------------------- +---------------------------------------------- +Case #1 (the element content is hidden): +---------------------------------------------- +:PROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:ROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +Text is revealed, because we have drawer in place of property-drawer +---------------------------------------------- +---------------------------------------------- +Case #2 (the element content is hidden): +---------------------------------------------- +:ROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:OPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +The text remains hidden since it is still a drawer. +---------------------------------------------- +---------------------------------------------- +Case #3: (the element content is hidden): +---------------------------------------------- +:FOO: +bar +tmp +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:FOO: +bar +:END: +tmp +:END: +---------------------------------------------- +The text is revealed because the drawer contents shrank. +---------------------------------------------- +---------------------------------------------- +Case #4: (the element content is hidden in both the drawers): +---------------------------------------------- +:FOO: +bar +tmp +:END: +:BAR: +jjd +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:FOO: +bar +tmp +:BAR: +jjd +:END: +---------------------------------------------- +The text is revealed in both the drawers because the drawers are merged +into a new drawer. +---------------------------------------------- +---------------------------------------------- +Case #5: (the element content is hidden) +---------------------------------------------- +:test: +Vivamus id enim. +:end: +---------------------------------------------- +is changed to +---------------------------------------------- +:drawer: +:test: +Vivamus id enim. +:end: +---------------------------------------------- +The text is revealed in the drawer because the drawer expended. +---------------------------------------------- +---------------------------------------------- +Case #6: (the element content is hidden): +---------------------------------------------- +:test: +Vivamus id enim. +:end: +---------------------------------------------- +is changed to +---------------------------------------------- +:test: +Vivamus id enim. +:end: +Nam a sapien. +:end: +---------------------------------------------- +The text remains hidden because drawer contents is always before the +first :end:." + (save-match-data + (save-excursion + (save-restriction + (goto-char (org-element-property :begin el)) + (let* ((newel (org--get-element-region-at-point + (mapcar (lambda (el) + (when (string-match-p (regexp-opt '("block" "drawer")) + (symbol-name (car el))) + (car el))) + org-track-element-modifications))) + (spec (if (string-match-p "block" (symbol-name (org-element-type el))) + 'org-hide-block + (if (string-match-p "drawer" (symbol-name (org-element-type el))) + 'org-hide-drawer + t))) + (toggle-func (if (eq spec 'org-hide-drawer) + #'org-hide-drawer-toggle + (if (eq spec 'org-hide-block) + #'org-hide-block-toggle + #'ignore)))) ; this should not happen + (if (and (equal (org-element-type el) (org-element-type newel)) + (equal (marker-position (org-element-property :begin el)) + (org-element-property :begin newel)) + (equal (marker-position (org-element-property :end el)) + (org-element-property :end newel))) + (when (text-property-any (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + 'invisible spec) + (goto-char (org-element-property :begin newel)) + (if (memq this-command org-track-element-modification-default-sensitive-commands) + ;; reveal if change was made by typing + (funcall toggle-func 'off) + ;; re-hide the inserted text + ;; FIXME: opening the drawer before hiding should not be needed here + (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis + (funcall toggle-func 'hide))) + ;; The element was destroyed. Reveal everything. + (org-flag-region (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + nil spec) + (when newel + (org-flag-region (org-element-property :begin newel) + (org-element-property :end newel) + nil spec)))))))) + +(defun org--before-element-change-function (beg end) + "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END." + (save-match-data + (save-excursion + (save-restriction + (widen) + (dolist (el (org--find-elements-in-region beg + end + (mapcar #'car org-track-element-modifications) + 'include-partial + 'include-neighbours)) + (let* ((beg-marker (copy-marker (org-element-property :begin el) 't)) + (end-marker (copy-marker (org-element-property :end el) 't))) + (when (and (marker-position beg-marker) (marker-position end-marker)) + (org-element-put-property el :begin beg-marker) + (org-element-put-property el :end end-marker) + (add-to-list 'org--modified-elements el)))))))) + +;; FIXME: this function may be called many times during routine modifications +;; The normal way to avoid this is `combine-after-change-calls' - not +;; the case in most org primitives. +(defun org--after-element-change-function (&rest _) + "Handle changed elements from `org--modified-elements'." + (dolist (el org--modified-elements) + (save-match-data + (save-excursion + (save-restriction + (widen) + (when-let* ((type (org-element-type el)) + (change-func (plist-get (alist-get type org-track-element-modifications) + :after-change-function))) + (with-demoted-errors + (funcall (symbol-function change-func) el))))))) + (setq org--modified-elements nil)) + (defvar org-mode-map) (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. @@ -4818,6 +5194,9 @@ The following commands are available: ;; Activate before-change-function (setq-local org-table-may-need-update t) (add-hook 'before-change-functions 'org-before-change-function nil 'local) + (add-hook 'before-change-functions 'org--before-element-change-function nil 'local) + ;; Activate after-change-function + (add-hook 'after-change-functions 'org--after-element-change-function nil 'local) ;; Check for running clock before killing a buffer (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) ;; Initialize macros templates. @@ -4869,6 +5248,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5050,8 +5433,8 @@ stacked delimiters is N. Escaping delimiters is not possible." (when verbatim? (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 2) (match-end 2) - '(display t invisible t intangible t))) + (org-remove-text-properties (match-beginning 2) (match-end 2) + '(display t invisible t intangible t))) (add-text-properties (match-beginning 2) (match-end 2) '(font-lock-multiline t org-emphasis t)) (when (and org-hide-emphasis-markers @@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links." (if (not (eq 'bracket style)) (add-text-properties start end properties) ;; Handle invisible parts in bracket links. - (remove-text-properties start end '(invisible nil)) + (org-remove-text-properties start end '(invisible nil)) (let ((hidden (append `(invisible ,(or (org-link-get-parameter type :display) @@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links." (defun org-activate-code (limit) (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) t)) (defcustom org-src-fontify-natively t @@ -5258,8 +5641,8 @@ by a #." (setq block-end (match-beginning 0)) ; includes the final newline. (when quoting (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline) - (remove-text-properties beg end-of-endline - '(display t invisible t intangible t))) + (org-remove-text-properties beg end-of-endline + '(display t invisible t intangible t))) (add-text-properties beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) (org-remove-flyspell-overlays-in beg bol-after-beginline) @@ -5313,8 +5696,8 @@ by a #." '(font-lock-fontified t face org-document-info)))) ((string-prefix-p "+caption" dc1) (org-remove-flyspell-overlays-in (match-end 2) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) ;; Handle short captions. (save-excursion (beginning-of-line) @@ -5336,8 +5719,8 @@ by a #." '(font-lock-fontified t face font-lock-comment-face))) (t ;; just any other in-buffer setting, but not indented (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) (add-text-properties beg (match-end 0) '(font-lock-fontified t face org-meta-line)) t)))))) @@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) - (remove-text-properties beg end - '(mouse-face t keymap t org-linked-text t - invisible t intangible t - org-emphasis t)) + (org-remove-text-properties beg end + '(mouse-face t keymap t org-linked-text t + invisible t + intangible t + org-emphasis t)) (org-remove-font-lock-display-properties beg end))) (defconst org-script-display '(((raise -0.3) (height 0.7)) @@ -5970,6 +6354,29 @@ open and agenda-wise Org files." ;;;; Headlines visibility +(defun org-hide-entry () + "Hide the body directly following this heading." + (interactive) + (save-excursion + (outline-back-to-heading) + (outline-end-of-heading) + (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline))) + +(defun org-hide-subtree () + "Hide everything after this heading at deeper levels." + (interactive) + (org-flag-subtree t)) + +(defun org-hide-sublevels (levels) + "Hide everything but the top LEVELS levels of headers, in whole buffer. +This also unhides the top heading-less body, if any. + +Interactively, the prefix argument supplies the value of LEVELS. +When invoked without a prefix argument, LEVELS defaults to the level +of the current heading, or to 1 if the current line is not a heading." + (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region)) + (org-hide-sublevels levels))) + (defun org-show-entry () "Show the body directly following this heading. Show the heading too, if it is currently invisible." @@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible." 'outline) (org-cycle-hide-property-drawers 'children)))) +(defun org-show-heading () + "Show the current heading and move to its end." + (org-flag-region (- (point) + (if (bobp) 0 + (if (and outline-blank-line + (eq (char-before (1- (point))) ?\n)) + 2 1))) + (progn (outline-end-of-heading) (point)) + nil)) + (defun org-show-children (&optional level) "Show all direct subheadings of this heading. Prefix arg LEVEL is how many levels below the current level @@ -6031,6 +6448,11 @@ heading to appear." (org-flag-region (point) (save-excursion (org-end-of-subtree t t)) nil 'outline)) +(defun org-show-branches () + "Show all subheadings of this heading, but not their bodies." + (interactive) + (org-show-children 1000)) + ;;;; Blocks and drawers visibility (defun org--hide-wrapper-toggle (element category force no-error) @@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful." (unless (let ((eol (line-end-position))) (and (> eol start) (/= eol end))) (let* ((spec (cond ((eq category 'block) 'org-hide-block) - ((eq type 'property-drawer) 'outline) - (t 'org-hide-drawer))) + ((eq category 'drawer) 'org-hide-drawer) + (t 'outline))) (flag (cond ((eq force 'off) nil) (force t) @@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of (when (org-at-property-drawer-p) (let* ((case-fold-search t) (end (re-search-forward org-property-end-re))) - ;; Property drawers use `outline' invisibility spec - ;; so they can be swallowed once we hide the - ;; outline. - (org-flag-region start end t 'outline))))))))))) + (org-flag-region start end t 'org-hide-drawer))))))))))) ;;;; Visibility cycling @@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level." (org-narrow-to-subtree) (org-content)))) ((or "all" "showall") - (outline-show-subtree)) + (org-show-subtree)) (_ nil))) (org-end-of-subtree))))))) @@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'." (while (re-search-forward re nil t) (when (and (not (org-invisible-p)) (org-invisible-p (line-end-position))) - (outline-hide-entry)))) + (org-hide-entry)))) (org-cycle-hide-property-drawers 'all) (org-cycle-show-empty-lines 'overview))))) @@ -6683,8 +7102,13 @@ information." ;; expose it. (dolist (o (overlays-at (point))) (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) + '(outline)) (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard." (skip-chars-forward " \t\n\r") (setq beg (point)) (when (and (org-invisible-p) visp) - (save-excursion (outline-show-heading))) + (save-excursion (org-show-heading))) ;; Shift if necessary. (unless (= shift 0) (save-restriction @@ -8103,7 +8527,7 @@ function is being called interactively." (point)) what "children") (goto-char start) - (outline-show-subtree) + (org-show-subtree) (outline-next-heading)) (t ;; we will sort the top-level entries in this file @@ -13150,7 +13574,7 @@ drawer is immediately hidden." (inhibit-read-only t)) (unless (bobp) (insert "\n")) (insert ":PROPERTIES:\n:END:") - (org-flag-region (line-end-position 0) (point) t 'outline) + (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer) (when (or (eobp) (= begin (point-min))) (insert "\n")) (org-indent-region begin (point)))))) @@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer." (defun org-show-branches-buffer () "Show all branches in the buffer." (org-flag-above-first-heading) - (outline-hide-sublevels 1) + (org-hide-sublevels 1) (unless (eobp) - (outline-show-branches) + (org-show-branches) (while (outline-get-next-sibling) - (outline-show-branches))) + (org-show-branches))) (goto-char (point-min))) (defun org-kill-note-or-show-branches () @@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer." (t (let ((beg (progn (org-back-to-heading) (point))) (end (save-excursion (org-end-of-subtree t t) (point)))) - (outline-hide-subtree) - (outline-show-branches) + (org-hide-subtree) + (org-show-branches) (org-hide-archived-subtrees beg end))))) (defun org-delete-indentation (&optional arg) @@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'. ARG is the level to hide." (if (org-before-first-heading-p) (progn (org-flag-above-first-heading) - (outline-hide-sublevels (or arg 1)) + (org-hide-sublevels (or arg 1)) (goto-char (point-min))) - (outline-hide-subtree) + (org-hide-subtree) (org-show-children arg)))) (defun org-ctrl-c-star () @@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) \f + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + (setq spec (get-text-property pos 'invisible)) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (org-flag-region (car region) (cdr region) nil spec))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (org-flag-region (overlay-start ov) (overlay-end ov) t spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + +\f + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode [-- Attachment #3: Type: text/plain, Size: 17187 bytes --] Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > > [The patch itself will be provided in the following email] > > I have five updates from the previous version of the patch: > > 1. I implemented a simplified version of element parsing to detect > changes in folded drawers or blocks. No computationally expensive calls > of org-element-at-point or org-element-parse-buffer are needed now. > > 2. The patch is now compatible with master (commit 2e96dc639). I > reverted the earlier change in folding drawers and blocks. Now, they are > back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would > achieve nothing when we use text properties. > > 3. 'invisible text property can now be nested. This is important, for > example, when text inside drawers contains fontified links (which also > use 'invisible text property to hide parts of the link). Now, the old > 'invisible spec is recovered after unfolding. > > 4. Some outline-* function calls in org referred to outline-flag-region > implementation, which is not in sync with org-flag-region in this patch. > I have implemented their org-* versions and replaced the calls > throughout .el files. Actually, some org-* versions were already > implemented in org, but not used for some reason (or not mentioned in > the manual). I have updated the relevant sections of manual. These > changes might be relevant to org independently of this feature branch. > > 5. I have managed to get a working version of outline folding via text > properties. However, that approach has a big downside - folding state > cannot be different in indirect buffer when we use text properties. I > have seen packages relying on this feature of org and I do not see any > obvious way to achieve different folding state in indirect buffer while > using text properties for outline folding. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > >> Of course we can. It is only necessary to focus on changes that would >> break the structure of the element. This does not entail a full parsing. > > I have limited parsing to matching beginning and end of a drawer/block. > The basic functions are org--get-element-region-at-point, > org--get-next-element-region-at-point, and org--find-elements-in-region. > They are simplified versions of org-element-* parsers and do not require > parsing everything from the beginning of the section. > > For now, I keep everything in org.el, but those simplified parsers > probably belong to org-element.el. > >> If we can stick with `after-change-functions' (or local equivalent), >> that's better. It is more predictable than `before-change-functions' and >> alike. > > For now, I still used before/after-change-functions combination. > I see the following problems with using only after-change-functions: > > 1. They are not guaranteed to be called after every single change: > > From (elisp) Change Hooks: > "... some complex primitives call ‘before-change-functions’ once before > making changes, and then call ‘after-change-functions’ zero or more > times" > > The consequence of it is a possibility that region passed to the > after-change-functions is quite big (including all the singular changes, > even if they are distant). This region may contain changed drawers as > well and unchanged drawers and needs to be parsed to determine which > drawers need to be re-folded. > >> And, more importantly, they are not meant to be used together, i.e., you >> cannot assume that a single call to `before-change-functions' always >> happens before calling `after-change-functions'. This can be tricky if >> you want to use the former to pass information to the latter. > > The fact that before-change-functions can be called multiple times > before after-change-functions, is trivially solved by using buffer-local > changes register (see org--modified-elements). The register is populated > by before-change-functions and cleared by after-change-functions. > >> Well, `before-change-fuctions' and `after-change-functions' are not >> clean at all: you modify an unrelated part of the buffer, but still call >> those to check if a drawer needs to be unfolded somewhere. > > 2. As you pointed, instead of global before-change-functions, we can use > modification-hooks text property on sensitive parts of the > drawers/blocks. This would work, but I am concerned about one annoying > special case: > > ------------------------------------------------------------------------- > :BLAH: <inserted outside any of the existing drawers> > > <some text> > > :DRAWER: <folded> > Donec at pede. > :END: > ------------------------------------------------------------------------- > In this example, the user would not be able to unfold the folder DRAWER > because it will technically become a part of a new giant BLAH drawer. > This may be especially annoying if <some text> is more than one screen > long and there is no easy way to identify why unfolding does not work > (with point at :DRAWER:). > > Because of this scenario, limiting before-change-functions to folded > drawers is not sufficient. Any change in text may need to trigger > unfolding. > > In the patch, I always register possible modifications in the > blocks/drawers intersecting with the modified region + a drawer/block > right next to the region. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the nested 'invisible text property implementation. > > The idea is to keep 'invisible property stack push and popping from it > as we add/remove 'invisible text property. All the work is done in > org-flag-region. > > This was originally intended for folding outlines via text properties. > Since using text properties for folding outlines is not a good idea, > nested text properties have much less use. As I mentioned, they do > preserve link fontification, but I am not sure if it worth it for the > overhead to org-flag-region. Keeping this here mostly in the case if > someone has any ideas how it can be useful. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on replaced outline-* -> org-* function calls. > > I have implemented org-* versions of the following functions: > > - outline-hide-entry > - outline-hide-subtree > - outline-hide-sublevels > - outline-show-heading > - outline-show-branches > > The org-* versions trivially use org-flag-region instead of > outline-flag-region. > > Replaced outline-* calls where org- versions were already available: > > - outline-show-entry > - outline-show-all > - outline-show-subtree > > I reflected the new (including already available) functions in the > manual and removed some defalias from org-compat.el where they are not > needed. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > 1. after-change-functions use org-hide-drawer/block-toggle to > fold/unfold after modification. However, I just found that they call > org-element-at-point, which slows down modifications in folded > drawers/blocks. For example, realigning a long table inside folded > drawer takes >1sec, while it is instant in the unfolded drawer. > > 2. org-toggle-custom-properties is terribly slow on large org documents, > similarly to folded drawers on master. It should be relatively easy to > use text properties there instead of overlays. > > 3. Multiple calls to before/after-change-functions is still a problem. I > am looking into following ways to reduce this number: > - reduce the number of elements registered as potentially modified > + do not add duplicates to org--modified-elements > + do not add unfolded elements to org--modified-elements > + register after-change-function as post-command hook and remove it > from global after-change-functions. This way, it will be called > twice per command only. > - determine common region containing org--modified-elements. if change > is happening within that region, there is no need to parse > drawers/blocks there again. > > P.S. > >>> It was mostly an annoyance, because they returned different results on >>> the same element. Specifically, they returned different :post-blank and >>> :end properties, which does not sound right. >> >> OK. If you have a reproducible recipe, I can look into it and see what >> can be done. > > Recipe to have different (org-element-at-point) and > (org-element-parse-buffer 'element) > ------------------------------------------------------------------------- > <point-min> > :PROPERTIES: > :CREATED: [2020-05-23 Sat 02:32] > :END: > > > <point-max> > ------------------------------------------------------------------------- > > > Best, > Ihor > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Hello, >> >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an >>>> element is a O(N) operation by the number of elements before it in >>>> a section. In particular, it is not bounded, and not mitigated by >>>> a cache. For large documents, it is going to be unbearably slow, too. >>> >>> Ouch. I thought it is faster. >>> What do you mean by "not mitigated by a cache"? >> >> Parsing starts from the closest headline, every time. So, if Org parses >> the Nth element in the entry two times, it really parses 2N elements. >> >> With a cache, assuming the buffer wasn't modified, Org would parse >> N elements only. With a smarter cache, with fine grained cache >> invalidation, it could also reduce the number of subsequent parsed >> elements. >> >>> The reason I would like to utilise org-element parser to make tracking >>> modifications more robust. Using details of the syntax would make the >>> code fragile if any modifications are made to syntax in future. >> >> I don't think the code would be more fragile. Also, the syntax we're >> talking about is not going to be modified anytime soon. Moreover, if >> folding breaks, it is usually visible, so the bug will not be unnoticed. >> >> This code is going to be as low-level as it can be. >> >>> Debugging bugs in modification functions is not easy, according to my >>> experience. >> >> No, it's not. >> >> But this is not really related to whether you use Element or not. >> >>> One possible way to avoid performance issues during modification is >>> running parser in advance. For example, folding an element may >>> as well add information about the element to its text properties. >>> This will not degrade performance of folding since we are already >>> parsing the element during folding (at least, in >>> org-hide-drawer-toggle). >> >> We can use this information stored at fold time. But I'm not even sure >> we need it. >> >>> The problem with parsing an element during folding is that we cannot >>> easily detect changes like below without re-parsing. >> >> Of course we can. It is only necessary to focus on changes that would >> break the structure of the element. This does not entail a full parsing. >> >>> :PROPERTIES: <folded> >>> :CREATED: [2020-05-18 Mon] >>> :END: <- added line >>> :ID: test >>> :END: >>> >>> or even >>> >>> :PROPERTIES: >>> :CREATED: [2020-05-18 Mon] >>> :ID: test >>> :END: <- delete this line >>> >>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion> >>> test >>> :END: >> >> Please have a look at the "sensitive parts" I wrote about. This takes >> care of this kind of breakage. >> >>> The re-parsing can be done via regexp, as you suggested, but I don't >>> like this idea, because it will end up re-implementing >>> org-element-*-parser. >> >> You may have misunderstood my suggestion. See below. >> >>> Would it be acceptable to run org-element-*-parser >>> in after-change-functions? >> >> I'd rather not do that. This is unnecessary consing, and matching, etc. >> >>> If I understand correctly, it is not as easy. >>> Consider the following example: >>> >>> :PROPERTIES: >>> :CREATED: [2020-05-18 Mon] >>> <region-beginning> >>> :ID: example >>> :END: >>> >>> <... a lot of text, maybe containing other drawers ...> >>> >>> Nullam rutrum. >>> Pellentesque dapibus suscipit ligula. >>> <region-end> >>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. >>> >>> If the region gets deleted, the modification hooks from chars inside >>> drawer will be called as (hook-function <region-beginning> >>> <region-end>). So, there is still a need to find the drawer somehow to >>> mark it as about to be modified (modification hooks are ran before >>> actual modification). >> >> If we can stick with `after-change-functions' (or local equivalent), >> that's better. It is more predictable than `before-change-functions' and >> alike. >> >> If it is a deletion, here is the kind of checks we could do, depending >> on when they are performed. >> >> Before actual changes : >> >> 1. The deletion is happening within a folded drawer (unnecessary step >> in local functions). >> 2. The change deleted the sensitive line ":END:". >> 3. Conclusion : unfold. >> >> Or, after actual changes : >> >> 1. The deletion involves a drawer. >> 2. Text properties indicate that the beginning of the propertized part >> of the buffer start with org-drawer-regexp, but doesn't end with >> `org-property-end-re'. A "sensitive part" disappeared! >> 3. Conclusion : unfold >> >> This is far away from parsing. IMO, a few checks cover all cases. Let me >> know if you have questions about it. >> >> Also, note that the kind of change you describe will happen perhaps >> 0.01% of the time. Most change are about one character, or a single >> line, long. >> >>> The only difference between using modification hooks and >>> before-change-functions is that modification hooks will trigger less >>> frequently. >> >> Exactly. Much less frequently. But extra care is required, as you noted >> already. >> >>> Considering the performance of org-element-at-point, it is >>> probably worth doing. Initially, I wanted to avoid it because setting a >>> single before-change-functions hook sounded cleaner than setting >>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks. >> >> Well, `before-change-fuctions' and `after-change-functions' are not >> clean at all: you modify an unrelated part of the buffer, but still call >> those to check if a drawer needs to be unfolded somewhere. >> >> And, more importantly, they are not meant to be used together, i.e., you >> cannot assume that a single call to `before-change-functions' always >> happens before calling `after-change-functions'. This can be tricky if >> you want to use the former to pass information to the latter. >> >> But I understand that they are easier to use than their local >> counterparts. If you stick with (before|after)-change-functions, the >> function being called needs to drop the ball very quickly if the >> modification is not about folding changes. Also, I very much suggest to >> stick to only `after-change-functions', if feasible (I think it is), per >> above. >> >>> Moreover, these text properties would be copied by default if one uses >>> buffer-substring. Then, the hooks will also trigger later in the yanked >>> text, which may cause all kinds of bugs. >> >> Indeed, that would be something to handle specifically. I.e., >> destructive modifications (i.e., those that unfold) could clear such >> properties. >> >> It is more work. I don't know if it is worth the trouble if we can get >> out quickly of `after-change-functions' for unrelated changes. >> >>> It was mostly an annoyance, because they returned different results on >>> the same element. Specifically, they returned different :post-blank and >>> :end properties, which does not sound right. >> >> OK. If you have a reproducible recipe, I can look into it and see what >> can be done. >> >> Regards, >> >> -- >> Nicolas Goaziou > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply related [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-23 13:53 ` Ihor Radchenko @ 2020-05-23 15:26 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-05-23 15:26 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Github link to the patch: https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef Ihor Radchenko <yantar92@gmail.com> writes: > The patch is attached > > diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el > index 9f8677871..ab470ea9b 100644 > --- a/contrib/lisp/org-notify.el > +++ b/contrib/lisp/org-notify.el > @@ -246,7 +246,7 @@ seconds. The default value for SECS is 20." > (switch-to-buffer (find-file-noselect file)) > (org-with-wide-buffer > (goto-char begin) > - (outline-show-entry)) > + (org-show-entry)) > (goto-char begin) > (search-forward "DEADLINE: <") > (search-forward ":") > diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el > index bfc4d6c3e..2312b235c 100644 > --- a/contrib/lisp/org-velocity.el > +++ b/contrib/lisp/org-velocity.el > @@ -325,7 +325,7 @@ use it." > (save-excursion > (when narrow > (org-narrow-to-subtree)) > - (outline-show-all))) > + (org-show-all))) > > (defun org-velocity-edit-entry/inline (heading) > "Edit entry at HEADING in the original buffer." > diff --git a/doc/org-manual.org b/doc/org-manual.org > index 96b160175..2ebe94538 100644 > --- a/doc/org-manual.org > +++ b/doc/org-manual.org > @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and > Switch back to the startup visibility of the buffer (see [[*Initial > visibility]]). > > -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) :: > +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) :: > > #+cindex: show all, command > #+kindex: C-u C-u C-u TAB > - #+findex: outline-show-all > + #+findex: org-show-all > Show all, including drawers. > > - {{{kbd(C-c C-r)}}} (~org-reveal~) :: > @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and > headings. With a double prefix argument, also show the entire > subtree of the parent. > > -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) :: > +- {{{kbd(C-c C-k)}}} (~org-show-branches~) :: > > #+cindex: show branches, command > #+kindex: C-c C-k > - #+findex: outline-show-branches > + #+findex: org-show-branches > Expose all the headings of the subtree, but not their bodies. > > -- {{{kbd(C-c TAB)}}} (~outline-show-children~) :: > +- {{{kbd(C-c TAB)}}} (~org-show-children~) :: > > #+cindex: show children, command > #+kindex: C-c TAB > - #+findex: outline-show-children > + #+findex: org-show-children > Expose all direct children of the subtree. With a numeric prefix > argument {{{var(N)}}}, expose all children down to level > {{{var(N)}}}. > @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way: > command (see [[*Visibility Cycling]]). You can force cycling archived > subtrees with {{{kbd(C-TAB)}}}, or by setting the option > ~org-cycle-open-archived-trees~. Also normal outline commands, like > - ~outline-show-all~, open archived subtrees. > + ~org-show-all~, open archived subtrees. > > - > #+vindex: org-sparse-tree-open-archived-trees > diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el > index ab13f926c..ad9244940 100644 > --- a/lisp/org-agenda.el > +++ b/lisp/org-agenda.el > @@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'." > (t " %-12:c%?-12t% s"))) > (start 0) > varform vars var e c f opt) > - (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)" > + (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)" > s start) > (setq var (or (cdr (assoc (match-string 4 s) > '(("c" . category) ("t" . time) ("l" . level) ("s" . extra) > @@ -9138,20 +9138,20 @@ if it was hidden in the outline." > ((and (called-interactively-p 'any) (= more 1)) > (message "Remote: show with default settings")) > ((= more 2) > - (outline-show-entry) > + (org-show-entry) > (org-show-children) > (save-excursion > (org-back-to-heading) > (run-hook-with-args 'org-cycle-hook 'children)) > (message "Remote: CHILDREN")) > ((= more 3) > - (outline-show-subtree) > + (org-show-subtree) > (save-excursion > (org-back-to-heading) > (run-hook-with-args 'org-cycle-hook 'subtree)) > (message "Remote: SUBTREE")) > ((> more 3) > - (outline-show-subtree) > + (org-show-subtree) > (message "Remote: SUBTREE AND ALL DRAWERS"))) > (select-window win))) > > diff --git a/lisp/org-archive.el b/lisp/org-archive.el > index d3e12d17b..d864dad8a 100644 > --- a/lisp/org-archive.el > +++ b/lisp/org-archive.el > @@ -330,7 +330,7 @@ direct children of this heading." > (insert (if datetree-date "" "\n") heading "\n") > (end-of-line 0)) > ;; Make the subtree visible > - (outline-show-subtree) > + (org-show-subtree) > (if org-archive-reversed-order > (progn > (org-back-to-heading t) > diff --git a/lisp/org-colview.el b/lisp/org-colview.el > index e50a4d7c8..e656df555 100644 > --- a/lisp/org-colview.el > +++ b/lisp/org-colview.el > @@ -699,7 +699,7 @@ FUN is a function called with no argument." > (move-beginning-of-line 2) > (org-at-heading-p t))))) > (unwind-protect (funcall fun) > - (when hide-body (outline-hide-entry))))) > + (when hide-body (org-hide-entry))))) > > (defun org-columns-previous-allowed-value () > "Switch to the previous allowed value for this column." > diff --git a/lisp/org-compat.el b/lisp/org-compat.el > index 635a38dcd..8fe271896 100644 > --- a/lisp/org-compat.el > +++ b/lisp/org-compat.el > @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer." > ;;; Emacs < 25.1 compatibility > > (when (< emacs-major-version 25) > - (defalias 'outline-hide-entry 'hide-entry) > - (defalias 'outline-hide-sublevels 'hide-sublevels) > - (defalias 'outline-hide-subtree 'hide-subtree) > (defalias 'outline-show-branches 'show-branches) > (defalias 'outline-show-children 'show-children) > - (defalias 'outline-show-entry 'show-entry) > (defalias 'outline-show-subtree 'show-subtree) > (defalias 'xref-find-definitions 'find-tag) > (defalias 'format-message 'format) > diff --git a/lisp/org-element.el b/lisp/org-element.el > index ac41b7650..2d5c8d771 100644 > --- a/lisp/org-element.el > +++ b/lisp/org-element.el > @@ -4320,7 +4320,7 @@ element or object. Meaningful values are `first-section', > TYPE is the type of the current element or object. > > If PARENT? is non-nil, assume the next element or object will be > -located inside the current one. " > +located inside the current one." > (if parent? > (pcase type > (`headline 'section) > diff --git a/lisp/org-keys.el b/lisp/org-keys.el > index c006e9c12..deb5d7b90 100644 > --- a/lisp/org-keys.el > +++ b/lisp/org-keys.el > @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names." > #'org-next-visible-heading) > (define-key org-mode-map [remap outline-previous-visible-heading] > #'org-previous-visible-heading) > -(define-key org-mode-map [remap show-children] #'org-show-children) > +(define-key org-mode-map [remap outline-show-children] #'org-show-children) > > ;;;; Make `C-c C-x' a prefix key > (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap)) > diff --git a/lisp/org-macs.el b/lisp/org-macs.el > index a02f713ca..fa0a658f0 100644 > --- a/lisp/org-macs.el > +++ b/lisp/org-macs.el > @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." > > > \f > -;;; Overlays > +;;; Overlays and text properties > > (defun org-overlay-display (ovl text &optional face evap) > "Make overlay OVL display TEXT with face FACE." > @@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays." > (delete (delete-overlay ov)) > (t (push ov found)))))) > > +(defun org-remove-text-properties (start end properties &optional object) > + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. > +Do not remove invisible text properties specified by 'outline, > +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this > +is needed to keep outlines, drawers, and blocks hidden unless they are > +toggled by user. > +Note: The below may be too specific and create troubles if more > +invisibility specs are added to org in future" > + (when (plist-member properties 'invisible) > + (let ((pos start) > + next spec) > + (while (< pos end) > + (setq next (next-single-property-change pos 'invisible nil end) > + spec (get-text-property pos 'invisible)) > + (unless (memq spec (list 'org-hide-block > + 'org-hide-drawer > + 'outline)) > + (remove-text-properties pos next '(invisible nil) object)) > + (setq pos next)))) > + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) > + (remove-text-properties start end properties-stripped object))) > + > +(defun org--find-text-property-region (pos prop) > + "Find a region containing PROP text property around point POS." > + (let* ((beg (and (get-text-property pos prop) pos)) > + (end beg)) > + (when beg > + ;; when beg is the first point in the region, `previous-single-property-change' > + ;; will return nil. > + (setq beg (or (previous-single-property-change pos prop) > + beg)) > + ;; when end is the last point in the region, `next-single-property-change' > + ;; will return nil. > + (setq end (or (next-single-property-change pos prop) > + end)) > + (unless (= beg end) ; this should not happen > + (cons beg end))))) > + > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > - (remove-overlays from to 'invisible spec) > - ;; Use `front-advance' since text right before to the beginning of > - ;; the overlay belongs to the visible line than to the contents. > - (when flag > - (let ((o (make-overlay from to nil 'front-advance))) > - (overlay-put o 'evaporate t) > - (overlay-put o 'invisible spec) > - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > - > + (pcase spec > + ('outline > + (remove-overlays from to 'invisible spec) > + ;; Use `front-advance' since text right before to the beginning of > + ;; the overlay belongs to the visible line than to the contents. > + (when flag > + (let ((o (make-overlay from to nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > + (_ > + ;; Use text properties instead of overlays for speed. > + ;; Overlays are too slow (Emacs Bug#35453). > + (with-silent-modifications > + ;; keep a backup stack of old text properties > + (save-excursion > + (goto-char from) > + (while (< (point) to) > + (let ((old-spec (get-text-property (point) 'invisible)) > + (end (next-single-property-change (point) 'invisible nil to))) > + (when old-spec > + (alter-text-property (point) end 'org-property-stack-invisible > + (lambda (stack) > + (if (or (eq old-spec (car stack)) > + (eq spec old-spec) > + (eq old-spec 'outline)) > + stack > + (cons old-spec stack))))) > + (goto-char end)))) > + > + ;; cleanup everything > + (remove-text-properties from to '(invisible nil)) > + > + ;; Recover properties from the backup stack > + (unless flag > + (save-excursion > + (goto-char from) > + (while (< (point) to) > + (let ((stack (get-text-property (point) 'org-property-stack-invisible)) > + (end (next-single-property-change (point) 'org-property-stack-invisible nil to))) > + (if (not stack) > + (remove-text-properties (point) end '(org-property-stack-invisible nil)) > + (put-text-property (point) end 'invisible (car stack)) > + (alter-text-property (point) end 'org-property-stack-invisible > + (lambda (stack) > + (cdr stack)))) > + (goto-char end))))) > + > + (when flag > + (put-text-property from to 'rear-non-sticky nil) > + (put-text-property from to 'front-sticky t) > + (put-text-property from to 'invisible spec)))))) > > \f > ;;; Regexp matching > diff --git a/lisp/org-src.el b/lisp/org-src.el > index c9eef744e..e89a1c580 100644 > --- a/lisp/org-src.el > +++ b/lisp/org-src.el > @@ -523,8 +523,8 @@ Leave point in edit buffer." > (org-src-switch-to-buffer buffer 'edit) > ;; Insert contents. > (insert contents) > - (remove-text-properties (point-min) (point-max) > - '(display nil invisible nil intangible nil)) > + (org-remove-text-properties (point-min) (point-max) > + '(display nil invisible nil intangible nil)) > (unless preserve-ind (org-do-remove-indentation)) > (set-buffer-modified-p nil) > (setq buffer-file-name nil) > diff --git a/lisp/org-table.el b/lisp/org-table.el > index 6462b99c4..75801161b 100644 > --- a/lisp/org-table.el > +++ b/lisp/org-table.el > @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'." > (arg > (let ((b (save-excursion (skip-chars-backward "^|") (point))) > (e (save-excursion (skip-chars-forward "^|\r\n") (point)))) > - (remove-text-properties b e '(invisible t intangible t)) > + (org-remove-text-properties b e '(invisible t intangible t)) > (if (and (boundp 'font-lock-mode) font-lock-mode) > (font-lock-fontify-block)))) > (t > @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'." > (setq word-wrap t) > (goto-char (setq p (point-max))) > (insert (org-trim field)) > - (remove-text-properties p (point-max) '(invisible t intangible t)) > + (org-remove-text-properties p (point-max) '(invisible t intangible t)) > (goto-char p) > (setq-local org-finish-function 'org-table-finish-edit-field) > (setq-local org-window-configuration cw) > diff --git a/lisp/org.el b/lisp/org.el > index e577dc661..360974135 100644 > --- a/lisp/org.el > +++ b/lisp/org.el > @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") > (declare-function cdlatex-math-symbol "ext:cdlatex") > (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) > (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) > +(declare-function isearch-filter-visible "isearch" (beg end)) > (declare-function org-add-archive-files "org-archive" (files)) > (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) > (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) > @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") > > (defvar ffap-url-regexp) > (defvar org-element-paragraph-separate) > +(defvar org-element-all-objects) > +(defvar org-element-all-elements) > +(defvar org-element-greater-elements) > (defvar org-indent-indentation-per-level) > (defvar org-radio-target-regexp) > (defvar org-target-link-regexp) > @@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.") > > ;;;; Define the Org mode > > +;;; Handling buffer modifications > + > (defun org-before-change-function (_beg _end) > "Every change indicates that a table might need an update." > (setq org-table-may-need-update t)) > + > +(defvar-local org--modified-elements nil > + "List of elements, marked as recently modified. > +There is no guarantee that the elements in this list are fully parsed. > +Only the element type, :begin and :end properties of the elements are > +guaranteed to be available. The :begin and :end element properties > +contain markers instead of positions.") > + > +(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command) > + "List of commands triggerring element modifications unconditionally.") > + > +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$") > + (property-drawer . ,org-property-start-re) > + (drawer . ,org-drawer-regexp) > + (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$") > + (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$")) > + "Alist of regexps matching beginning of elements. > +Group 1 in the regexps (if any) contains the element type.") > + > +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$") > + (property-drawer . ,org-property-end-re) > + (drawer . ,org-property-end-re) > + (quote-block . "^[ \t]*#\\+end_quote[ \t]*$") > + (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$")) > + "Alist of regexps matching end of elements. > +Group 1 in the regexps (if any) contains the element type or END.") > + > +(defvar org-track-element-modifications > + `((property-drawer . (:after-change-function > + org--drawer-or-block-unfold-maybe)) > + (drawer . (:after-change-function > + org--drawer-or-block-unfold-maybe)) > + (center-block . (:after-change-function > + org--drawer-or-block-unfold-maybe)) > + (quote-block . (:after-change-function > + org--drawer-or-block-unfold-maybe)) > + (special-block . (:after-change-function > + org--drawer-or-block-unfold-maybe))) > + "Alist of elements to be tracked for modifications. > +The modification is only triggered according to :sensitive-re-list and > +:sensitive-command-list (see below). > +Each element of the alist is a cons of an element symbol and plist > +defining how and when the modifications are handled. > +In case of recursive elements/duplicates, the first element from the > +list is considered. > +The plist can have the following properties: > +- :element-beginning-re :: regex matching beginning of the element > + (default) :: (alist-get element org--element-beginning-re-alist) > +- :element-end-re :: regex matching end of the element > + (default) :: (alist-get element org--element-end-re-alist) > +- :after-change-function :: function called after the modification > +The function must accept a single argument - element from > +`org--modified-elements'.") > + > +(defun org--get-element-region-at-point (types) > + "Return TYPES element at point or nil. > +If TYPES is a list, return first element at point from the list. The > +returned value is partially parsed element only containing :begin and > +:end properties. Only elements listed in > +org--element-beginning-re-alist and org--element-end-re-alist can be > +parsed here." > + (catch 'exit > + (dolist (type (if (listp types) types (list types))) > + (let ((begin-re (alist-get type org--element-beginning-re-alist)) > + (end-re (alist-get type org--element-end-re-alist)) > + (begin-limit (save-excursion (org-with-limited-levels > + (org-back-to-heading-or-point-min 'invisible-ok)) > + (point))) > + (end-limit (or (save-excursion (outline-next-heading)) > + (point-max))) > + (point (point)) > + begin end closest-begin) > + (when (and begin-re end-re) > + (save-excursion > + (end-of-line) > + (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point))) > + (when (re-search-forward end-re end-limit 'noerror) (setq end (point))) > + (setq closest-begin begin) > + ;; slurp unmatched begin-re > + (when (and begin end) > + (goto-char begin) > + (while (and (re-search-backward begin-re begin-limit 'noerror) > + (= end (save-excursion (re-search-forward end-re end-limit 'noerror)))) > + (setq begin (point))) > + (when (and (>= point begin) (<= point end)) > + (throw 'exit > + (list type > + (list > + :begin begin > + :end end))))))))))) > + > +(defun org--get-next-element-region-at-point (types &optional limit previous) > + "Return TYPES element after point or nil. > +If TYPES is a list, return first element after point from the list. > +If PREVIOUS is non-nil, return first TYPES element before point. > +Limit search by LIMIT or previous/next heading. > +The returned value is partially parsed element only containing :begin > +and :end properties. Only elements listed in > +org--element-beginning-re-alist and org--element-end-re-alist can be > +parsed here." > + (catch 'exit > + (dolist (type (if (listp types) types (list types))) > + (let* ((begin-re (alist-get type org--element-beginning-re-alist)) > + (end-re (alist-get type org--element-end-re-alist)) > + (limit (or limit (if previous > + (save-excursion > + (org-with-limited-levels > + (org-back-to-heading-or-point-min 'invisible-ok) > + (point))) > + (or (save-excursion (outline-next-heading)) > + (point-max))))) > + begin end) > + (when (and begin-re end-re) > + (save-excursion > + (if previous > + (when (re-search-backward begin-re limit 'noerror) > + (when-let ((el (org--get-element-region-at-point type))) > + (setq begin (org-element-property :begin el)) > + (setq end (org-element-property :end el)))) > + (when (re-search-forward begin-re limit 'noerror) > + (when-let ((el (org--get-element-region-at-point type))) > + (setq begin (org-element-property :begin el)) > + (setq end (org-element-property :end el)))))) > + (when (and begin end) > + (throw 'exit > + (list type > + (list > + :begin begin > + :end end))))))))) > + > +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours) > + "Find all elements from ELEMENTS in region BEG . END. > +All the listed elements must be resolvable by > +`org--get-element-region-at-point'. > +Include elements if they are partially inside region when > +INCLUDE-PARTIAL is non-nil. > +Include preceding/subsequent neighbouring elements when no partial > +element is found at the beginning/end of the region and > +INCLUDE-NEIGHBOURS is non-nil." > + (when include-partial > + (org-with-point-at beg > + (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements)))) > + (if new-beg > + (setq beg new-beg) > + (when (and include-neighbours > + (setq new-beg (org-element-property :begin > + (org--get-next-element-region-at-point elements > + (point-min) > + 'previous)))) > + (setq beg new-beg)))) > + (when (memq 'headline elements) > + (when-let ((new-beg (save-excursion > + (org-with-limited-levels (outline-previous-heading))))) > + (setq beg new-beg)))) > + (org-with-point-at end > + (let ((new-end (org-element-property :end (org--get-element-region-at-point elements)))) > + (if new-end > + (setq end new-end) > + (when (and include-neighbours > + (setq new-end (org-element-property :end > + (org--get-next-element-region-at-point elements (point-max))))) > + (setq end new-end)))) > + (when (memq 'headline elements) > + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) > + (setq end (1- new-end)))))) > + (save-excursion > + (save-restriction > + (narrow-to-region beg end) > + (goto-char (point-min)) > + (let (result el) > + (while (setq el (org--get-next-element-region-at-point elements end)) > + (push el result) > + (goto-char (org-element-property :end el))) > + result)))) > + > +(defun org--drawer-or-block-unfold-maybe (el) > + "Update visibility of modified folded drawer/block EL. > +If text was added to hidden drawer/block, make sure that the text is > +also hidden, unless the change was done by a command listed in > +`org-track-element-modification-default-sensitive-commands'. If the > +modification destroyed the drawer/block, reveal the hidden text in > +former drawer/block. If the modification shrinked/expanded the > +drawer/block beyond the hidden text, reveal the affected > +drawers/blocks as well. > +Examples: > +---------------------------------------------- > +---------------------------------------------- > +Case #1 (the element content is hidden): > +---------------------------------------------- > +:PROPERTIES: > +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > +:END: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:ROPERTIES: > +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > +:END: > +---------------------------------------------- > +Text is revealed, because we have drawer in place of property-drawer > +---------------------------------------------- > +---------------------------------------------- > +Case #2 (the element content is hidden): > +---------------------------------------------- > +:ROPERTIES: > +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > +:END: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:OPERTIES: > +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 > +:END: > +---------------------------------------------- > +The text remains hidden since it is still a drawer. > +---------------------------------------------- > +---------------------------------------------- > +Case #3: (the element content is hidden): > +---------------------------------------------- > +:FOO: > +bar > +tmp > +:END: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:FOO: > +bar > +:END: > +tmp > +:END: > +---------------------------------------------- > +The text is revealed because the drawer contents shrank. > +---------------------------------------------- > +---------------------------------------------- > +Case #4: (the element content is hidden in both the drawers): > +---------------------------------------------- > +:FOO: > +bar > +tmp > +:END: > +:BAR: > +jjd > +:END: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:FOO: > +bar > +tmp > +:BAR: > +jjd > +:END: > +---------------------------------------------- > +The text is revealed in both the drawers because the drawers are merged > +into a new drawer. > +---------------------------------------------- > +---------------------------------------------- > +Case #5: (the element content is hidden) > +---------------------------------------------- > +:test: > +Vivamus id enim. > +:end: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:drawer: > +:test: > +Vivamus id enim. > +:end: > +---------------------------------------------- > +The text is revealed in the drawer because the drawer expended. > +---------------------------------------------- > +---------------------------------------------- > +Case #6: (the element content is hidden): > +---------------------------------------------- > +:test: > +Vivamus id enim. > +:end: > +---------------------------------------------- > +is changed to > +---------------------------------------------- > +:test: > +Vivamus id enim. > +:end: > +Nam a sapien. > +:end: > +---------------------------------------------- > +The text remains hidden because drawer contents is always before the > +first :end:." > + (save-match-data > + (save-excursion > + (save-restriction > + (goto-char (org-element-property :begin el)) > + (let* ((newel (org--get-element-region-at-point > + (mapcar (lambda (el) > + (when (string-match-p (regexp-opt '("block" "drawer")) > + (symbol-name (car el))) > + (car el))) > + org-track-element-modifications))) > + (spec (if (string-match-p "block" (symbol-name (org-element-type el))) > + 'org-hide-block > + (if (string-match-p "drawer" (symbol-name (org-element-type el))) > + 'org-hide-drawer > + t))) > + (toggle-func (if (eq spec 'org-hide-drawer) > + #'org-hide-drawer-toggle > + (if (eq spec 'org-hide-block) > + #'org-hide-block-toggle > + #'ignore)))) ; this should not happen > + (if (and (equal (org-element-type el) (org-element-type newel)) > + (equal (marker-position (org-element-property :begin el)) > + (org-element-property :begin newel)) > + (equal (marker-position (org-element-property :end el)) > + (org-element-property :end newel))) > + (when (text-property-any (marker-position (org-element-property :begin el)) > + (marker-position (org-element-property :end el)) > + 'invisible spec) > + (goto-char (org-element-property :begin newel)) > + (if (memq this-command org-track-element-modification-default-sensitive-commands) > + ;; reveal if change was made by typing > + (funcall toggle-func 'off) > + ;; re-hide the inserted text > + ;; FIXME: opening the drawer before hiding should not be needed here > + (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis > + (funcall toggle-func 'hide))) > + ;; The element was destroyed. Reveal everything. > + (org-flag-region (marker-position (org-element-property :begin el)) > + (marker-position (org-element-property :end el)) > + nil spec) > + (when newel > + (org-flag-region (org-element-property :begin newel) > + (org-element-property :end newel) > + nil spec)))))))) > + > +(defun org--before-element-change-function (beg end) > + "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END." > + (save-match-data > + (save-excursion > + (save-restriction > + (widen) > + (dolist (el (org--find-elements-in-region beg > + end > + (mapcar #'car org-track-element-modifications) > + 'include-partial > + 'include-neighbours)) > + (let* ((beg-marker (copy-marker (org-element-property :begin el) 't)) > + (end-marker (copy-marker (org-element-property :end el) 't))) > + (when (and (marker-position beg-marker) (marker-position end-marker)) > + (org-element-put-property el :begin beg-marker) > + (org-element-put-property el :end end-marker) > + (add-to-list 'org--modified-elements el)))))))) > + > +;; FIXME: this function may be called many times during routine modifications > +;; The normal way to avoid this is `combine-after-change-calls' - not > +;; the case in most org primitives. > +(defun org--after-element-change-function (&rest _) > + "Handle changed elements from `org--modified-elements'." > + (dolist (el org--modified-elements) > + (save-match-data > + (save-excursion > + (save-restriction > + (widen) > + (when-let* ((type (org-element-type el)) > + (change-func (plist-get (alist-get type org-track-element-modifications) > + :after-change-function))) > + (with-demoted-errors > + (funcall (symbol-function change-func) el))))))) > + (setq org--modified-elements nil)) > + > (defvar org-mode-map) > (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. > (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. > @@ -4818,6 +5194,9 @@ The following commands are available: > ;; Activate before-change-function > (setq-local org-table-may-need-update t) > (add-hook 'before-change-functions 'org-before-change-function nil 'local) > + (add-hook 'before-change-functions 'org--before-element-change-function nil 'local) > + ;; Activate after-change-function > + (add-hook 'after-change-functions 'org--after-element-change-function nil 'local) > ;; Check for running clock before killing a buffer > (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) > ;; Initialize macros templates. > @@ -4869,6 +5248,10 @@ The following commands are available: > (setq-local outline-isearch-open-invisible-function > (lambda (&rest _) (org-show-context 'isearch))) > > + ;; Make isearch search in blocks hidden via text properties > + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) > + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) > + > ;; Setup the pcomplete hooks > (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) > (setq-local pcomplete-command-name-function #'org-command-at-point) > @@ -5050,8 +5433,8 @@ stacked delimiters is N. Escaping delimiters is not possible." > (when verbatim? > (org-remove-flyspell-overlays-in > (match-beginning 0) (match-end 0)) > - (remove-text-properties (match-beginning 2) (match-end 2) > - '(display t invisible t intangible t))) > + (org-remove-text-properties (match-beginning 2) (match-end 2) > + '(display t invisible t intangible t))) > (add-text-properties (match-beginning 2) (match-end 2) > '(font-lock-multiline t org-emphasis t)) > (when (and org-hide-emphasis-markers > @@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links." > (if (not (eq 'bracket style)) > (add-text-properties start end properties) > ;; Handle invisible parts in bracket links. > - (remove-text-properties start end '(invisible nil)) > + (org-remove-text-properties start end '(invisible nil)) > (let ((hidden > (append `(invisible > ,(or (org-link-get-parameter type :display) > @@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links." > (defun org-activate-code (limit) > (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) > (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) > - (remove-text-properties (match-beginning 0) (match-end 0) > - '(display t invisible t intangible t)) > + (org-remove-text-properties (match-beginning 0) (match-end 0) > + '(display t invisible t intangible t)) > t)) > > (defcustom org-src-fontify-natively t > @@ -5258,8 +5641,8 @@ by a #." > (setq block-end (match-beginning 0)) ; includes the final newline. > (when quoting > (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline) > - (remove-text-properties beg end-of-endline > - '(display t invisible t intangible t))) > + (org-remove-text-properties beg end-of-endline > + '(display t invisible t intangible t))) > (add-text-properties > beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) > (org-remove-flyspell-overlays-in beg bol-after-beginline) > @@ -5313,8 +5696,8 @@ by a #." > '(font-lock-fontified t face org-document-info)))) > ((string-prefix-p "+caption" dc1) > (org-remove-flyspell-overlays-in (match-end 2) (match-end 0)) > - (remove-text-properties (match-beginning 0) (match-end 0) > - '(display t invisible t intangible t)) > + (org-remove-text-properties (match-beginning 0) (match-end 0) > + '(display t invisible t intangible t)) > ;; Handle short captions. > (save-excursion > (beginning-of-line) > @@ -5336,8 +5719,8 @@ by a #." > '(font-lock-fontified t face font-lock-comment-face))) > (t ;; just any other in-buffer setting, but not indented > (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) > - (remove-text-properties (match-beginning 0) (match-end 0) > - '(display t invisible t intangible t)) > + (org-remove-text-properties (match-beginning 0) (match-end 0) > + '(display t invisible t intangible t)) > (add-text-properties beg (match-end 0) > '(font-lock-fontified t face org-meta-line)) > t)))))) > @@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group." > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > - (remove-text-properties beg end > - '(mouse-face t keymap t org-linked-text t > - invisible t intangible t > - org-emphasis t)) > + (org-remove-text-properties beg end > + '(mouse-face t keymap t org-linked-text t > + invisible t > + intangible t > + org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) > > (defconst org-script-display '(((raise -0.3) (height 0.7)) > @@ -5970,6 +6354,29 @@ open and agenda-wise Org files." > > ;;;; Headlines visibility > > +(defun org-hide-entry () > + "Hide the body directly following this heading." > + (interactive) > + (save-excursion > + (outline-back-to-heading) > + (outline-end-of-heading) > + (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline))) > + > +(defun org-hide-subtree () > + "Hide everything after this heading at deeper levels." > + (interactive) > + (org-flag-subtree t)) > + > +(defun org-hide-sublevels (levels) > + "Hide everything but the top LEVELS levels of headers, in whole buffer. > +This also unhides the top heading-less body, if any. > + > +Interactively, the prefix argument supplies the value of LEVELS. > +When invoked without a prefix argument, LEVELS defaults to the level > +of the current heading, or to 1 if the current line is not a heading." > + (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region)) > + (org-hide-sublevels levels))) > + > (defun org-show-entry () > "Show the body directly following this heading. > Show the heading too, if it is currently invisible." > @@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible." > 'outline) > (org-cycle-hide-property-drawers 'children)))) > > +(defun org-show-heading () > + "Show the current heading and move to its end." > + (org-flag-region (- (point) > + (if (bobp) 0 > + (if (and outline-blank-line > + (eq (char-before (1- (point))) ?\n)) > + 2 1))) > + (progn (outline-end-of-heading) (point)) > + nil)) > + > (defun org-show-children (&optional level) > "Show all direct subheadings of this heading. > Prefix arg LEVEL is how many levels below the current level > @@ -6031,6 +6448,11 @@ heading to appear." > (org-flag-region > (point) (save-excursion (org-end-of-subtree t t)) nil 'outline)) > > +(defun org-show-branches () > + "Show all subheadings of this heading, but not their bodies." > + (interactive) > + (org-show-children 1000)) > + > ;;;; Blocks and drawers visibility > > (defun org--hide-wrapper-toggle (element category force no-error) > @@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful." > (unless (let ((eol (line-end-position))) > (and (> eol start) (/= eol end))) > (let* ((spec (cond ((eq category 'block) 'org-hide-block) > - ((eq type 'property-drawer) 'outline) > - (t 'org-hide-drawer))) > + ((eq category 'drawer) 'org-hide-drawer) > + (t 'outline))) > (flag > (cond ((eq force 'off) nil) > (force t) > @@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of > (when (org-at-property-drawer-p) > (let* ((case-fold-search t) > (end (re-search-forward org-property-end-re))) > - ;; Property drawers use `outline' invisibility spec > - ;; so they can be swallowed once we hide the > - ;; outline. > - (org-flag-region start end t 'outline))))))))))) > + (org-flag-region start end t 'org-hide-drawer))))))))))) > > ;;;; Visibility cycling > > @@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level." > (org-narrow-to-subtree) > (org-content)))) > ((or "all" "showall") > - (outline-show-subtree)) > + (org-show-subtree)) > (_ nil))) > (org-end-of-subtree))))))) > > @@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'." > (while (re-search-forward re nil t) > (when (and (not (org-invisible-p)) > (org-invisible-p (line-end-position))) > - (outline-hide-entry)))) > + (org-hide-entry)))) > (org-cycle-hide-property-drawers 'all) > (org-cycle-show-empty-lines 'overview))))) > > @@ -6683,8 +7102,13 @@ information." > ;; expose it. > (dolist (o (overlays-at (point))) > (when (memq (overlay-get o 'invisible) > - '(org-hide-block org-hide-drawer outline)) > + '(outline)) > (delete-overlay o))) > + (when (memq (get-text-property (point) 'invisible) > + '(org-hide-block org-hide-drawer)) > + (let ((spec (get-text-property (point) 'invisible)) > + (region (org--find-text-property-region (point) 'invisible))) > + (org-flag-region (car region) (cdr region) nil spec))) > (unless (org-before-first-heading-p) > (org-with-limited-levels > (cl-case detail > @@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard." > (skip-chars-forward " \t\n\r") > (setq beg (point)) > (when (and (org-invisible-p) visp) > - (save-excursion (outline-show-heading))) > + (save-excursion (org-show-heading))) > ;; Shift if necessary. > (unless (= shift 0) > (save-restriction > @@ -8103,7 +8527,7 @@ function is being called interactively." > (point)) > what "children") > (goto-char start) > - (outline-show-subtree) > + (org-show-subtree) > (outline-next-heading)) > (t > ;; we will sort the top-level entries in this file > @@ -13150,7 +13574,7 @@ drawer is immediately hidden." > (inhibit-read-only t)) > (unless (bobp) (insert "\n")) > (insert ":PROPERTIES:\n:END:") > - (org-flag-region (line-end-position 0) (point) t 'outline) > + (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer) > (when (or (eobp) (= begin (point-min))) (insert "\n")) > (org-indent-region begin (point)))))) > > @@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer." > (defun org-show-branches-buffer () > "Show all branches in the buffer." > (org-flag-above-first-heading) > - (outline-hide-sublevels 1) > + (org-hide-sublevels 1) > (unless (eobp) > - (outline-show-branches) > + (org-show-branches) > (while (outline-get-next-sibling) > - (outline-show-branches))) > + (org-show-branches))) > (goto-char (point-min))) > > (defun org-kill-note-or-show-branches () > @@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer." > (t > (let ((beg (progn (org-back-to-heading) (point))) > (end (save-excursion (org-end-of-subtree t t) (point)))) > - (outline-hide-subtree) > - (outline-show-branches) > + (org-hide-subtree) > + (org-show-branches) > (org-hide-archived-subtrees beg end))))) > > (defun org-delete-indentation (&optional arg) > @@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'. ARG is the level to hide." > (if (org-before-first-heading-p) > (progn > (org-flag-above-first-heading) > - (outline-hide-sublevels (or arg 1)) > + (org-hide-sublevels (or arg 1)) > (goto-char (point-min))) > - (outline-hide-subtree) > + (org-hide-subtree) > (org-show-children arg)))) > > (defun org-ctrl-c-star () > @@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'." > (t default-org-info-node)))))) > > \f > + > +;;; Make isearch search in some text hidden via text propertoes > + > +(defvar org--isearch-overlays nil > + "List of overlays temporarily created during isearch. > +This is used to allow searching in regions hidden via text properties. > +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. > +Any text hidden via text properties is not revealed even if `search-invisible' > +is set to 't.") > + > +;; Not sure if it needs to be a user option > +;; One might want to reveal hidden text in, for example, hidden parts of the links. > +;; Currently, hidden text in links is never revealed by isearch. > +(defvar org-isearch-specs '(org-hide-block > + org-hide-drawer) > + "List of text invisibility specs to be searched by isearch. > +By default ([2020-05-09 Sat]), isearch does not search in hidden text, > +which was made invisible using text properties. Isearch will be forced > +to search in hidden text with any of the listed 'invisible property value.") > + > +(defun org--create-isearch-overlays (beg end) > + "Replace text property invisibility spec by overlays between BEG and END. > +All the regions with invisibility text property spec from > +`org-isearch-specs' will be changed to use overlays instead > +of text properties. The created overlays will be stored in > +`org--isearch-overlays'." > + (let ((pos beg)) > + (while (< pos end) > + (when-let* ((spec (get-text-property pos 'invisible)) > + (spec (memq spec org-isearch-specs)) > + (region (org--find-text-property-region pos 'invisible))) > + (setq spec (get-text-property pos 'invisible)) > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] > + ;; overlay for 'outline blocks. > + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) > + (overlay-put o 'evaporate t) > + (overlay-put o 'invisible spec) > + ;; `delete-overlay' here means that spec information will be lost > + ;; for the region. The region will remain visible. > + (overlay-put o 'isearch-open-invisible #'delete-overlay) > + (push o org--isearch-overlays)) > + (org-flag-region (car region) (cdr region) nil spec))) > + (setq pos (next-single-property-change pos 'invisible nil end))))) > + > +(defun org--isearch-filter-predicate (beg end) > + "Return non-nil if text between BEG and END is deemed visible by Isearch. > +This function is intended to be used as `isearch-filter-predicate'. > +Unlike `isearch-filter-visible', make text with 'invisible text property > +value listed in `org-isearch-specs' visible to Isearch." > + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text > + (isearch-filter-visible beg end)) > + > +(defun org--clear-isearch-overlay (ov) > + "Convert OV region back into using text properties." > + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays > + ;; Changing text properties is considered buffer modification. > + ;; We do not want it here. > + (with-silent-modifications > + (org-flag-region (overlay-start ov) (overlay-end ov) t spec))) > + (when (member ov isearch-opened-overlays) > + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) > + (delete-overlay ov)) > + > +(defun org--clear-isearch-overlays () > + "Convert overlays from `org--isearch-overlays' back into using text properties." > + (when org--isearch-overlays > + (mapc #'org--clear-isearch-overlay org--isearch-overlays) > + (setq org--isearch-overlays nil))) > + > +\f > + > ;;; Finish up > > (add-hook 'org-mode-hook ;remove overlays when changing major mode > > > Ihor Radchenko <yantar92@gmail.com> writes: > >> Hello, >> >> [The patch itself will be provided in the following email] >> >> I have five updates from the previous version of the patch: >> >> 1. I implemented a simplified version of element parsing to detect >> changes in folded drawers or blocks. No computationally expensive calls >> of org-element-at-point or org-element-parse-buffer are needed now. >> >> 2. The patch is now compatible with master (commit 2e96dc639). I >> reverted the earlier change in folding drawers and blocks. Now, they are >> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would >> achieve nothing when we use text properties. >> >> 3. 'invisible text property can now be nested. This is important, for >> example, when text inside drawers contains fontified links (which also >> use 'invisible text property to hide parts of the link). Now, the old >> 'invisible spec is recovered after unfolding. >> >> 4. Some outline-* function calls in org referred to outline-flag-region >> implementation, which is not in sync with org-flag-region in this patch. >> I have implemented their org-* versions and replaced the calls >> throughout .el files. Actually, some org-* versions were already >> implemented in org, but not used for some reason (or not mentioned in >> the manual). I have updated the relevant sections of manual. These >> changes might be relevant to org independently of this feature branch. >> >> 5. I have managed to get a working version of outline folding via text >> properties. However, that approach has a big downside - folding state >> cannot be different in indirect buffer when we use text properties. I >> have seen packages relying on this feature of org and I do not see any >> obvious way to achieve different folding state in indirect buffer while >> using text properties for outline folding. >> >> ----------------------------------------------------------------------- >> ----------------------------------------------------------------------- >> >> More details on the new implementation for tracking changes: >> >>> Of course we can. It is only necessary to focus on changes that would >>> break the structure of the element. This does not entail a full parsing. >> >> I have limited parsing to matching beginning and end of a drawer/block. >> The basic functions are org--get-element-region-at-point, >> org--get-next-element-region-at-point, and org--find-elements-in-region. >> They are simplified versions of org-element-* parsers and do not require >> parsing everything from the beginning of the section. >> >> For now, I keep everything in org.el, but those simplified parsers >> probably belong to org-element.el. >> >>> If we can stick with `after-change-functions' (or local equivalent), >>> that's better. It is more predictable than `before-change-functions' and >>> alike. >> >> For now, I still used before/after-change-functions combination. >> I see the following problems with using only after-change-functions: >> >> 1. They are not guaranteed to be called after every single change: >> >> From (elisp) Change Hooks: >> "... some complex primitives call ‘before-change-functions’ once before >> making changes, and then call ‘after-change-functions’ zero or more >> times" >> >> The consequence of it is a possibility that region passed to the >> after-change-functions is quite big (including all the singular changes, >> even if they are distant). This region may contain changed drawers as >> well and unchanged drawers and needs to be parsed to determine which >> drawers need to be re-folded. >> >>> And, more importantly, they are not meant to be used together, i.e., you >>> cannot assume that a single call to `before-change-functions' always >>> happens before calling `after-change-functions'. This can be tricky if >>> you want to use the former to pass information to the latter. >> >> The fact that before-change-functions can be called multiple times >> before after-change-functions, is trivially solved by using buffer-local >> changes register (see org--modified-elements). The register is populated >> by before-change-functions and cleared by after-change-functions. >> >>> Well, `before-change-fuctions' and `after-change-functions' are not >>> clean at all: you modify an unrelated part of the buffer, but still call >>> those to check if a drawer needs to be unfolded somewhere. >> >> 2. As you pointed, instead of global before-change-functions, we can use >> modification-hooks text property on sensitive parts of the >> drawers/blocks. This would work, but I am concerned about one annoying >> special case: >> >> ------------------------------------------------------------------------- >> :BLAH: <inserted outside any of the existing drawers> >> >> <some text> >> >> :DRAWER: <folded> >> Donec at pede. >> :END: >> ------------------------------------------------------------------------- >> In this example, the user would not be able to unfold the folder DRAWER >> because it will technically become a part of a new giant BLAH drawer. >> This may be especially annoying if <some text> is more than one screen >> long and there is no easy way to identify why unfolding does not work >> (with point at :DRAWER:). >> >> Because of this scenario, limiting before-change-functions to folded >> drawers is not sufficient. Any change in text may need to trigger >> unfolding. >> >> In the patch, I always register possible modifications in the >> blocks/drawers intersecting with the modified region + a drawer/block >> right next to the region. >> >> ----------------------------------------------------------------------- >> ----------------------------------------------------------------------- >> >> More details on the nested 'invisible text property implementation. >> >> The idea is to keep 'invisible property stack push and popping from it >> as we add/remove 'invisible text property. All the work is done in >> org-flag-region. >> >> This was originally intended for folding outlines via text properties. >> Since using text properties for folding outlines is not a good idea, >> nested text properties have much less use. As I mentioned, they do >> preserve link fontification, but I am not sure if it worth it for the >> overhead to org-flag-region. Keeping this here mostly in the case if >> someone has any ideas how it can be useful. >> >> ----------------------------------------------------------------------- >> ----------------------------------------------------------------------- >> >> More details on replaced outline-* -> org-* function calls. >> >> I have implemented org-* versions of the following functions: >> >> - outline-hide-entry >> - outline-hide-subtree >> - outline-hide-sublevels >> - outline-show-heading >> - outline-show-branches >> >> The org-* versions trivially use org-flag-region instead of >> outline-flag-region. >> >> Replaced outline-* calls where org- versions were already available: >> >> - outline-show-entry >> - outline-show-all >> - outline-show-subtree >> >> I reflected the new (including already available) functions in the >> manual and removed some defalias from org-compat.el where they are not >> needed. >> >> ----------------------------------------------------------------------- >> ----------------------------------------------------------------------- >> >> Further work: >> >> 1. after-change-functions use org-hide-drawer/block-toggle to >> fold/unfold after modification. However, I just found that they call >> org-element-at-point, which slows down modifications in folded >> drawers/blocks. For example, realigning a long table inside folded >> drawer takes >1sec, while it is instant in the unfolded drawer. >> >> 2. org-toggle-custom-properties is terribly slow on large org documents, >> similarly to folded drawers on master. It should be relatively easy to >> use text properties there instead of overlays. >> >> 3. Multiple calls to before/after-change-functions is still a problem. I >> am looking into following ways to reduce this number: >> - reduce the number of elements registered as potentially modified >> + do not add duplicates to org--modified-elements >> + do not add unfolded elements to org--modified-elements >> + register after-change-function as post-command hook and remove it >> from global after-change-functions. This way, it will be called >> twice per command only. >> - determine common region containing org--modified-elements. if change >> is happening within that region, there is no need to parse >> drawers/blocks there again. >> >> P.S. >> >>>> It was mostly an annoyance, because they returned different results on >>>> the same element. Specifically, they returned different :post-blank and >>>> :end properties, which does not sound right. >>> >>> OK. If you have a reproducible recipe, I can look into it and see what >>> can be done. >> >> Recipe to have different (org-element-at-point) and >> (org-element-parse-buffer 'element) >> ------------------------------------------------------------------------- >> <point-min> >> :PROPERTIES: >> :CREATED: [2020-05-23 Sat 02:32] >> :END: >> >> >> <point-max> >> ------------------------------------------------------------------------- >> >> >> Best, >> Ihor >> >> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: >> >>> Hello, >>> >>> Ihor Radchenko <yantar92@gmail.com> writes: >>> >>>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an >>>>> element is a O(N) operation by the number of elements before it in >>>>> a section. In particular, it is not bounded, and not mitigated by >>>>> a cache. For large documents, it is going to be unbearably slow, too. >>>> >>>> Ouch. I thought it is faster. >>>> What do you mean by "not mitigated by a cache"? >>> >>> Parsing starts from the closest headline, every time. So, if Org parses >>> the Nth element in the entry two times, it really parses 2N elements. >>> >>> With a cache, assuming the buffer wasn't modified, Org would parse >>> N elements only. With a smarter cache, with fine grained cache >>> invalidation, it could also reduce the number of subsequent parsed >>> elements. >>> >>>> The reason I would like to utilise org-element parser to make tracking >>>> modifications more robust. Using details of the syntax would make the >>>> code fragile if any modifications are made to syntax in future. >>> >>> I don't think the code would be more fragile. Also, the syntax we're >>> talking about is not going to be modified anytime soon. Moreover, if >>> folding breaks, it is usually visible, so the bug will not be unnoticed. >>> >>> This code is going to be as low-level as it can be. >>> >>>> Debugging bugs in modification functions is not easy, according to my >>>> experience. >>> >>> No, it's not. >>> >>> But this is not really related to whether you use Element or not. >>> >>>> One possible way to avoid performance issues during modification is >>>> running parser in advance. For example, folding an element may >>>> as well add information about the element to its text properties. >>>> This will not degrade performance of folding since we are already >>>> parsing the element during folding (at least, in >>>> org-hide-drawer-toggle). >>> >>> We can use this information stored at fold time. But I'm not even sure >>> we need it. >>> >>>> The problem with parsing an element during folding is that we cannot >>>> easily detect changes like below without re-parsing. >>> >>> Of course we can. It is only necessary to focus on changes that would >>> break the structure of the element. This does not entail a full parsing. >>> >>>> :PROPERTIES: <folded> >>>> :CREATED: [2020-05-18 Mon] >>>> :END: <- added line >>>> :ID: test >>>> :END: >>>> >>>> or even >>>> >>>> :PROPERTIES: >>>> :CREATED: [2020-05-18 Mon] >>>> :ID: test >>>> :END: <- delete this line >>>> >>>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion> >>>> test >>>> :END: >>> >>> Please have a look at the "sensitive parts" I wrote about. This takes >>> care of this kind of breakage. >>> >>>> The re-parsing can be done via regexp, as you suggested, but I don't >>>> like this idea, because it will end up re-implementing >>>> org-element-*-parser. >>> >>> You may have misunderstood my suggestion. See below. >>> >>>> Would it be acceptable to run org-element-*-parser >>>> in after-change-functions? >>> >>> I'd rather not do that. This is unnecessary consing, and matching, etc. >>> >>>> If I understand correctly, it is not as easy. >>>> Consider the following example: >>>> >>>> :PROPERTIES: >>>> :CREATED: [2020-05-18 Mon] >>>> <region-beginning> >>>> :ID: example >>>> :END: >>>> >>>> <... a lot of text, maybe containing other drawers ...> >>>> >>>> Nullam rutrum. >>>> Pellentesque dapibus suscipit ligula. >>>> <region-end> >>>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. >>>> >>>> If the region gets deleted, the modification hooks from chars inside >>>> drawer will be called as (hook-function <region-beginning> >>>> <region-end>). So, there is still a need to find the drawer somehow to >>>> mark it as about to be modified (modification hooks are ran before >>>> actual modification). >>> >>> If we can stick with `after-change-functions' (or local equivalent), >>> that's better. It is more predictable than `before-change-functions' and >>> alike. >>> >>> If it is a deletion, here is the kind of checks we could do, depending >>> on when they are performed. >>> >>> Before actual changes : >>> >>> 1. The deletion is happening within a folded drawer (unnecessary step >>> in local functions). >>> 2. The change deleted the sensitive line ":END:". >>> 3. Conclusion : unfold. >>> >>> Or, after actual changes : >>> >>> 1. The deletion involves a drawer. >>> 2. Text properties indicate that the beginning of the propertized part >>> of the buffer start with org-drawer-regexp, but doesn't end with >>> `org-property-end-re'. A "sensitive part" disappeared! >>> 3. Conclusion : unfold >>> >>> This is far away from parsing. IMO, a few checks cover all cases. Let me >>> know if you have questions about it. >>> >>> Also, note that the kind of change you describe will happen perhaps >>> 0.01% of the time. Most change are about one character, or a single >>> line, long. >>> >>>> The only difference between using modification hooks and >>>> before-change-functions is that modification hooks will trigger less >>>> frequently. >>> >>> Exactly. Much less frequently. But extra care is required, as you noted >>> already. >>> >>>> Considering the performance of org-element-at-point, it is >>>> probably worth doing. Initially, I wanted to avoid it because setting a >>>> single before-change-functions hook sounded cleaner than setting >>>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks. >>> >>> Well, `before-change-fuctions' and `after-change-functions' are not >>> clean at all: you modify an unrelated part of the buffer, but still call >>> those to check if a drawer needs to be unfolded somewhere. >>> >>> And, more importantly, they are not meant to be used together, i.e., you >>> cannot assume that a single call to `before-change-functions' always >>> happens before calling `after-change-functions'. This can be tricky if >>> you want to use the former to pass information to the latter. >>> >>> But I understand that they are easier to use than their local >>> counterparts. If you stick with (before|after)-change-functions, the >>> function being called needs to drop the ball very quickly if the >>> modification is not about folding changes. Also, I very much suggest to >>> stick to only `after-change-functions', if feasible (I think it is), per >>> above. >>> >>>> Moreover, these text properties would be copied by default if one uses >>>> buffer-substring. Then, the hooks will also trigger later in the yanked >>>> text, which may cause all kinds of bugs. >>> >>> Indeed, that would be something to handle specifically. I.e., >>> destructive modifications (i.e., those that unfold) could clear such >>> properties. >>> >>> It is more work. I don't know if it is worth the trouble if we can get >>> out quickly of `after-change-functions' for unrelated changes. >>> >>>> It was mostly an annoyance, because they returned different results on >>>> the same element. Specifically, they returned different :post-blank and >>>> :end properties, which does not sound right. >>> >>> OK. If you have a reproducible recipe, I can look into it and see what >>> can be done. >>> >>> Regards, >>> >>> -- >>> Nicolas Goaziou >> >> -- >> Ihor Radchenko, >> PhD, >> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) >> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China >> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-23 13:52 ` Ihor Radchenko 2020-05-23 13:53 ` Ihor Radchenko @ 2020-05-26 8:33 ` Nicolas Goaziou 2020-06-02 9:21 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-05-26 8:33 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > I have five updates from the previous version of the patch: Thank you. > 1. I implemented a simplified version of element parsing to detect > changes in folded drawers or blocks. No computationally expensive calls > of org-element-at-point or org-element-parse-buffer are needed now. > > 2. The patch is now compatible with master (commit 2e96dc639). I > reverted the earlier change in folding drawers and blocks. Now, they are > back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would > achieve nothing when we use text properties. > > 3. 'invisible text property can now be nested. This is important, for > example, when text inside drawers contains fontified links (which also > use 'invisible text property to hide parts of the link). Now, the old > 'invisible spec is recovered after unfolding. Interesting. I'm running out of time, so I cannot properly inspect the code right now. I'll try to do that before the end of the week. > 4. Some outline-* function calls in org referred to outline-flag-region > implementation, which is not in sync with org-flag-region in this patch. > I have implemented their org-* versions and replaced the calls > throughout .el files. Actually, some org-* versions were already > implemented in org, but not used for some reason (or not mentioned in > the manual). I have updated the relevant sections of manual. These > changes might be relevant to org independently of this feature branch. Yes, we certainly want to move to org-specific versions in all cases. > 5. I have managed to get a working version of outline folding via text > properties. However, that approach has a big downside - folding state > cannot be different in indirect buffer when we use text properties. I > have seen packages relying on this feature of org and I do not see any > obvious way to achieve different folding state in indirect buffer while > using text properties for outline folding. Hmm. Good point. This is a serious issue to consider. Even if we don't use text properties for outline, this also affects drawers and blocks. > For now, I still used before/after-change-functions combination. You shouldn't. > I see the following problems with using only after-change-functions: > > 1. They are not guaranteed to be called after every single change: Of course they are! See below. > From (elisp) Change Hooks: > "... some complex primitives call ‘before-change-functions’ once before > making changes, and then call ‘after-change-functions’ zero or more > times" "zero" means there are no changes at all, so, `after-change-functions' are not called, which is expected. > The consequence of it is a possibility that region passed to the > after-change-functions is quite big (including all the singular changes, > even if they are distant). This region may contain changed drawers as > well and unchanged drawers and needs to be parsed to determine which > drawers need to be re-folded. It seems you're getting it backwards. `before-change-functions' are the functions being called with a possibly wide, imprecise, region to handle: When that happens, the arguments to ‘before-change-functions’ will enclose a region in which the individual changes are made, but won’t necessarily be the minimal such region however, after-change-functions calls are always minimal: and the arguments to each successive call of ‘after-change-functions’ will then delimit the part of text being changed exactly. If you stick to `after-change-functions', there will be no such thing as you describe. >> And, more importantly, they are not meant to be used together, i.e., you >> cannot assume that a single call to `before-change-functions' always >> happens before calling `after-change-functions'. This can be tricky if >> you want to use the former to pass information to the latter. > > The fact that before-change-functions can be called multiple times > before after-change-functions, is trivially solved by using buffer-local > changes register (see org--modified-elements). Famous last words. Been there, done that, and it failed. Let me quote the manual: In general, we advise to use either before- or the after-change hooks, but not both. So, let me insist: don't do that. If you don't agree with me, let's at least agree with Emacs developers. > The register is populated by before-change-functions and cleared by > after-change-functions. You cannot expect `after-change-functions' to clear what `before-change-functions' did. This is likely to introduce pernicious bugs. Sorry if it sounds like FUD, but bugs in those areas are just horrible to squash. >> Well, `before-change-fuctions' and `after-change-functions' are not >> clean at all: you modify an unrelated part of the buffer, but still call >> those to check if a drawer needs to be unfolded somewhere. > > 2. As you pointed, instead of global before-change-functions, we can use > modification-hooks text property on sensitive parts of the > drawers/blocks. This would work, but I am concerned about one annoying > special case: > > ------------------------------------------------------------------------- > :BLAH: <inserted outside any of the existing drawers> > > <some text> > > :DRAWER: <folded> > Donec at pede. > :END: > ------------------------------------------------------------------------- > In this example, the user would not be able to unfold the folder DRAWER > because it will technically become a part of a new giant BLAH drawer. > This may be especially annoying if <some text> is more than one screen > long and there is no easy way to identify why unfolding does not work > (with point at :DRAWER:). You shouldn't be bothered by the case you're describing here, for multiple reasons. First, this issue already arises in the current implementation. No one bothered so far: this change is very unlikely to happen. If it becomes an issue, we could make sure that `org-reveal' handles this. But, more importantly, we actually /want it/ as a feature. Indeed, if DRAWER is expanded every time ":BLAH:" is inserted above, then inserting a drawer manually would unfold /all/ drawers in the section. The user is more likely to write first ":BLAH:" (everything is unfolded) then ":END:" than ":END:", then ":BLAH:". > Because of this scenario, limiting before-change-functions to folded > drawers is not sufficient. Any change in text may need to trigger > unfolding. after-change-functions is more appropriate than before-change-functions, and local parsing, as explained in this thread, is more efficient than re-inventing the parser. > In the patch, I always register possible modifications in the > blocks/drawers intersecting with the modified region + a drawer/block > right next to the region. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the nested 'invisible text property implementation. > > The idea is to keep 'invisible property stack push and popping from it > as we add/remove 'invisible text property. All the work is done in > org-flag-region. This sounds like a good idea. > This was originally intended for folding outlines via text properties. > Since using text properties for folding outlines is not a good idea, > nested text properties have much less use. AFAIU, they have. You mention link fontification, but there are other pieces that we could switch to text properties instead of overlays, e.g., Babel hashes, narrowed table columns… > 3. Multiple calls to before/after-change-functions is still a problem. I > am looking into following ways to reduce this number: > - reduce the number of elements registered as potentially modified > + do not add duplicates to org--modified-elements > + do not add unfolded elements to org--modified-elements > + register after-change-function as post-command hook and remove it > from global after-change-functions. This way, it will be called > twice per command only. > - determine common region containing org--modified-elements. if change > is happening within that region, there is no need to parse > drawers/blocks there again. This is over-engineering. Again, please focus on local changes, as discussed before. > Recipe to have different (org-element-at-point) and > (org-element-parse-buffer 'element) > ------------------------------------------------------------------------- > <point-min> > :PROPERTIES: > :CREATED: [2020-05-23 Sat 02:32] > :END: > > > <point-max> > ------------------------------------------------------------------------- I didn't look at this situation in particular, but there are cases where different :post-blank values are inevitable, for example at the end of a section. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-05-26 8:33 ` Nicolas Goaziou @ 2020-06-02 9:21 ` Ihor Radchenko 2020-06-02 9:23 ` Ihor Radchenko ` (2 more replies) 0 siblings, 3 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-02 9:21 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Hello, [The patch itself will be provided in the following email] I have three updates from the previous version of the patch: 1. I managed to implement buffer-local text properties. Now, outline folding also uses text properties without a need to give up independent folding in indirect buffers. 2. The code handling modifications in folded drawers/blocks was rewritten. The new code uses after-change-functions to re-hide text inserted in the middle of folded regions; and text properties to unfold folded drawers/blocks if one changes BEGIN/END line. 3. [experimental] Started working on improving memory and cpu footprint of the old code related to folding/unfolding. org-hide-drawer-all now works significantly faster because I can utilise simplified drawer parser, which require a lot less memory. Overall, I managed to reduce Emacs memory footprint after loading all my agenda_files twice. The loading is also noticeably faster. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the buffer-local text properties: I have found char-property-alias-alist variable that controls how Emacs calculates text property value if the property is not set. This variable can be buffer-local, which allows independent 'invisible states in different buffers. All the implementation stays in org--get-buffer-local-text-property-symbol, which takes care about generating unique property name and mapping it to 'invisible (or any other) text property. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the new implementation for tracking changes: I simplified the code as suggested, without using pairs of before- and after-change-functions. Handling text inserted into folded/invisible region is handled by a simple after-change function. After testing, it turned out that simple re-hiding text based on 'invisible property of the text before/after the inserted region works pretty well. Modifications to BEGIN/END line of the drawers and blocks is handled via 'modification-hooks + 'insert-behind-hooks text properties (there is no after-change-functions analogue for text properties in Emacs). The property is applied during folding and the modification-hook function is made aware about the drawer/block boundaries (via apply-partially passing element containing :begin :end markers for the current drawer/block). Passing the element boundary is important because the 'modification-hook will not directly know where it belongs to. Only the modified region (which can be larger than the drawer) is passed to the function. In the worst case, the region can be the whole buffer (if one runs revert-buffer). It turned out that adding 'modification-hook text property takes a significant cpu time (partially, because we need to take care about possible existing 'modification-hook value, see org--add-to-list-text-property). For now, I decided to not clear the modification hooks during unfolding because of poor performance. However, this approach would lead to partial unfolding in the following case: :asd: :drawer: lksjdfksdfjl sdfsdfsdf :end: If :asd: was inserted in front of folded :drawer:, changes in :drawer: line of the new folded :asd: drawer would reveal the text between :drawer: and :end:. Let me know what you think on this. > You shouldn't be bothered by the case you're describing here, for > multiple reasons. > > First, this issue already arises in the current implementation. No one > bothered so far: this change is very unlikely to happen. If it becomes > an issue, we could make sure that `org-reveal' handles this. > > But, more importantly, we actually /want it/ as a feature. Indeed, if > DRAWER is expanded every time ":BLAH:" is inserted above, then inserting > a drawer manually would unfold /all/ drawers in the section. The user is > more likely to write first ":BLAH:" (everything is unfolded) then > ":END:" than ":END:", then ":BLAH:". Agree. This allowed me to simplify the code significantly. > It seems you're getting it backwards. `before-change-functions' are the > functions being called with a possibly wide, imprecise, region to > handle: > > When that happens, the arguments to ‘before-change-functions’ will > enclose a region in which the individual changes are made, but won’t > necessarily be the minimal such region > > however, after-change-functions calls are always minimal: > > and the arguments to each successive call of > ‘after-change-functions’ will then delimit the part of text being > changed exactly. > > If you stick to `after-change-functions', there will be no such thing as > you describe. You are right here, I missed that before-change-functions are likely to be called on large regions. I thought that the regions are same for before/after-change-functions, but after-change-functions could be called more than 1 time. After second thought, your vision that it is mostly 0 or 1 times should be the majority of cases in practice. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on reducing cpu and memory footprint of org buffers: My simplified implementation of element boundary parser (org--get-element-region-at-point) appears to be much faster and also uses much less memory in comparison with org-element-at-point. Moreover, not all the places where org-element-at-point is called actually need the full parsed element. For example, org-hide-drawer-all, org-hide-drawer-toggle, org-hide-block-toggle, and org--hide-wrapper-toggle only need element type and some information about the element boundaries - the information we can get from org--get-element-region-at-point. The following version of org-hide-drawer-all seems to work much faster in comparison with original: (defun org-hide-drawer-all () "Fold all drawers in the current buffer." (save-excursion (goto-char (point-min)) (while (re-search-forward org-drawer-regexp nil t) (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer))) (type (org-element-type drawer))) (org-hide-drawer-toggle t nil drawer) ;; Make sure to skip drawer entirely or we might flag it ;; another time when matching its ending line with ;; `org-drawer-regexp'. (goto-char (org-element-property :end drawer)))))) What do you think about the idea of making use of org--get-element-region-at-point in org code base? ----------------------------------------------------------------------- ----------------------------------------------------------------------- Further work: 1. Look into other code using overlays. Specifically, org-toggle-custom-properties, Babel hashes, and narrowed table columns. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> I have five updates from the previous version of the patch: > > Thank you. > >> 1. I implemented a simplified version of element parsing to detect >> changes in folded drawers or blocks. No computationally expensive calls >> of org-element-at-point or org-element-parse-buffer are needed now. >> >> 2. The patch is now compatible with master (commit 2e96dc639). I >> reverted the earlier change in folding drawers and blocks. Now, they are >> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would >> achieve nothing when we use text properties. >> >> 3. 'invisible text property can now be nested. This is important, for >> example, when text inside drawers contains fontified links (which also >> use 'invisible text property to hide parts of the link). Now, the old >> 'invisible spec is recovered after unfolding. > > Interesting. I'm running out of time, so I cannot properly inspect the > code right now. I'll try to do that before the end of the week. > >> 4. Some outline-* function calls in org referred to outline-flag-region >> implementation, which is not in sync with org-flag-region in this patch. >> I have implemented their org-* versions and replaced the calls >> throughout .el files. Actually, some org-* versions were already >> implemented in org, but not used for some reason (or not mentioned in >> the manual). I have updated the relevant sections of manual. These >> changes might be relevant to org independently of this feature branch. > > Yes, we certainly want to move to org-specific versions in all cases. > >> 5. I have managed to get a working version of outline folding via text >> properties. However, that approach has a big downside - folding state >> cannot be different in indirect buffer when we use text properties. I >> have seen packages relying on this feature of org and I do not see any >> obvious way to achieve different folding state in indirect buffer while >> using text properties for outline folding. > > Hmm. Good point. This is a serious issue to consider. Even if we don't > use text properties for outline, this also affects drawers and blocks. > >> For now, I still used before/after-change-functions combination. > > You shouldn't. > >> I see the following problems with using only after-change-functions: >> >> 1. They are not guaranteed to be called after every single change: > > Of course they are! See below. > >> From (elisp) Change Hooks: >> "... some complex primitives call ‘before-change-functions’ once before >> making changes, and then call ‘after-change-functions’ zero or more >> times" > > "zero" means there are no changes at all, so, `after-change-functions' > are not called, which is expected. > >> The consequence of it is a possibility that region passed to the >> after-change-functions is quite big (including all the singular changes, >> even if they are distant). This region may contain changed drawers as >> well and unchanged drawers and needs to be parsed to determine which >> drawers need to be re-folded. > > It seems you're getting it backwards. `before-change-functions' are the > functions being called with a possibly wide, imprecise, region to > handle: > > When that happens, the arguments to ‘before-change-functions’ will > enclose a region in which the individual changes are made, but won’t > necessarily be the minimal such region > > however, after-change-functions calls are always minimal: > > and the arguments to each successive call of > ‘after-change-functions’ will then delimit the part of text being > changed exactly. > > If you stick to `after-change-functions', there will be no such thing as > you describe. > >>> And, more importantly, they are not meant to be used together, i.e., you >>> cannot assume that a single call to `before-change-functions' always >>> happens before calling `after-change-functions'. This can be tricky if >>> you want to use the former to pass information to the latter. >> >> The fact that before-change-functions can be called multiple times >> before after-change-functions, is trivially solved by using buffer-local >> changes register (see org--modified-elements). > > Famous last words. Been there, done that, and it failed. > > Let me quote the manual: > > In general, we advise to use either before- or the after-change > hooks, but not both. > > So, let me insist: don't do that. If you don't agree with me, let's at > least agree with Emacs developers. > >> The register is populated by before-change-functions and cleared by >> after-change-functions. > > You cannot expect `after-change-functions' to clear what > `before-change-functions' did. This is likely to introduce pernicious > bugs. Sorry if it sounds like FUD, but bugs in those areas are just > horrible to squash. > >>> Well, `before-change-fuctions' and `after-change-functions' are not >>> clean at all: you modify an unrelated part of the buffer, but still call >>> those to check if a drawer needs to be unfolded somewhere. >> >> 2. As you pointed, instead of global before-change-functions, we can use >> modification-hooks text property on sensitive parts of the >> drawers/blocks. This would work, but I am concerned about one annoying >> special case: >> >> ------------------------------------------------------------------------- >> :BLAH: <inserted outside any of the existing drawers> >> >> <some text> >> >> :DRAWER: <folded> >> Donec at pede. >> :END: >> ------------------------------------------------------------------------- >> In this example, the user would not be able to unfold the folder DRAWER >> because it will technically become a part of a new giant BLAH drawer. >> This may be especially annoying if <some text> is more than one screen >> long and there is no easy way to identify why unfolding does not work >> (with point at :DRAWER:). > > You shouldn't be bothered by the case you're describing here, for > multiple reasons. > > First, this issue already arises in the current implementation. No one > bothered so far: this change is very unlikely to happen. If it becomes > an issue, we could make sure that `org-reveal' handles this. > > But, more importantly, we actually /want it/ as a feature. Indeed, if > DRAWER is expanded every time ":BLAH:" is inserted above, then inserting > a drawer manually would unfold /all/ drawers in the section. The user is > more likely to write first ":BLAH:" (everything is unfolded) then > ":END:" than ":END:", then ":BLAH:". > >> Because of this scenario, limiting before-change-functions to folded >> drawers is not sufficient. Any change in text may need to trigger >> unfolding. > > after-change-functions is more appropriate than before-change-functions, > and local parsing, as explained in this thread, is more efficient than > re-inventing the parser. > >> In the patch, I always register possible modifications in the >> blocks/drawers intersecting with the modified region + a drawer/block >> right next to the region. >> >> ----------------------------------------------------------------------- >> ----------------------------------------------------------------------- >> >> More details on the nested 'invisible text property implementation. >> >> The idea is to keep 'invisible property stack push and popping from it >> as we add/remove 'invisible text property. All the work is done in >> org-flag-region. > > This sounds like a good idea. > >> This was originally intended for folding outlines via text properties. >> Since using text properties for folding outlines is not a good idea, >> nested text properties have much less use. > > AFAIU, they have. You mention link fontification, but there are other > pieces that we could switch to text properties instead of overlays, > e.g., Babel hashes, narrowed table columns… > >> 3. Multiple calls to before/after-change-functions is still a problem. I >> am looking into following ways to reduce this number: >> - reduce the number of elements registered as potentially modified >> + do not add duplicates to org--modified-elements >> + do not add unfolded elements to org--modified-elements >> + register after-change-function as post-command hook and remove it >> from global after-change-functions. This way, it will be called >> twice per command only. >> - determine common region containing org--modified-elements. if change >> is happening within that region, there is no need to parse >> drawers/blocks there again. > > This is over-engineering. Again, please focus on local changes, as > discussed before. > >> Recipe to have different (org-element-at-point) and >> (org-element-parse-buffer 'element) >> ------------------------------------------------------------------------- >> <point-min> >> :PROPERTIES: >> :CREATED: [2020-05-23 Sat 02:32] >> :END: >> >> >> <point-max> >> ------------------------------------------------------------------------- > > I didn't look at this situation in particular, but there are cases where > different :post-blank values are inevitable, for example at the end of > a section. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 9:21 ` Ihor Radchenko @ 2020-06-02 9:23 ` Ihor Radchenko 2020-06-02 12:10 ` Bastien 2020-06-02 9:25 ` Ihor Radchenko 2020-06-05 7:26 ` Nicolas Goaziou 2 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-06-02 9:23 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 44 bytes --] The patch (against 758b039c0) is attached. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: featuredrawertextprop-20200602.patch --] [-- Type: text/x-diff, Size: 45456 bytes --] diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el index 9f8677871..ab470ea9b 100644 --- a/contrib/lisp/org-notify.el +++ b/contrib/lisp/org-notify.el @@ -246,7 +246,7 @@ seconds. The default value for SECS is 20." (switch-to-buffer (find-file-noselect file)) (org-with-wide-buffer (goto-char begin) - (outline-show-entry)) + (org-show-entry)) (goto-char begin) (search-forward "DEADLINE: <") (search-forward ":") diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el index bfc4d6c3e..2312b235c 100644 --- a/contrib/lisp/org-velocity.el +++ b/contrib/lisp/org-velocity.el @@ -325,7 +325,7 @@ use it." (save-excursion (when narrow (org-narrow-to-subtree)) - (outline-show-all))) + (org-show-all))) (defun org-velocity-edit-entry/inline (heading) "Edit entry at HEADING in the original buffer." diff --git a/doc/org-manual.org b/doc/org-manual.org index 92252179b..ff3e31abe 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and Switch back to the startup visibility of the buffer (see [[*Initial visibility]]). -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) :: +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) :: #+cindex: show all, command #+kindex: C-u C-u C-u TAB - #+findex: outline-show-all + #+findex: org-show-all Show all, including drawers. - {{{kbd(C-c C-r)}}} (~org-reveal~) :: @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and headings. With a double prefix argument, also show the entire subtree of the parent. -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) :: +- {{{kbd(C-c C-k)}}} (~org-show-branches~) :: #+cindex: show branches, command #+kindex: C-c C-k - #+findex: outline-show-branches + #+findex: org-show-branches Expose all the headings of the subtree, but not their bodies. -- {{{kbd(C-c TAB)}}} (~outline-show-children~) :: +- {{{kbd(C-c TAB)}}} (~org-show-children~) :: #+cindex: show children, command #+kindex: C-c TAB - #+findex: outline-show-children + #+findex: org-show-children Expose all direct children of the subtree. With a numeric prefix argument {{{var(N)}}}, expose all children down to level {{{var(N)}}}. @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way: command (see [[*Visibility Cycling]]). You can force cycling archived subtrees with {{{kbd(C-TAB)}}}, or by setting the option ~org-cycle-open-archived-trees~. Also normal outline commands, like - ~outline-show-all~, open archived subtrees. + ~org-show-all~, open archived subtrees. - #+vindex: org-sparse-tree-open-archived-trees diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index f07c3b801..a9c4d9eb2 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -6824,7 +6824,7 @@ and stored in the variable `org-prefix-format-compiled'." (t " %-12:c%?-12t% s"))) (start 0) varform vars var e c f opt) - (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)" + (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)" s start) (setq var (or (cdr (assoc (match-string 4 s) '(("c" . category) ("t" . time) ("l" . level) ("s" . extra) @@ -9136,20 +9136,20 @@ if it was hidden in the outline." ((and (called-interactively-p 'any) (= more 1)) (message "Remote: show with default settings")) ((= more 2) - (outline-show-entry) + (org-show-entry) (org-show-children) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'children)) (message "Remote: CHILDREN")) ((= more 3) - (outline-show-subtree) + (org-show-subtree) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'subtree)) (message "Remote: SUBTREE")) ((> more 3) - (outline-show-subtree) + (org-show-subtree) (message "Remote: SUBTREE AND ALL DRAWERS"))) (select-window win))) diff --git a/lisp/org-archive.el b/lisp/org-archive.el index d3e12d17b..d864dad8a 100644 --- a/lisp/org-archive.el +++ b/lisp/org-archive.el @@ -330,7 +330,7 @@ direct children of this heading." (insert (if datetree-date "" "\n") heading "\n") (end-of-line 0)) ;; Make the subtree visible - (outline-show-subtree) + (org-show-subtree) (if org-archive-reversed-order (progn (org-back-to-heading t) diff --git a/lisp/org-colview.el b/lisp/org-colview.el index e50a4d7c8..e656df555 100644 --- a/lisp/org-colview.el +++ b/lisp/org-colview.el @@ -699,7 +699,7 @@ FUN is a function called with no argument." (move-beginning-of-line 2) (org-at-heading-p t))))) (unwind-protect (funcall fun) - (when hide-body (outline-hide-entry))))) + (when hide-body (org-hide-entry))))) (defun org-columns-previous-allowed-value () "Switch to the previous allowed value for this column." diff --git a/lisp/org-compat.el b/lisp/org-compat.el index 635a38dcd..8fe271896 100644 --- a/lisp/org-compat.el +++ b/lisp/org-compat.el @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer." ;;; Emacs < 25.1 compatibility (when (< emacs-major-version 25) - (defalias 'outline-hide-entry 'hide-entry) - (defalias 'outline-hide-sublevels 'hide-sublevels) - (defalias 'outline-hide-subtree 'hide-subtree) (defalias 'outline-show-branches 'show-branches) (defalias 'outline-show-children 'show-children) - (defalias 'outline-show-entry 'show-entry) (defalias 'outline-show-subtree 'show-subtree) (defalias 'xref-find-definitions 'find-tag) (defalias 'format-message 'format) diff --git a/lisp/org-element.el b/lisp/org-element.el index ac41b7650..2d5c8d771 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -4320,7 +4320,7 @@ element or object. Meaningful values are `first-section', TYPE is the type of the current element or object. If PARENT? is non-nil, assume the next element or object will be -located inside the current one. " +located inside the current one." (if parent? (pcase type (`headline 'section) diff --git a/lisp/org-keys.el b/lisp/org-keys.el index 37df29983..a714dec0f 100644 --- a/lisp/org-keys.el +++ b/lisp/org-keys.el @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names." #'org-next-visible-heading) (define-key org-mode-map [remap outline-previous-visible-heading] #'org-previous-visible-heading) -(define-key org-mode-map [remap show-children] #'org-show-children) +(define-key org-mode-map [remap outline-show-children] #'org-show-children) ;;;; Make `C-c C-x' a prefix key (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap)) diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..681b5a404 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." \f -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,26 +705,138 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org-remove-text-properties (start end properties &optional object) + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. +Do not remove invisible text properties specified by 'outline, +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this +is needed to keep outlines, drawers, and blocks hidden unless they are +toggled by user. +Note: The below may be too specific and create troubles if more +invisibility specs are added to org in future" + (when (plist-member properties 'invisible) + (let ((pos start) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer + 'outline)) + (remove-text-properties pos next '(invisible nil) object)) + (setq pos next)))) + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) + (remove-text-properties start end properties-stripped object))) + +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + +(defun org--add-to-list-text-property (from to prop element) + "Add element to text property PROP, whos value should be a list." + (add-text-properties from to `(,prop ,(list element))) ; create if none + ;; add to existing + (alter-text-property from to + prop + (lambda (val) + (if (member element val) + val + (cons element val))))) + +(defun org--remove-from-list-text-property (from to prop element) + "Remove ELEMENT from text propery PROP, whos value should be a list." + (let ((pos from)) + (while (< pos to) + (when-let ((val (get-text-property pos prop))) + (if (equal val (list element)) + (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil)) + (put-text-property pos (next-single-char-property-change pos prop nil to) + prop (remove element (get-text-property pos prop))))) + (setq pos (next-single-char-property-change pos prop nil to))))) + +(defun org--get-buffer-local-text-property-symbol (prop &optional buffer) + "Compute unique symbol suitable to be used as buffer-local in BUFFER for PROP." + (let* ((buf (or buffer (current-buffer)))) + (let ((local-prop-string (format "org--%s-buffer-local-%S" (symbol-name prop) (sxhash buf)))) + (with-current-buffer buf + (unless (string-equal (symbol-name (car (alist-get prop char-property-alias-alist))) + local-prop-string) + (let ((local-prop (make-symbol local-prop-string))) + ;; copy old property + (when-let ((old-prop (car (alist-get prop char-property-alias-alist)))) + (org-with-wide-buffer + (let ((pos (point-min))) + (while (< pos (point-max)) + (when-let (val (get-text-property pos old-prop)) + (put-text-property pos (next-single-char-property-change pos old-prop) local-prop val)) + (setq pos (next-single-char-property-change pos old-prop)))))) + (setq-local char-property-alias-alist + (cons (list prop local-prop) + (remove (assq prop char-property-alias-alist) + char-property-alias-alist))))) + (car (alist-get prop char-property-alias-alist)))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + ;; Use text properties instead of overlays for speed. + ;; Overlays are too slow (Emacs Bug#35453). + (with-silent-modifications + ;; keep a backup stack of old text properties + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((old-spec (get-text-property (point) (org--get-buffer-local-text-property-symbol 'invisible))) + (end (next-single-property-change (point) (org--get-buffer-local-text-property-symbol 'invisible) nil to))) + (when old-spec + (alter-text-property (point) end (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible) + (lambda (stack) + (if (or (eq old-spec (car stack)) + (eq spec old-spec) + (eq old-spec 'outline)) + stack + (cons old-spec stack))))) + (goto-char end)))) + + ;; cleanup everything + (remove-text-properties from to (list (org--get-buffer-local-text-property-symbol 'invisible) nil)) + + ;; Recover properties from the backup stack + (unless flag + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((stack (get-text-property (point) (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible))) + (end (next-single-property-change (point) (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible) nil to))) + (if (not stack) + (remove-text-properties (point) end '(org-property-stack-invisible nil)) + (put-text-property (point) end (org--get-buffer-local-text-property-symbol 'invisible) (car stack)) + (alter-text-property (point) end (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible) + (lambda (stack) + (cdr stack)))) + (goto-char end))))) + + (when flag + (put-text-property from to (org--get-buffer-local-text-property-symbol 'invisible) spec)))) \f ;;; Regexp matching (defsubst org-pos-in-match-range (pos n) - (and (match-beginning n) - (<= (match-beginning n) pos) - (>= (match-end n) pos))) +(and (match-beginning n) + (<= (match-beginning n) pos) + (>= (match-end n) pos))) (defun org-skip-whitespace () "Skip over space, tabs and newline characters." diff --git a/lisp/org-src.el b/lisp/org-src.el index 6f6c544dc..9e8a50044 100644 --- a/lisp/org-src.el +++ b/lisp/org-src.el @@ -529,8 +529,8 @@ Leave point in edit buffer." (org-src-switch-to-buffer buffer 'edit) ;; Insert contents. (insert contents) - (remove-text-properties (point-min) (point-max) - '(display nil invisible nil intangible nil)) + (org-remove-text-properties (point-min) (point-max) + '(display nil invisible nil intangible nil)) (unless preserve-ind (org-do-remove-indentation)) (set-buffer-modified-p nil) (setq buffer-file-name nil) diff --git a/lisp/org-table.el b/lisp/org-table.el index 6462b99c4..75801161b 100644 --- a/lisp/org-table.el +++ b/lisp/org-table.el @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'." (arg (let ((b (save-excursion (skip-chars-backward "^|") (point))) (e (save-excursion (skip-chars-forward "^|\r\n") (point)))) - (remove-text-properties b e '(invisible t intangible t)) + (org-remove-text-properties b e '(invisible t intangible t)) (if (and (boundp 'font-lock-mode) font-lock-mode) (font-lock-fontify-block)))) (t @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'." (setq word-wrap t) (goto-char (setq p (point-max))) (insert (org-trim field)) - (remove-text-properties p (point-max) '(invisible t intangible t)) + (org-remove-text-properties p (point-max) '(invisible t intangible t)) (goto-char p) (setq-local org-finish-function 'org-table-finish-edit-field) (setq-local org-window-configuration cw) diff --git a/lisp/org.el b/lisp/org.el index f201138f1..6f5aa4b7e 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") (defvar ffap-url-regexp) (defvar org-element-paragraph-separate) +(defvar org-element-all-objects) +(defvar org-element-all-elements) +(defvar org-element-greater-elements) (defvar org-indent-indentation-per-level) (defvar org-radio-target-regexp) (defvar org-target-link-regexp) @@ -4734,9 +4738,174 @@ This is for getting out of special buffers like capture.") ;;;; Define the Org mode +;;; Handling buffer modifications + (defun org-before-change-function (_beg _end) "Every change indicates that a table might need an update." (setq org-table-may-need-update t)) + +(defun org-after-change-function (from to len) + "Hide text in region if it follows and is followedby invisible text." + (when-let ((spec-to (get-text-property to 'invisible)) + (spec-from (get-text-property (max (point-min) (1- from)) 'invisible))) + (when (eq spec-to spec-from) + (org-flag-region from to 't spec-to)))) + + +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$") + (property-drawer . ,org-property-start-re) + (drawer . ,org-drawer-regexp) + (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$") + (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$")) + "Alist of regexps matching beginning of elements. +Group 1 in the regexps (if any) contains the element type.") + +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$") + (property-drawer . ,org-property-end-re) + (drawer . ,org-property-end-re) + (quote-block . "^[ \t]*#\\+end_quote[ \t]*$") + (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$")) + "Alist of regexps matching end of elements. +Group 1 in the regexps (if any) contains the element type or END.") + +(defvar org-track-element-modifications + `(property-drawer + drawer + center-block + quote-block + special-block) + "Alist of elements to be tracked for modifications. +The modification is only triggered when beginning/end line of the element is modified.") + +(defun org--get-element-region-at-point (types) + "Return TYPES element at point or nil. +If TYPES is a list, return first element at point from the list. The +returned value is partially parsed element only containing :begin and +:end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (begin-limit (save-excursion (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok)) + (point))) + (end-limit (or (save-excursion (outline-next-heading)) + (point-max))) + (point (point)) + begin end) + (when (and begin-re end-re) + (save-excursion + (end-of-line) + (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point))) + (when (re-search-forward end-re end-limit 'noerror) (setq end (point))) + ;; slurp unmatched begin-re + (when (and begin end) + (goto-char begin) + (while (and (re-search-backward begin-re begin-limit 'noerror) + (= end (save-excursion (re-search-forward end-re end-limit 'noerror)))) + (setq begin (point))) + (when (and (>= point begin) (<= point end)) + (throw 'exit + (let ((begin (copy-marker begin 't)) + (end (copy-marker end nil))) + (list type + (list + :begin begin + :post-affiliated begin + :contents-begin (save-excursion (goto-char begin) (copy-marker (1+ (line-end-position)) + 't)) + :contents-end (save-excursion (goto-char end) (copy-marker (1- (line-beginning-position)) + nil)) + :end end)))))))))))) + +(defun org--get-next-element-region-at-point (types &optional limit previous) + "Return TYPES element after point or nil. +If TYPES is a list, return first element after point from the list. +If PREVIOUS is non-nil, return first TYPES element before point. +Limit search by LIMIT or previous/next heading. +The returned value is partially parsed element only containing :begin +and :end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let* ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (limit (or limit (if previous + (save-excursion + (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok) + (point))) + (or (save-excursion (outline-next-heading)) + (point-max))))) + el) + (when (and begin-re end-re) + (save-excursion + (if previous + (when (re-search-backward begin-re limit 'noerror) + (setq el (org--get-element-region-at-point type))) + (when (re-search-forward begin-re limit 'noerror) + (setq el (org--get-element-region-at-point type))))) + (when el + (throw 'exit + el))))))) + +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours) + "Find all elements from ELEMENTS in region BEG . END. +All the listed elements must be resolvable by +`org--get-element-region-at-point'. +Include elements if they are partially inside region when +INCLUDE-PARTIAL is non-nil. +Include preceding/subsequent neighbouring elements when no partial +element is found at the beginning/end of the region and +INCLUDE-NEIGHBOURS is non-nil." + (when include-partial + (org-with-point-at beg + (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements)))) + (if new-beg + (setq beg new-beg) + (when (and include-neighbours + (setq new-beg (org-element-property :begin + (org--get-next-element-region-at-point elements + (point-min) + 'previous)))) + (setq beg new-beg)))) + (when (memq 'headline elements) + (when-let ((new-beg (save-excursion + (org-with-limited-levels (outline-previous-heading))))) + (setq beg new-beg)))) + (org-with-point-at end + (let ((new-end (org-element-property :end (org--get-element-region-at-point elements)))) + (if new-end + (setq end new-end) + (when (and include-neighbours + (setq new-end (org-element-property :end + (org--get-next-element-region-at-point elements (point-max))))) + (setq end new-end)))) + (when (memq 'headline elements) + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) + (setq end (1- new-end)))))) + (save-excursion + (save-restriction + (narrow-to-region beg end) + (goto-char (point-min)) + (let (result el) + (while (setq el (org--get-next-element-region-at-point elements end)) + (push el result) + (goto-char (org-element-property :end el))) + result)))) + +(defun org--unfold-elements-in-region (el &rest _) + "Unfold EL element." + (when-let ((category (if (string-match-p "block" (symbol-name (org-element-type el))) + 'block + (when (string-match-p "drawer" (symbol-name (org-element-type el))) + 'drawer)))) + (org-with-point-at (org-element-property :begin el) + (org--hide-wrapper-toggle el category 'off nil)))) + (defvar org-mode-map) (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. @@ -4818,6 +4987,8 @@ The following commands are available: ;; Activate before-change-function (setq-local org-table-may-need-update t) (add-hook 'before-change-functions 'org-before-change-function nil 'local) + ;; Activate after-change-function + (add-hook 'after-change-functions 'org-after-change-function nil 'local) ;; Check for running clock before killing a buffer (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) ;; Initialize macros templates. @@ -4869,6 +5040,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5050,8 +5225,8 @@ stacked delimiters is N. Escaping delimiters is not possible." (when verbatim? (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 2) (match-end 2) - '(display t invisible t intangible t))) + (org-remove-text-properties (match-beginning 2) (match-end 2) + '(display t invisible t intangible t))) (add-text-properties (match-beginning 2) (match-end 2) '(font-lock-multiline t org-emphasis t)) (when (and org-hide-emphasis-markers @@ -5166,7 +5341,7 @@ This includes angle, plain, and bracket links." (if (not (eq 'bracket style)) (add-text-properties start end properties) ;; Handle invisible parts in bracket links. - (remove-text-properties start end '(invisible nil)) + (org-remove-text-properties start end '(invisible nil)) (let ((hidden (append `(invisible ,(or (org-link-get-parameter type :display) @@ -5186,8 +5361,8 @@ This includes angle, plain, and bracket links." (defun org-activate-code (limit) (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) t)) (defcustom org-src-fontify-natively t @@ -5258,8 +5433,8 @@ by a #." (setq block-end (match-beginning 0)) ; includes the final newline. (when quoting (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline) - (remove-text-properties beg end-of-endline - '(display t invisible t intangible t))) + (org-remove-text-properties beg end-of-endline + '(display t invisible t intangible t))) (add-text-properties beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) (org-remove-flyspell-overlays-in beg bol-after-beginline) @@ -5313,9 +5488,9 @@ by a #." '(font-lock-fontified t face org-document-info)))) ((string-prefix-p "+caption" dc1) (org-remove-flyspell-overlays-in (match-end 2) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) - ;; Handle short captions + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) + ;; Handle short captions. (save-excursion (beginning-of-line) (looking-at (rx (group (zero-or-more blank) @@ -5336,8 +5511,8 @@ by a #." '(font-lock-fontified t face font-lock-comment-face))) (t ;; Just any other in-buffer setting, but not indented (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) (add-text-properties beg (match-end 0) '(font-lock-fontified t face org-meta-line)) t)))))) @@ -5859,10 +6034,11 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) - (remove-text-properties beg end - '(mouse-face t keymap t org-linked-text t - invisible t intangible t - org-emphasis t)) + (org-remove-text-properties beg end + '(mouse-face t keymap t org-linked-text t + invisible t + intangible t + org-emphasis t)) (org-remove-font-lock-display-properties beg end))) (defconst org-script-display '(((raise -0.3) (height 0.7)) @@ -5970,6 +6146,29 @@ open and agenda-wise Org files." ;;;; Headlines visibility +(defun org-hide-entry () + "Hide the body directly following this heading." + (interactive) + (save-excursion + (outline-back-to-heading) + (outline-end-of-heading) + (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline))) + +(defun org-hide-subtree () + "Hide everything after this heading at deeper levels." + (interactive) + (org-flag-subtree t)) + +(defun org-hide-sublevels (levels) + "Hide everything but the top LEVELS levels of headers, in whole buffer. +This also unhides the top heading-less body, if any. + +Interactively, the prefix argument supplies the value of LEVELS. +When invoked without a prefix argument, LEVELS defaults to the level +of the current heading, or to 1 if the current line is not a heading." + (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region)) + (org-hide-sublevels levels))) + (defun org-show-entry () "Show the body directly following this heading. Show the heading too, if it is currently invisible." @@ -5988,6 +6187,17 @@ Show the heading too, if it is currently invisible." 'outline) (org-cycle-hide-property-drawers 'children)))) +(defun org-show-heading () + "Show the current heading and move to its end." + (org-flag-region (- (point) + (if (bobp) 0 + (if (and outline-blank-line + (eq (char-before (1- (point))) ?\n)) + 2 1))) + (progn (outline-end-of-heading) (point)) + nil + 'outline)) + (defun org-show-children (&optional level) "Show all direct subheadings of this heading. Prefix arg LEVEL is how many levels below the current level @@ -6031,6 +6241,11 @@ heading to appear." (org-flag-region (point) (save-excursion (org-end-of-subtree t t)) nil 'outline)) +(defun org-show-branches () + "Show all subheadings of this heading, but not their bodies." + (interactive) + (org-show-children 1000)) + ;;;; Blocks and drawers visibility (defun org--hide-wrapper-toggle (element category force no-error) @@ -6064,13 +6279,39 @@ Return a non-nil value when toggling is successful." (unless (let ((eol (line-end-position))) (and (> eol start) (/= eol end))) (let* ((spec (cond ((eq category 'block) 'org-hide-block) - ((eq type 'property-drawer) 'outline) - (t 'org-hide-drawer))) + ((eq category 'drawer) 'org-hide-drawer) + (t 'outline))) (flag (cond ((eq force 'off) nil) (force t) ((eq (get-char-property start 'invisible) spec) nil) (t t)))) + ;; Make beginning/end of blocks sensitive to modifications + ;; we never remove the hooks because modification of parts + ;; of blocks is practically more rare in comparison with + ;; folding/unfolding. Removing modification hooks would + ;; cost more CPU time. + (when flag + (with-silent-modifications + (let ((el (org--get-element-region-at-point + (org-element-type element)))) + (unless (member (apply-partially #'org--unfold-elements-in-region el) + (get-text-property (org-element-property :begin element) + 'modification-hooks)) + ;; first line + (org--add-to-list-text-property (org-element-property :begin element) start + 'modification-hooks + (apply-partially #'org--unfold-elements-in-region el)) + (org--add-to-list-text-property (org-element-property :begin element) start + 'insert-behind-hooks + (apply-partially #'org--unfold-elements-in-region el)) + ;; last line + (org--add-to-list-text-property (save-excursion (goto-char end) (line-beginning-position)) end + 'modification-hooks + (apply-partially #'org--unfold-elements-in-region el)) + (org--add-to-list-text-property (save-excursion (goto-char end) (line-beginning-position)) end + 'insert-behind-hooks + (apply-partially #'org--unfold-elements-in-region el)))))) (org-flag-region start end flag spec)) ;; When the block is hidden away, make sure point is left in ;; a visible part of the buffer. @@ -6118,24 +6359,16 @@ Return a non-nil value when toggling is successful." (defun org-hide-drawer-all () "Fold all drawers in the current buffer." - (org-show-all '(drawers)) (save-excursion (goto-char (point-min)) (while (re-search-forward org-drawer-regexp nil t) - (let* ((drawer (org-element-at-point)) - (type (org-element-type drawer))) - (when (memq type '(drawer property-drawer)) - ;; We are sure regular drawers are unfolded because of - ;; `org-show-all' call above. However, property drawers may - ;; be folded, or in a folded headline. In that case, do not - ;; re-hide it. - (unless (and (eq type 'property-drawer) - (eq 'outline (get-char-property (point) 'invisible))) - (org-hide-drawer-toggle t nil drawer)) - ;; Make sure to skip drawer entirely or we might flag it - ;; another time when matching its ending line with - ;; `org-drawer-regexp'. - (goto-char (org-element-property :end drawer))))))) + (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer))) + (type (org-element-type drawer))) + (org-hide-drawer-toggle t nil drawer) + ;; Make sure to skip drawer entirely or we might flag it + ;; another time when matching its ending line with + ;; `org-drawer-regexp'. + (goto-char (org-element-property :end drawer)))))) (defun org-cycle-hide-property-drawers (state) "Re-hide all drawers after a visibility state change. @@ -6150,18 +6383,16 @@ STATE should be one of the symbols listed in the docstring of (t (save-excursion (org-end-of-subtree t)))))) (org-with-point-at beg (while (re-search-forward org-property-start-re end t) - (pcase (get-char-property-and-overlay (point) 'invisible) + (pcase (get-char-property (point) 'invisible) ;; Do not fold already folded drawers. - (`(outline . ,o) (goto-char (overlay-end o))) + ('outline + (goto-char (min end (next-single-char-property-change (point) 'invisible)))) (_ (let ((start (match-end 0))) (when (org-at-property-drawer-p) (let* ((case-fold-search t) (end (re-search-forward org-property-end-re))) - ;; Property drawers use `outline' invisibility spec - ;; so they can be swallowed once we hide the - ;; outline. - (org-flag-region start end t 'outline))))))))))) + (org-flag-region start end t 'org-hide-drawer))))))))))) ;;;; Visibility cycling @@ -6536,7 +6767,7 @@ With a numeric prefix, show all headlines up to that level." (org-narrow-to-subtree) (org-content)))) ((or "all" "showall") - (outline-show-subtree)) + (org-show-subtree)) (_ nil))) (org-end-of-subtree))))))) @@ -6609,7 +6840,7 @@ This function is the default value of the hook `org-cycle-hook'." (while (re-search-forward re nil t) (when (and (not (org-invisible-p)) (org-invisible-p (line-end-position))) - (outline-hide-entry)))) + (org-hide-entry)))) (org-cycle-hide-property-drawers 'all) (org-cycle-show-empty-lines 'overview))))) @@ -6681,10 +6912,11 @@ information." (org-show-entry) ;; If point is hidden within a drawer or a block, make sure to ;; expose it. - (dolist (o (overlays-at (point))) - (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) - (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -6900,9 +7132,10 @@ unconditionally." ;; When INVISIBLE-OK is non-nil, ensure newly created headline ;; is visible. (unless invisible-ok - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (move-overlay o (overlay-start o) (line-end-position 0))) + (pcase (get-char-property (point) 'invisible) + ('outline + (let ((region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (line-end-position 0) (cdr region) nil 'outline))) (_ nil)))) ;; At a headline... ((org-at-heading-p) @@ -7499,7 +7732,6 @@ case." (setq txt (buffer-substring beg end)) (org-save-markers-in-region beg end) (delete-region beg end) - (org-remove-empty-overlays-at beg) (unless (= beg (point-min)) (org-flag-region (1- beg) beg nil 'outline)) (unless (bobp) (org-flag-region (1- (point)) (point) nil 'outline)) (and (not (bolp)) (looking-at "\n") (forward-char 1)) @@ -7661,7 +7893,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard." (skip-chars-forward " \t\n\r") (setq beg (point)) (when (and (org-invisible-p) visp) - (save-excursion (outline-show-heading))) + (save-excursion (org-show-heading))) ;; Shift if necessary. (unless (= shift 0) (save-restriction @@ -8103,7 +8335,7 @@ function is being called interactively." (point)) what "children") (goto-char start) - (outline-show-subtree) + (org-show-subtree) (outline-next-heading)) (t ;; we will sort the top-level entries in this file @@ -13158,7 +13390,7 @@ drawer is immediately hidden." (inhibit-read-only t)) (unless (bobp) (insert "\n")) (insert ":PROPERTIES:\n:END:") - (org-flag-region (line-end-position 0) (point) t 'outline) + (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer) (when (or (eobp) (= begin (point-min))) (insert "\n")) (org-indent-region begin (point)))))) @@ -17621,11 +17853,11 @@ Move point to the beginning of first heading or end of buffer." (defun org-show-branches-buffer () "Show all branches in the buffer." (org-flag-above-first-heading) - (outline-hide-sublevels 1) + (org-hide-sublevels 1) (unless (eobp) - (outline-show-branches) + (org-show-branches) (while (outline-get-next-sibling) - (outline-show-branches))) + (org-show-branches))) (goto-char (point-min))) (defun org-kill-note-or-show-branches () @@ -17639,8 +17871,8 @@ Move point to the beginning of first heading or end of buffer." (t (let ((beg (progn (org-back-to-heading) (point))) (end (save-excursion (org-end-of-subtree t t) (point)))) - (outline-hide-subtree) - (outline-show-branches) + (org-hide-subtree) + (org-show-branches) (org-hide-archived-subtrees beg end))))) (defun org-delete-indentation (&optional arg) @@ -17796,9 +18028,9 @@ Otherwise, call `org-show-children'. ARG is the level to hide." (if (org-before-first-heading-p) (progn (org-flag-above-first-heading) - (outline-hide-sublevels (or arg 1)) + (org-hide-sublevels (or arg 1)) (goto-char (point-min))) - (outline-hide-subtree) + (org-hide-subtree) (org-show-children arg)))) (defun org-ctrl-c-star () @@ -20475,17 +20707,17 @@ With ARG, repeats or can move backward if negative." (end-of-line)) (while (and (< arg 0) (re-search-backward regexp nil :move)) (unless (bobp) - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (goto-char (overlay-start o)) + (pcase (get-char-property (point) 'invisible) + ('outline + (goto-char (car (org--find-text-property-region (point) 'invisible))) (beginning-of-line)) (_ nil))) (cl-incf arg)) - (while (and (> arg 0) (re-search-forward regexp nil t)) - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (goto-char (overlay-end o)) - (skip-chars-forward " \t\n") + (while (and (> arg 0) (re-search-forward regexp nil :move)) + (pcase (get-char-property (point) 'invisible) + ('outline + (goto-char (cdr (org--find-text-property-region (point) 'invisible))) + (skip-chars-forward " \t\n") (end-of-line)) (_ (end-of-line))) @@ -20943,6 +21175,80 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) \f + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + (setq spec (get-text-property pos 'invisible)) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (org-flag-region (car region) (cdr region) nil spec))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (org-flag-region (overlay-start ov) (overlay-end ov) t spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + +\f + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode [-- Attachment #3: Type: text/plain, Size: 17692 bytes --] Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > > [The patch itself will be provided in the following email] > > I have three updates from the previous version of the patch: > > 1. I managed to implement buffer-local text properties. > Now, outline folding also uses text properties without a need to give > up independent folding in indirect buffers. > > 2. The code handling modifications in folded drawers/blocks was > rewritten. The new code uses after-change-functions to re-hide text > inserted in the middle of folded regions; and text properties to > unfold folded drawers/blocks if one changes BEGIN/END line. > > 3. [experimental] Started working on improving memory and cpu footprint > of the old code related to folding/unfolding. org-hide-drawer-all now > works significantly faster because I can utilise simplified drawer > parser, which require a lot less memory. Overall, I managed to reduce > Emacs memory footprint after loading all my agenda_files twice. The > loading is also noticeably faster. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the buffer-local text properties: > > I have found char-property-alias-alist variable that controls how Emacs > calculates text property value if the property is not set. This variable > can be buffer-local, which allows independent 'invisible states in > different buffers. > > All the implementation stays in > org--get-buffer-local-text-property-symbol, which takes care about > generating unique property name and mapping it to 'invisible (or any > other) text property. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > > I simplified the code as suggested, without using pairs of before- and > after-change-functions. > > Handling text inserted into folded/invisible region is handled by a > simple after-change function. After testing, it turned out that simple > re-hiding text based on 'invisible property of the text before/after the > inserted region works pretty well. > > Modifications to BEGIN/END line of the drawers and blocks is handled via > 'modification-hooks + 'insert-behind-hooks text properties (there is no > after-change-functions analogue for text properties in Emacs). The > property is applied during folding and the modification-hook function is > made aware about the drawer/block boundaries (via apply-partially > passing element containing :begin :end markers for the current > drawer/block). Passing the element boundary is important because the > 'modification-hook will not directly know where it belongs to. Only the > modified region (which can be larger than the drawer) is passed to the > function. In the worst case, the region can be the whole buffer (if one > runs revert-buffer). > > It turned out that adding 'modification-hook text property takes a > significant cpu time (partially, because we need to take care about > possible existing 'modification-hook value, see > org--add-to-list-text-property). For now, I decided to not clear the > modification hooks during unfolding because of poor performance. > However, this approach would lead to partial unfolding in the following > case: > > :asd: > :drawer: > lksjdfksdfjl > sdfsdfsdf > :end: > > If :asd: was inserted in front of folded :drawer:, changes in :drawer: > line of the new folded :asd: drawer would reveal the text between > :drawer: and :end:. > > Let me know what you think on this. > >> You shouldn't be bothered by the case you're describing here, for >> multiple reasons. >> >> First, this issue already arises in the current implementation. No one >> bothered so far: this change is very unlikely to happen. If it becomes >> an issue, we could make sure that `org-reveal' handles this. >> >> But, more importantly, we actually /want it/ as a feature. Indeed, if >> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting >> a drawer manually would unfold /all/ drawers in the section. The user is >> more likely to write first ":BLAH:" (everything is unfolded) then >> ":END:" than ":END:", then ":BLAH:". > > Agree. This allowed me to simplify the code significantly. > >> It seems you're getting it backwards. `before-change-functions' are the >> functions being called with a possibly wide, imprecise, region to >> handle: >> >> When that happens, the arguments to ‘before-change-functions’ will >> enclose a region in which the individual changes are made, but won’t >> necessarily be the minimal such region >> >> however, after-change-functions calls are always minimal: >> >> and the arguments to each successive call of >> ‘after-change-functions’ will then delimit the part of text being >> changed exactly. >> >> If you stick to `after-change-functions', there will be no such thing as >> you describe. > > You are right here, I missed that before-change-functions are likely to > be called on large regions. I thought that the regions are same for > before/after-change-functions, but after-change-functions could be > called more than 1 time. After second thought, your vision that it is > mostly 0 or 1 times should be the majority of cases in practice. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on reducing cpu and memory footprint of org buffers: > > My simplified implementation of element boundary parser > (org--get-element-region-at-point) appears to be much faster and also > uses much less memory in comparison with org-element-at-point. > Moreover, not all the places where org-element-at-point is called > actually need the full parsed element. For example, org-hide-drawer-all, > org-hide-drawer-toggle, org-hide-block-toggle, and > org--hide-wrapper-toggle only need element type and some information > about the element boundaries - the information we can get from > org--get-element-region-at-point. > > The following version of org-hide-drawer-all seems to work much faster > in comparison with original: > > (defun org-hide-drawer-all () > "Fold all drawers in the current buffer." > (save-excursion > (goto-char (point-min)) > (while (re-search-forward org-drawer-regexp nil t) > (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer))) > (type (org-element-type drawer))) > (org-hide-drawer-toggle t nil drawer) > ;; Make sure to skip drawer entirely or we might flag it > ;; another time when matching its ending line with > ;; `org-drawer-regexp'. > (goto-char (org-element-property :end drawer)))))) > > What do you think about the idea of making use of > org--get-element-region-at-point in org code base? > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > 1. Look into other code using overlays. Specifically, > org-toggle-custom-properties, Babel hashes, and narrowed table columns. > > Best, > Ihor > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Hello, >> >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>> I have five updates from the previous version of the patch: >> >> Thank you. >> >>> 1. I implemented a simplified version of element parsing to detect >>> changes in folded drawers or blocks. No computationally expensive calls >>> of org-element-at-point or org-element-parse-buffer are needed now. >>> >>> 2. The patch is now compatible with master (commit 2e96dc639). I >>> reverted the earlier change in folding drawers and blocks. Now, they are >>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would >>> achieve nothing when we use text properties. >>> >>> 3. 'invisible text property can now be nested. This is important, for >>> example, when text inside drawers contains fontified links (which also >>> use 'invisible text property to hide parts of the link). Now, the old >>> 'invisible spec is recovered after unfolding. >> >> Interesting. I'm running out of time, so I cannot properly inspect the >> code right now. I'll try to do that before the end of the week. >> >>> 4. Some outline-* function calls in org referred to outline-flag-region >>> implementation, which is not in sync with org-flag-region in this patch. >>> I have implemented their org-* versions and replaced the calls >>> throughout .el files. Actually, some org-* versions were already >>> implemented in org, but not used for some reason (or not mentioned in >>> the manual). I have updated the relevant sections of manual. These >>> changes might be relevant to org independently of this feature branch. >> >> Yes, we certainly want to move to org-specific versions in all cases. >> >>> 5. I have managed to get a working version of outline folding via text >>> properties. However, that approach has a big downside - folding state >>> cannot be different in indirect buffer when we use text properties. I >>> have seen packages relying on this feature of org and I do not see any >>> obvious way to achieve different folding state in indirect buffer while >>> using text properties for outline folding. >> >> Hmm. Good point. This is a serious issue to consider. Even if we don't >> use text properties for outline, this also affects drawers and blocks. >> >>> For now, I still used before/after-change-functions combination. >> >> You shouldn't. >> >>> I see the following problems with using only after-change-functions: >>> >>> 1. They are not guaranteed to be called after every single change: >> >> Of course they are! See below. >> >>> From (elisp) Change Hooks: >>> "... some complex primitives call ‘before-change-functions’ once before >>> making changes, and then call ‘after-change-functions’ zero or more >>> times" >> >> "zero" means there are no changes at all, so, `after-change-functions' >> are not called, which is expected. >> >>> The consequence of it is a possibility that region passed to the >>> after-change-functions is quite big (including all the singular changes, >>> even if they are distant). This region may contain changed drawers as >>> well and unchanged drawers and needs to be parsed to determine which >>> drawers need to be re-folded. >> >> It seems you're getting it backwards. `before-change-functions' are the >> functions being called with a possibly wide, imprecise, region to >> handle: >> >> When that happens, the arguments to ‘before-change-functions’ will >> enclose a region in which the individual changes are made, but won’t >> necessarily be the minimal such region >> >> however, after-change-functions calls are always minimal: >> >> and the arguments to each successive call of >> ‘after-change-functions’ will then delimit the part of text being >> changed exactly. >> >> If you stick to `after-change-functions', there will be no such thing as >> you describe. >> >>>> And, more importantly, they are not meant to be used together, i.e., you >>>> cannot assume that a single call to `before-change-functions' always >>>> happens before calling `after-change-functions'. This can be tricky if >>>> you want to use the former to pass information to the latter. >>> >>> The fact that before-change-functions can be called multiple times >>> before after-change-functions, is trivially solved by using buffer-local >>> changes register (see org--modified-elements). >> >> Famous last words. Been there, done that, and it failed. >> >> Let me quote the manual: >> >> In general, we advise to use either before- or the after-change >> hooks, but not both. >> >> So, let me insist: don't do that. If you don't agree with me, let's at >> least agree with Emacs developers. >> >>> The register is populated by before-change-functions and cleared by >>> after-change-functions. >> >> You cannot expect `after-change-functions' to clear what >> `before-change-functions' did. This is likely to introduce pernicious >> bugs. Sorry if it sounds like FUD, but bugs in those areas are just >> horrible to squash. >> >>>> Well, `before-change-fuctions' and `after-change-functions' are not >>>> clean at all: you modify an unrelated part of the buffer, but still call >>>> those to check if a drawer needs to be unfolded somewhere. >>> >>> 2. As you pointed, instead of global before-change-functions, we can use >>> modification-hooks text property on sensitive parts of the >>> drawers/blocks. This would work, but I am concerned about one annoying >>> special case: >>> >>> ------------------------------------------------------------------------- >>> :BLAH: <inserted outside any of the existing drawers> >>> >>> <some text> >>> >>> :DRAWER: <folded> >>> Donec at pede. >>> :END: >>> ------------------------------------------------------------------------- >>> In this example, the user would not be able to unfold the folder DRAWER >>> because it will technically become a part of a new giant BLAH drawer. >>> This may be especially annoying if <some text> is more than one screen >>> long and there is no easy way to identify why unfolding does not work >>> (with point at :DRAWER:). >> >> You shouldn't be bothered by the case you're describing here, for >> multiple reasons. >> >> First, this issue already arises in the current implementation. No one >> bothered so far: this change is very unlikely to happen. If it becomes >> an issue, we could make sure that `org-reveal' handles this. >> >> But, more importantly, we actually /want it/ as a feature. Indeed, if >> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting >> a drawer manually would unfold /all/ drawers in the section. The user is >> more likely to write first ":BLAH:" (everything is unfolded) then >> ":END:" than ":END:", then ":BLAH:". >> >>> Because of this scenario, limiting before-change-functions to folded >>> drawers is not sufficient. Any change in text may need to trigger >>> unfolding. >> >> after-change-functions is more appropriate than before-change-functions, >> and local parsing, as explained in this thread, is more efficient than >> re-inventing the parser. >> >>> In the patch, I always register possible modifications in the >>> blocks/drawers intersecting with the modified region + a drawer/block >>> right next to the region. >>> >>> ----------------------------------------------------------------------- >>> ----------------------------------------------------------------------- >>> >>> More details on the nested 'invisible text property implementation. >>> >>> The idea is to keep 'invisible property stack push and popping from it >>> as we add/remove 'invisible text property. All the work is done in >>> org-flag-region. >> >> This sounds like a good idea. >> >>> This was originally intended for folding outlines via text properties. >>> Since using text properties for folding outlines is not a good idea, >>> nested text properties have much less use. >> >> AFAIU, they have. You mention link fontification, but there are other >> pieces that we could switch to text properties instead of overlays, >> e.g., Babel hashes, narrowed table columns… >> >>> 3. Multiple calls to before/after-change-functions is still a problem. I >>> am looking into following ways to reduce this number: >>> - reduce the number of elements registered as potentially modified >>> + do not add duplicates to org--modified-elements >>> + do not add unfolded elements to org--modified-elements >>> + register after-change-function as post-command hook and remove it >>> from global after-change-functions. This way, it will be called >>> twice per command only. >>> - determine common region containing org--modified-elements. if change >>> is happening within that region, there is no need to parse >>> drawers/blocks there again. >> >> This is over-engineering. Again, please focus on local changes, as >> discussed before. >> >>> Recipe to have different (org-element-at-point) and >>> (org-element-parse-buffer 'element) >>> ------------------------------------------------------------------------- >>> <point-min> >>> :PROPERTIES: >>> :CREATED: [2020-05-23 Sat 02:32] >>> :END: >>> >>> >>> <point-max> >>> ------------------------------------------------------------------------- >> >> I didn't look at this situation in particular, but there are cases where >> different :post-blank values are inevitable, for example at the end of >> a section. >> >> Regards, >> >> -- >> Nicolas Goaziou > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply related [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 9:23 ` Ihor Radchenko @ 2020-06-02 12:10 ` Bastien 2020-06-02 13:12 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Bastien @ 2020-06-02 12:10 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Nicolas Goaziou Hi Ihor, Ihor Radchenko <yantar92@gmail.com> writes: > The patch (against 758b039c0) is attached. Thanks -- just a quick note, in case you missed the message: we are in feature free for core functionalities, so we have time to work on this welcome enhancement for Org 9.5, which will give us time to properly test it too. -- Bastien ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 12:10 ` Bastien @ 2020-06-02 13:12 ` Ihor Radchenko 2020-06-02 13:23 ` Bastien 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-06-02 13:12 UTC (permalink / raw) To: Bastien; +Cc: emacs-orgmode, Nicolas Goaziou > Thanks -- just a quick note, in case you missed the message: we are in > feature free for core functionalities, so we have time to work on this > welcome enhancement for Org 9.5, which will give us time to properly > test it too. I do not expect it to be merged any time soon. The patch is modifying low-level internals. It certainly needs a careful testing under various user configs. Not to mention that so big patch will require FSF paperwork, unless I miss something. Best, Ihor Bastien <bzg@gnu.org> writes: > Hi Ihor, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> The patch (against 758b039c0) is attached. > > Thanks -- just a quick note, in case you missed the message: we are in > feature free for core functionalities, so we have time to work on this > welcome enhancement for Org 9.5, which will give us time to properly > test it too. > > -- > Bastien -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 13:12 ` Ihor Radchenko @ 2020-06-02 13:23 ` Bastien 2020-06-02 13:30 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Bastien @ 2020-06-02 13:23 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Nicolas Goaziou Hi Ihor, Ihor Radchenko <yantar92@gmail.com> writes: >> Thanks -- just a quick note, in case you missed the message: we are in >> feature free for core functionalities, so we have time to work on this >> welcome enhancement for Org 9.5, which will give us time to properly >> test it too. > > I do not expect it to be merged any time soon. The patch is modifying > low-level internals. It certainly needs a careful testing under various > user configs. Indeed, thanks for your patience. > Not to mention that so big patch will require FSF > paperwork, unless I miss something. Oh, I thought this was already done. Do you need to submit the form or do you wait for the FSF confirmation? -- Bastien ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 13:23 ` Bastien @ 2020-06-02 13:30 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-02 13:30 UTC (permalink / raw) To: Bastien; +Cc: emacs-orgmode, Nicolas Goaziou > Oh, I thought this was already done. Do you need to submit the form > or do you wait for the FSF confirmation? Need to submit. Bastien <bzg@gnu.org> writes: > Hi Ihor, > > Ihor Radchenko <yantar92@gmail.com> writes: > >>> Thanks -- just a quick note, in case you missed the message: we are in >>> feature free for core functionalities, so we have time to work on this >>> welcome enhancement for Org 9.5, which will give us time to properly >>> test it too. >> >> I do not expect it to be merged any time soon. The patch is modifying >> low-level internals. It certainly needs a careful testing under various >> user configs. > > Indeed, thanks for your patience. > >> Not to mention that so big patch will require FSF >> paperwork, unless I miss something. > > Oh, I thought this was already done. Do you need to submit the form > or do you wait for the FSF confirmation? > > -- > Bastien -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 9:21 ` Ihor Radchenko 2020-06-02 9:23 ` Ihor Radchenko @ 2020-06-02 9:25 ` Ihor Radchenko 2020-06-05 7:26 ` Nicolas Goaziou 2 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-02 9:25 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Github link to the patch: https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > > [The patch itself will be provided in the following email] > > I have three updates from the previous version of the patch: > > 1. I managed to implement buffer-local text properties. > Now, outline folding also uses text properties without a need to give > up independent folding in indirect buffers. > > 2. The code handling modifications in folded drawers/blocks was > rewritten. The new code uses after-change-functions to re-hide text > inserted in the middle of folded regions; and text properties to > unfold folded drawers/blocks if one changes BEGIN/END line. > > 3. [experimental] Started working on improving memory and cpu footprint > of the old code related to folding/unfolding. org-hide-drawer-all now > works significantly faster because I can utilise simplified drawer > parser, which require a lot less memory. Overall, I managed to reduce > Emacs memory footprint after loading all my agenda_files twice. The > loading is also noticeably faster. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the buffer-local text properties: > > I have found char-property-alias-alist variable that controls how Emacs > calculates text property value if the property is not set. This variable > can be buffer-local, which allows independent 'invisible states in > different buffers. > > All the implementation stays in > org--get-buffer-local-text-property-symbol, which takes care about > generating unique property name and mapping it to 'invisible (or any > other) text property. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > > I simplified the code as suggested, without using pairs of before- and > after-change-functions. > > Handling text inserted into folded/invisible region is handled by a > simple after-change function. After testing, it turned out that simple > re-hiding text based on 'invisible property of the text before/after the > inserted region works pretty well. > > Modifications to BEGIN/END line of the drawers and blocks is handled via > 'modification-hooks + 'insert-behind-hooks text properties (there is no > after-change-functions analogue for text properties in Emacs). The > property is applied during folding and the modification-hook function is > made aware about the drawer/block boundaries (via apply-partially > passing element containing :begin :end markers for the current > drawer/block). Passing the element boundary is important because the > 'modification-hook will not directly know where it belongs to. Only the > modified region (which can be larger than the drawer) is passed to the > function. In the worst case, the region can be the whole buffer (if one > runs revert-buffer). > > It turned out that adding 'modification-hook text property takes a > significant cpu time (partially, because we need to take care about > possible existing 'modification-hook value, see > org--add-to-list-text-property). For now, I decided to not clear the > modification hooks during unfolding because of poor performance. > However, this approach would lead to partial unfolding in the following > case: > > :asd: > :drawer: > lksjdfksdfjl > sdfsdfsdf > :end: > > If :asd: was inserted in front of folded :drawer:, changes in :drawer: > line of the new folded :asd: drawer would reveal the text between > :drawer: and :end:. > > Let me know what you think on this. > >> You shouldn't be bothered by the case you're describing here, for >> multiple reasons. >> >> First, this issue already arises in the current implementation. No one >> bothered so far: this change is very unlikely to happen. If it becomes >> an issue, we could make sure that `org-reveal' handles this. >> >> But, more importantly, we actually /want it/ as a feature. Indeed, if >> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting >> a drawer manually would unfold /all/ drawers in the section. The user is >> more likely to write first ":BLAH:" (everything is unfolded) then >> ":END:" than ":END:", then ":BLAH:". > > Agree. This allowed me to simplify the code significantly. > >> It seems you're getting it backwards. `before-change-functions' are the >> functions being called with a possibly wide, imprecise, region to >> handle: >> >> When that happens, the arguments to ‘before-change-functions’ will >> enclose a region in which the individual changes are made, but won’t >> necessarily be the minimal such region >> >> however, after-change-functions calls are always minimal: >> >> and the arguments to each successive call of >> ‘after-change-functions’ will then delimit the part of text being >> changed exactly. >> >> If you stick to `after-change-functions', there will be no such thing as >> you describe. > > You are right here, I missed that before-change-functions are likely to > be called on large regions. I thought that the regions are same for > before/after-change-functions, but after-change-functions could be > called more than 1 time. After second thought, your vision that it is > mostly 0 or 1 times should be the majority of cases in practice. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on reducing cpu and memory footprint of org buffers: > > My simplified implementation of element boundary parser > (org--get-element-region-at-point) appears to be much faster and also > uses much less memory in comparison with org-element-at-point. > Moreover, not all the places where org-element-at-point is called > actually need the full parsed element. For example, org-hide-drawer-all, > org-hide-drawer-toggle, org-hide-block-toggle, and > org--hide-wrapper-toggle only need element type and some information > about the element boundaries - the information we can get from > org--get-element-region-at-point. > > The following version of org-hide-drawer-all seems to work much faster > in comparison with original: > > (defun org-hide-drawer-all () > "Fold all drawers in the current buffer." > (save-excursion > (goto-char (point-min)) > (while (re-search-forward org-drawer-regexp nil t) > (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer))) > (type (org-element-type drawer))) > (org-hide-drawer-toggle t nil drawer) > ;; Make sure to skip drawer entirely or we might flag it > ;; another time when matching its ending line with > ;; `org-drawer-regexp'. > (goto-char (org-element-property :end drawer)))))) > > What do you think about the idea of making use of > org--get-element-region-at-point in org code base? > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > 1. Look into other code using overlays. Specifically, > org-toggle-custom-properties, Babel hashes, and narrowed table columns. > > Best, > Ihor > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Hello, >> >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>> I have five updates from the previous version of the patch: >> >> Thank you. >> >>> 1. I implemented a simplified version of element parsing to detect >>> changes in folded drawers or blocks. No computationally expensive calls >>> of org-element-at-point or org-element-parse-buffer are needed now. >>> >>> 2. The patch is now compatible with master (commit 2e96dc639). I >>> reverted the earlier change in folding drawers and blocks. Now, they are >>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would >>> achieve nothing when we use text properties. >>> >>> 3. 'invisible text property can now be nested. This is important, for >>> example, when text inside drawers contains fontified links (which also >>> use 'invisible text property to hide parts of the link). Now, the old >>> 'invisible spec is recovered after unfolding. >> >> Interesting. I'm running out of time, so I cannot properly inspect the >> code right now. I'll try to do that before the end of the week. >> >>> 4. Some outline-* function calls in org referred to outline-flag-region >>> implementation, which is not in sync with org-flag-region in this patch. >>> I have implemented their org-* versions and replaced the calls >>> throughout .el files. Actually, some org-* versions were already >>> implemented in org, but not used for some reason (or not mentioned in >>> the manual). I have updated the relevant sections of manual. These >>> changes might be relevant to org independently of this feature branch. >> >> Yes, we certainly want to move to org-specific versions in all cases. >> >>> 5. I have managed to get a working version of outline folding via text >>> properties. However, that approach has a big downside - folding state >>> cannot be different in indirect buffer when we use text properties. I >>> have seen packages relying on this feature of org and I do not see any >>> obvious way to achieve different folding state in indirect buffer while >>> using text properties for outline folding. >> >> Hmm. Good point. This is a serious issue to consider. Even if we don't >> use text properties for outline, this also affects drawers and blocks. >> >>> For now, I still used before/after-change-functions combination. >> >> You shouldn't. >> >>> I see the following problems with using only after-change-functions: >>> >>> 1. They are not guaranteed to be called after every single change: >> >> Of course they are! See below. >> >>> From (elisp) Change Hooks: >>> "... some complex primitives call ‘before-change-functions’ once before >>> making changes, and then call ‘after-change-functions’ zero or more >>> times" >> >> "zero" means there are no changes at all, so, `after-change-functions' >> are not called, which is expected. >> >>> The consequence of it is a possibility that region passed to the >>> after-change-functions is quite big (including all the singular changes, >>> even if they are distant). This region may contain changed drawers as >>> well and unchanged drawers and needs to be parsed to determine which >>> drawers need to be re-folded. >> >> It seems you're getting it backwards. `before-change-functions' are the >> functions being called with a possibly wide, imprecise, region to >> handle: >> >> When that happens, the arguments to ‘before-change-functions’ will >> enclose a region in which the individual changes are made, but won’t >> necessarily be the minimal such region >> >> however, after-change-functions calls are always minimal: >> >> and the arguments to each successive call of >> ‘after-change-functions’ will then delimit the part of text being >> changed exactly. >> >> If you stick to `after-change-functions', there will be no such thing as >> you describe. >> >>>> And, more importantly, they are not meant to be used together, i.e., you >>>> cannot assume that a single call to `before-change-functions' always >>>> happens before calling `after-change-functions'. This can be tricky if >>>> you want to use the former to pass information to the latter. >>> >>> The fact that before-change-functions can be called multiple times >>> before after-change-functions, is trivially solved by using buffer-local >>> changes register (see org--modified-elements). >> >> Famous last words. Been there, done that, and it failed. >> >> Let me quote the manual: >> >> In general, we advise to use either before- or the after-change >> hooks, but not both. >> >> So, let me insist: don't do that. If you don't agree with me, let's at >> least agree with Emacs developers. >> >>> The register is populated by before-change-functions and cleared by >>> after-change-functions. >> >> You cannot expect `after-change-functions' to clear what >> `before-change-functions' did. This is likely to introduce pernicious >> bugs. Sorry if it sounds like FUD, but bugs in those areas are just >> horrible to squash. >> >>>> Well, `before-change-fuctions' and `after-change-functions' are not >>>> clean at all: you modify an unrelated part of the buffer, but still call >>>> those to check if a drawer needs to be unfolded somewhere. >>> >>> 2. As you pointed, instead of global before-change-functions, we can use >>> modification-hooks text property on sensitive parts of the >>> drawers/blocks. This would work, but I am concerned about one annoying >>> special case: >>> >>> ------------------------------------------------------------------------- >>> :BLAH: <inserted outside any of the existing drawers> >>> >>> <some text> >>> >>> :DRAWER: <folded> >>> Donec at pede. >>> :END: >>> ------------------------------------------------------------------------- >>> In this example, the user would not be able to unfold the folder DRAWER >>> because it will technically become a part of a new giant BLAH drawer. >>> This may be especially annoying if <some text> is more than one screen >>> long and there is no easy way to identify why unfolding does not work >>> (with point at :DRAWER:). >> >> You shouldn't be bothered by the case you're describing here, for >> multiple reasons. >> >> First, this issue already arises in the current implementation. No one >> bothered so far: this change is very unlikely to happen. If it becomes >> an issue, we could make sure that `org-reveal' handles this. >> >> But, more importantly, we actually /want it/ as a feature. Indeed, if >> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting >> a drawer manually would unfold /all/ drawers in the section. The user is >> more likely to write first ":BLAH:" (everything is unfolded) then >> ":END:" than ":END:", then ":BLAH:". >> >>> Because of this scenario, limiting before-change-functions to folded >>> drawers is not sufficient. Any change in text may need to trigger >>> unfolding. >> >> after-change-functions is more appropriate than before-change-functions, >> and local parsing, as explained in this thread, is more efficient than >> re-inventing the parser. >> >>> In the patch, I always register possible modifications in the >>> blocks/drawers intersecting with the modified region + a drawer/block >>> right next to the region. >>> >>> ----------------------------------------------------------------------- >>> ----------------------------------------------------------------------- >>> >>> More details on the nested 'invisible text property implementation. >>> >>> The idea is to keep 'invisible property stack push and popping from it >>> as we add/remove 'invisible text property. All the work is done in >>> org-flag-region. >> >> This sounds like a good idea. >> >>> This was originally intended for folding outlines via text properties. >>> Since using text properties for folding outlines is not a good idea, >>> nested text properties have much less use. >> >> AFAIU, they have. You mention link fontification, but there are other >> pieces that we could switch to text properties instead of overlays, >> e.g., Babel hashes, narrowed table columns… >> >>> 3. Multiple calls to before/after-change-functions is still a problem. I >>> am looking into following ways to reduce this number: >>> - reduce the number of elements registered as potentially modified >>> + do not add duplicates to org--modified-elements >>> + do not add unfolded elements to org--modified-elements >>> + register after-change-function as post-command hook and remove it >>> from global after-change-functions. This way, it will be called >>> twice per command only. >>> - determine common region containing org--modified-elements. if change >>> is happening within that region, there is no need to parse >>> drawers/blocks there again. >> >> This is over-engineering. Again, please focus on local changes, as >> discussed before. >> >>> Recipe to have different (org-element-at-point) and >>> (org-element-parse-buffer 'element) >>> ------------------------------------------------------------------------- >>> <point-min> >>> :PROPERTIES: >>> :CREATED: [2020-05-23 Sat 02:32] >>> :END: >>> >>> >>> <point-max> >>> ------------------------------------------------------------------------- >> >> I didn't look at this situation in particular, but there are cases where >> different :post-blank values are inevitable, for example at the end of >> a section. >> >> Regards, >> >> -- >> Nicolas Goaziou > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-02 9:21 ` Ihor Radchenko 2020-06-02 9:23 ` Ihor Radchenko 2020-06-02 9:25 ` Ihor Radchenko @ 2020-06-05 7:26 ` Nicolas Goaziou 2020-06-05 8:18 ` Ihor Radchenko 2 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-06-05 7:26 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > [The patch itself will be provided in the following email] Thank you. > I have found char-property-alias-alist variable that controls how Emacs > calculates text property value if the property is not set. This variable > can be buffer-local, which allows independent 'invisible states in > different buffers. Great. I didn't know about this variable! > All the implementation stays in > org--get-buffer-local-text-property-symbol, which takes care about > generating unique property name and mapping it to 'invisible (or any > other) text property. See also `gensym'. Do we really need to use it for something else than `invisible'? If not, the tool doesn't need to be generic. > I simplified the code as suggested, without using pairs of before- and > after-change-functions. Great! > Handling text inserted into folded/invisible region is handled by a > simple after-change function. After testing, it turned out that simple > re-hiding text based on 'invisible property of the text before/after the > inserted region works pretty well. OK, but this may not be sufficient if we want to do slightly better than overlays in that area. This is not mandatory, though. > Modifications to BEGIN/END line of the drawers and blocks is handled via > 'modification-hooks + 'insert-behind-hooks text properties (there is no > after-change-functions analogue for text properties in Emacs). The > property is applied during folding and the modification-hook function is > made aware about the drawer/block boundaries (via apply-partially > passing element containing :begin :end markers for the current > drawer/block). Passing the element boundary is important because the > 'modification-hook will not directly know where it belongs to. Only the > modified region (which can be larger than the drawer) is passed to the > function. In the worst case, the region can be the whole buffer (if one > runs revert-buffer). As discussed before, I don't think you need to use `modification-hooks' or `insert-behind-hooks' if you already use `after-change-functions'. `after-change-functions' are also triggered upon text properties changes. So, what is the use case for the other hooks? > It turned out that adding 'modification-hook text property takes a > significant cpu time (partially, because we need to take care about > possible existing 'modification-hook value, see > org--add-to-list-text-property). For now, I decided to not clear the > modification hooks during unfolding because of poor performance. > However, this approach would lead to partial unfolding in the following > case: > > :asd: > :drawer: > lksjdfksdfjl > sdfsdfsdf > :end: > > If :asd: was inserted in front of folded :drawer:, changes in :drawer: > line of the new folded :asd: drawer would reveal the text between > :drawer: and :end:. > > Let me know what you think on this. I have first to understand the use case for `modification-hook'. But I think unfolding is the right thing to do in this situation, isn't it? > My simplified implementation of element boundary parser > (org--get-element-region-at-point) appears to be much faster and also > uses much less memory in comparison with org-element-at-point. > Moreover, not all the places where org-element-at-point is called > actually need the full parsed element. For example, org-hide-drawer-all, > org-hide-drawer-toggle, org-hide-block-toggle, and > org--hide-wrapper-toggle only need element type and some information > about the element boundaries - the information we can get from > org--get-element-region-at-point. [...] > What do you think about the idea of making use of > org--get-element-region-at-point in org code base? `org--get-element-region-at-point' is certainly faster, but it is also wrong, unfortunately. Org syntax is not context-free grammar. If you try to parse it locally, starting from anywhere, it will fail at some point. For example, your function would choke in the following case: [fn:1] Def1 #+begin_something [fn:2] Def2 #+end_something AFAIK, the only proper way to parse it is to start from a known position in the buffer. If you have no information about the buffer, the headline above is the position you want. With cache could help to start below. Anyway, in this particular case, you should not use `org--get-element-region-at-point'. Hopefully, we don't need to parse anything. In an earlier message, I suggested a few checks to make on the modified text in order to decide if something should be unfolded, or not. I suggest to start from there, and fix any shortcomings we might encounter. We're replacing overlays: low-level is good in this area. WDYT? Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-05 7:26 ` Nicolas Goaziou @ 2020-06-05 8:18 ` Ihor Radchenko 2020-06-05 13:50 ` Nicolas Goaziou 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-06-05 8:18 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > See also `gensym'. Do we really need to use it for something else than > `invisible'? If not, the tool doesn't need to be generic. For now, I also use it for buffer-local 'invisible stack. The stack is needed to preserve folding state of drawers/blocks inside folded outline. Though I am thinking about replacing the stack with separate text properties, like 'invisible-outline-buffer-local + 'invisible-drawer-buffer-local + 'invisible-block-buffer-local. Maintaining stack takes a noticeable percentage of CPU time in profiler. org--get-buffer-local-text-property-symbol must take care about situation with indirect buffers. When an indirect buffer is created from some org buffer, the old value of char-property-alias-alist is carried over. We need to detect this case and create new buffer-local symbol, which is unique to the newly created buffer (but not create it if the buffer-local property is already there). Then, the new symbol must replace the old alias in char-property-alias-alist + old folding state must be preserved (via copying the old invisibility specs into the new buffer-local text property). I do not see how gensym can benefit this logic. > OK, but this may not be sufficient if we want to do slightly better than > overlays in that area. This is not mandatory, though. Could you elaborate on what can be "slightly better"? > As discussed before, I don't think you need to use `modification-hooks' > or `insert-behind-hooks' if you already use `after-change-functions'. > > `after-change-functions' are also triggered upon text properties > changes. So, what is the use case for the other hooks? The problem is that `after-change-functions' cannot be a text property. Only `modification-hooks' and `insert-in-front/behind-hooks' can be a valid text property. If we use `after-change-functions', they will always be triggered, regardless if the change was made inside or outside folded region. >> :asd: >> :drawer: >> lksjdfksdfjl >> sdfsdfsdf >> :end: >> >> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >> line of the new folded :asd: drawer would reveal the text between >> :drawer: and :end:. >> >> Let me know what you think on this. > I have first to understand the use case for `modification-hook'. But > I think unfolding is the right thing to do in this situation, isn't it? That situation arises because the modification-hooks from ":drawer:" (they are set via text properties) only have information about the :drawer:...:end: drawer before the modifications (they were set when :drawer: was folded last time). So, they will only unfold a part of the new :asd: drawer. I do not see a simple way to unfold everything without re-parsing the drawer around the changed text. Actually, I am quite unhappy with the performance of modification-hooks set via text properties (I am using this patch on my Emacs during this week). It appears that setting the text properties costs a significant CPU time in practice, even though running the hooks is pretty fast. I will think about a way to handle modifications using global after-change-functions. > `org--get-element-region-at-point' is certainly faster, but it is also > wrong, unfortunately. > > Org syntax is not context-free grammar. If you try to parse it locally, > starting from anywhere, it will fail at some point. For example, your > function would choke in the following case: > > [fn:1] Def1 > #+begin_something > > [fn:2] Def2 > #+end_something I see. > AFAIK, the only proper way to parse it is to start from a known position > in the buffer. If you have no information about the buffer, the headline > above is the position you want. With cache could help to start below. > Anyway, in this particular case, you should not use > `org--get-element-region-at-point'. OK Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> [The patch itself will be provided in the following email] > > Thank you. > >> I have found char-property-alias-alist variable that controls how Emacs >> calculates text property value if the property is not set. This variable >> can be buffer-local, which allows independent 'invisible states in >> different buffers. > > Great. I didn't know about this variable! > >> All the implementation stays in >> org--get-buffer-local-text-property-symbol, which takes care about >> generating unique property name and mapping it to 'invisible (or any >> other) text property. > > See also `gensym'. Do we really need to use it for something else than > `invisible'? If not, the tool doesn't need to be generic. > >> I simplified the code as suggested, without using pairs of before- and >> after-change-functions. > > Great! > >> Handling text inserted into folded/invisible region is handled by a >> simple after-change function. After testing, it turned out that simple >> re-hiding text based on 'invisible property of the text before/after the >> inserted region works pretty well. > > OK, but this may not be sufficient if we want to do slightly better than > overlays in that area. This is not mandatory, though. > >> Modifications to BEGIN/END line of the drawers and blocks is handled via >> 'modification-hooks + 'insert-behind-hooks text properties (there is no >> after-change-functions analogue for text properties in Emacs). The >> property is applied during folding and the modification-hook function is >> made aware about the drawer/block boundaries (via apply-partially >> passing element containing :begin :end markers for the current >> drawer/block). Passing the element boundary is important because the >> 'modification-hook will not directly know where it belongs to. Only the >> modified region (which can be larger than the drawer) is passed to the >> function. In the worst case, the region can be the whole buffer (if one >> runs revert-buffer). > > As discussed before, I don't think you need to use `modification-hooks' > or `insert-behind-hooks' if you already use `after-change-functions'. > > `after-change-functions' are also triggered upon text properties > changes. So, what is the use case for the other hooks? > >> It turned out that adding 'modification-hook text property takes a >> significant cpu time (partially, because we need to take care about >> possible existing 'modification-hook value, see >> org--add-to-list-text-property). For now, I decided to not clear the >> modification hooks during unfolding because of poor performance. >> However, this approach would lead to partial unfolding in the following >> case: >> >> :asd: >> :drawer: >> lksjdfksdfjl >> sdfsdfsdf >> :end: >> >> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >> line of the new folded :asd: drawer would reveal the text between >> :drawer: and :end:. >> >> Let me know what you think on this. > > I have first to understand the use case for `modification-hook'. But > I think unfolding is the right thing to do in this situation, isn't it? > >> My simplified implementation of element boundary parser >> (org--get-element-region-at-point) appears to be much faster and also >> uses much less memory in comparison with org-element-at-point. >> Moreover, not all the places where org-element-at-point is called >> actually need the full parsed element. For example, org-hide-drawer-all, >> org-hide-drawer-toggle, org-hide-block-toggle, and >> org--hide-wrapper-toggle only need element type and some information >> about the element boundaries - the information we can get from >> org--get-element-region-at-point. > > [...] > >> What do you think about the idea of making use of >> org--get-element-region-at-point in org code base? > > `org--get-element-region-at-point' is certainly faster, but it is also > wrong, unfortunately. > > Org syntax is not context-free grammar. If you try to parse it locally, > starting from anywhere, it will fail at some point. For example, your > function would choke in the following case: > > [fn:1] Def1 > #+begin_something > > [fn:2] Def2 > #+end_something > > AFAIK, the only proper way to parse it is to start from a known position > in the buffer. If you have no information about the buffer, the headline > above is the position you want. With cache could help to start below. > Anyway, in this particular case, you should not use > `org--get-element-region-at-point'. > > Hopefully, we don't need to parse anything. In an earlier message, > I suggested a few checks to make on the modified text in order to decide > if something should be unfolded, or not. I suggest to start from there, > and fix any shortcomings we might encounter. We're replacing overlays: > low-level is good in this area. > > WDYT? > > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-05 8:18 ` Ihor Radchenko @ 2020-06-05 13:50 ` Nicolas Goaziou 2020-06-08 5:05 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Nicolas Goaziou @ 2020-06-05 13:50 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: >> See also `gensym'. Do we really need to use it for something else than >> `invisible'? If not, the tool doesn't need to be generic. > > For now, I also use it for buffer-local 'invisible stack. The stack is > needed to preserve folding state of drawers/blocks inside folded > outline. Though I am thinking about replacing the stack with separate > text properties, like 'invisible-outline-buffer-local + > 'invisible-drawer-buffer-local + 'invisible-block-buffer-local. > Maintaining stack takes a noticeable percentage of CPU time in profiler. > > org--get-buffer-local-text-property-symbol must take care about > situation with indirect buffers. When an indirect buffer is created from > some org buffer, the old value of char-property-alias-alist is carried > over. We need to detect this case and create new buffer-local symbol, > which is unique to the newly created buffer (but not create it if the > buffer-local property is already there). Then, the new symbol must > replace the old alias in char-property-alias-alist + old folding state > must be preserved (via copying the old invisibility specs into the new > buffer-local text property). I do not see how gensym can benefit this > logic. `gensym' is just a shorter, and somewhat standard way, to create a new uninterned symbol with a given prefix. You seem to re-invent it. What you do with that new symbol is orthogonal to that suggestion, of course. >> OK, but this may not be sufficient if we want to do slightly better than >> overlays in that area. This is not mandatory, though. > > Could you elaborate on what can be "slightly better"? IIRC, I gave examples of finer control of folding state after a change. Consider this _folded_ drawer: :BEGIN: Foo :END: Inserting ":END" in it should not unfold it, as it is currently the case with overlays, :BEGIN Foo :END :END: but a soon as the last ":" is inserted, the initial drawer could be expanded. :BEGIN Foo :END: :END: The latter case is not currently handled by overlays. This is what I call "slightly better". Also, note that this change is not related to opening and closing lines of the initial drawer, so sticking text properties on them would not help here. Another case is modifying those borders, e.g., :BEGIN: :BEGIN: Foo ------> Foo :END: :ND: which should expand the drawer. Your implementation catches this, but I'm pointing out that current implementation with overlays does not. Even though that's not strictly required for compatibility with overlays, it is a welcome slight improvement. >> As discussed before, I don't think you need to use `modification-hooks' >> or `insert-behind-hooks' if you already use `after-change-functions'. >> >> `after-change-functions' are also triggered upon text properties >> changes. So, what is the use case for the other hooks? > > The problem is that `after-change-functions' cannot be a text property. > Only `modification-hooks' and `insert-in-front/behind-hooks' can be a > valid text property. If we use `after-change-functions', they will > always be triggered, regardless if the change was made inside or outside > folded region. As discussed, text properties are local to the change, but require extra care when moving text around. You also observed serious overhead when using them. OTOH, even if `a-c-f' is not local, you can quickly determine if the change altered a folded element, so the overhead is limited, i.e., mostly checking for a text property at a given buffer position. To be clear, I initially thought that text properties were a superior choice, but I changed my mind a while ago, and I thought you had, too. IOW, `after-change-functions' is the way to go, since you have no strong reason to stick to text properties for this kind of function. >>> :asd: >>> :drawer: >>> lksjdfksdfjl >>> sdfsdfsdf >>> :end: >>> >>> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >>> line of the new folded :asd: drawer would reveal the text between >>> :drawer: and :end:. >>> >>> Let me know what you think on this. > >> I have first to understand the use case for `modification-hook'. But >> I think unfolding is the right thing to do in this situation, isn't it? > > That situation arises because the modification-hooks from ":drawer:" > (they are set via text properties) only have information about the > :drawer:...:end: drawer before the modifications (they were set when > :drawer: was folded last time). So, they will only unfold a part of the > new :asd: drawer. I do not see a simple way to unfold everything without > re-parsing the drawer around the changed text. Oh! I misread your message. I withdraw what I wrote. In this case, we don't want to unfold anything. The situation is not worse than what we have now, and trying to fix it would have repercussions down in the buffer, e.g., expanding drawers screen below. As a rule of thumb, I think we can pay attention to changes in the folded text, and its immediate surroundings (e.g., the opening line, which is not folded), but no further. As written above, slight changes are welcome, but let's not go overboard and parse a whole section just to know if we can expand a drawer. > Actually, I am quite unhappy with the performance of modification-hooks > set via text properties (I am using this patch on my Emacs during this > week). It appears that setting the text properties costs a significant > CPU time in practice, even though running the hooks is pretty fast. > I will think about a way to handle modifications using global > after-change-functions. That's better, IMO. I gave you a few ideas to quickly check if a change requires expansion, in an earlier mail. I suggest to start out from that. Let me know if you have questions about it. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-05 13:50 ` Nicolas Goaziou @ 2020-06-08 5:05 ` Ihor Radchenko 2020-06-08 5:06 ` Ihor Radchenko ` (2 more replies) 0 siblings, 3 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-08 5:05 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Hello, [The patch itself will be provided in the following email] I have four more updates from the previous version of the patch: 1. All the code handling modifications in folded drawers/blocks is moved to after-change-function. It works as follows: - if any text is inserted in the middle of hidden region, that text is also hidden; - if BEGIN/END line of a folded drawer do not match org-drawer-regexp and org-property-end-re, unfold it; - if org-property-end-re or new org-outline-regexp-bol is inserted in the middle of the drawer, unfold it; - the same logic for blocks. 2. The text property stack is rewritten using char-property-alias-alist. This is faster in comparison with previous approach, which involved modifying all the text properties every timer org-flag-region was called. 3. org-toggle-custom-properties-visibility is rewritten using text properties. I also took a freedom to implement a new feature here. Now, setting new `org-custom-properties-hide-emptied-drawers' to non-nil will result in hiding the whole property drawer if it contains only org-custom-properties. 4. This patch should work against 1aa095ccf. However, the merge was not trivial here. Recent commits actively used the fact that drawers and outlines are hidden via 'outline invisibility spec, which is not the case in this branch. I am not confident that I did not break anything during the merge, especially 1aa095ccf. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the new implementation for tracking changes: > I gave you a few ideas to quickly check if a change requires expansion, > in an earlier mail. I suggest to start out from that. Let me know if you > have questions about it. All the code lives in org-after-change-function. I tried to incorporate the earlier Nicholas' suggestions, except the parts related to intersecting blocks and drawers. I am not sure if I understand the parsing priority of blocks vs. drawers. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the text property stack: The earlier version of the code literally used stack to save pre-existing 'invisibility specs in org-flag-region. This was done on every invocation of org-flag-region, which made org-flag-region significantly slower. I re-implemented the same feature using char-property-alias-alist. Now, different invisibility specs live in separate text properties and can be safely modified independently. The specs are applied according to org--invisible-spec-priority-list. A side effect of current implementation is that char-property-alias-alist is fully controlled by org. All the pre-existing settings for 'invisible text property will be overwritten by org. > `gensym' is just a shorter, and somewhat standard way, to create a new > uninterned symbol with a given prefix. You seem to re-invent it. What > you do with that new symbol is orthogonal to that suggestion, of course. I do not think that `gensym' is suitable here. We don't want a new symbol every time org--get-buffer-local-invisible-property-symbol is called. It should return the same symbol if it is called from the same buffer multiple times. ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the org-toggle-custom-properties-visibility: The implementation showcases how to introduce new invisibility specs to org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) one also needs to add the spec into org--invisible-spec-priority-list: (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property) Searching for text with the given invisibility spec is done as follows: (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t) This last piece of code is probably not the most elegant. I am thinking if creating some higher-level interface would be more reasonable here. What do you think? The new customisation `org-custom-properties-hide-emptied-drawers' sounds logical for me since empty property drawers left after invoking org-toggle-custom-properties-visibility are rather useless according to my experience. If one already wants to hide parts of property drawers, I do not see a reason to show leftover :PROPERTIES: :END: ----------------------------------------------------------------------- ----------------------------------------------------------------------- More details on the merge with the latest master: I tried my best to not break anything. However, I am not sure if I understand all the recent commits. Could someone take a look if there is anything suspicious in org-next-visible-heading? Also, I have seen some optimisations making use of the fact that drawers and headlines both use 'outline invisibility spec. This change in the implementation details supposed to improve performance and should not be necessary if this patch is going to be merged. Would it be possible to refrain from abusing this particular implementation detail in the nearest commits on master (unless really necessary)? ----------------------------------------------------------------------- ----------------------------------------------------------------------- Further work: I would like to finalise the current patch and work on other code using overlays separately. This patch is already quite complicated as is. I do not want to introduce even more potential bugs by working on things not directly affected by this version of the patch. Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >>> See also `gensym'. Do we really need to use it for something else than >>> `invisible'? If not, the tool doesn't need to be generic. >> >> For now, I also use it for buffer-local 'invisible stack. The stack is >> needed to preserve folding state of drawers/blocks inside folded >> outline. Though I am thinking about replacing the stack with separate >> text properties, like 'invisible-outline-buffer-local + >> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local. >> Maintaining stack takes a noticeable percentage of CPU time in profiler. >> >> org--get-buffer-local-text-property-symbol must take care about >> situation with indirect buffers. When an indirect buffer is created from >> some org buffer, the old value of char-property-alias-alist is carried >> over. We need to detect this case and create new buffer-local symbol, >> which is unique to the newly created buffer (but not create it if the >> buffer-local property is already there). Then, the new symbol must >> replace the old alias in char-property-alias-alist + old folding state >> must be preserved (via copying the old invisibility specs into the new >> buffer-local text property). I do not see how gensym can benefit this >> logic. > > `gensym' is just a shorter, and somewhat standard way, to create a new > uninterned symbol with a given prefix. You seem to re-invent it. What > you do with that new symbol is orthogonal to that suggestion, of course. > >>> OK, but this may not be sufficient if we want to do slightly better than >>> overlays in that area. This is not mandatory, though. >> >> Could you elaborate on what can be "slightly better"? > > IIRC, I gave examples of finer control of folding state after a change. > Consider this _folded_ drawer: > > :BEGIN: > Foo > :END: > > Inserting ":END" in it should not unfold it, as it is currently the case > with overlays, > > :BEGIN > Foo > :END > :END: > > but a soon as the last ":" is inserted, the initial drawer could be > expanded. > > :BEGIN > Foo > :END: > :END: > > The latter case is not currently handled by overlays. This is what > I call "slightly better". > > Also, note that this change is not related to opening and closing lines > of the initial drawer, so sticking text properties on them would not > help here. > > Another case is modifying those borders, e.g., > > > :BEGIN: :BEGIN: > Foo ------> Foo > :END: :ND: > > which should expand the drawer. Your implementation catches this, but > I'm pointing out that current implementation with overlays does not. > Even though that's not strictly required for compatibility with > overlays, it is a welcome slight improvement. > >>> As discussed before, I don't think you need to use `modification-hooks' >>> or `insert-behind-hooks' if you already use `after-change-functions'. >>> >>> `after-change-functions' are also triggered upon text properties >>> changes. So, what is the use case for the other hooks? >> >> The problem is that `after-change-functions' cannot be a text property. >> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a >> valid text property. If we use `after-change-functions', they will >> always be triggered, regardless if the change was made inside or outside >> folded region. > > As discussed, text properties are local to the change, but require extra > care when moving text around. You also observed serious overhead when > using them. > > OTOH, even if `a-c-f' is not local, you can quickly determine if the > change altered a folded element, so the overhead is limited, i.e., > mostly checking for a text property at a given buffer position. > > To be clear, I initially thought that text properties were a superior > choice, but I changed my mind a while ago, and I thought you had, too. > IOW, `after-change-functions' is the way to go, since you have no strong > reason to stick to text properties for this kind of function. > >>>> :asd: >>>> :drawer: >>>> lksjdfksdfjl >>>> sdfsdfsdf >>>> :end: >>>> >>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >>>> line of the new folded :asd: drawer would reveal the text between >>>> :drawer: and :end:. >>>> >>>> Let me know what you think on this. >> >>> I have first to understand the use case for `modification-hook'. But >>> I think unfolding is the right thing to do in this situation, isn't it? >> >> That situation arises because the modification-hooks from ":drawer:" >> (they are set via text properties) only have information about the >> :drawer:...:end: drawer before the modifications (they were set when >> :drawer: was folded last time). So, they will only unfold a part of the >> new :asd: drawer. I do not see a simple way to unfold everything without >> re-parsing the drawer around the changed text. > > Oh! I misread your message. I withdraw what I wrote. In this case, we > don't want to unfold anything. The situation is not worse than what we > have now, and trying to fix it would have repercussions down in the > buffer, e.g., expanding drawers screen below. > > As a rule of thumb, I think we can pay attention to changes in the > folded text, and its immediate surroundings (e.g., the opening line, > which is not folded), but no further. > > As written above, slight changes are welcome, but let's not go overboard > and parse a whole section just to know if we can expand a drawer. > >> Actually, I am quite unhappy with the performance of modification-hooks >> set via text properties (I am using this patch on my Emacs during this >> week). It appears that setting the text properties costs a significant >> CPU time in practice, even though running the hooks is pretty fast. >> I will think about a way to handle modifications using global >> after-change-functions. > > That's better, IMO. > > I gave you a few ideas to quickly check if a change requires expansion, > in an earlier mail. I suggest to start out from that. Let me know if you > have questions about it. -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-08 5:05 ` Ihor Radchenko @ 2020-06-08 5:06 ` Ihor Radchenko 2020-06-08 5:08 ` Ihor Radchenko 2020-06-10 17:14 ` Nicolas Goaziou 2 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-08 5:06 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 44 bytes --] The patch (against 1aa095ccf) is attached. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: featuredrawertextprop-20200608.patch --] [-- Type: text/x-diff, Size: 50930 bytes --] diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el index 9f8677871..ab470ea9b 100644 --- a/contrib/lisp/org-notify.el +++ b/contrib/lisp/org-notify.el @@ -246,7 +246,7 @@ seconds. The default value for SECS is 20." (switch-to-buffer (find-file-noselect file)) (org-with-wide-buffer (goto-char begin) - (outline-show-entry)) + (org-show-entry)) (goto-char begin) (search-forward "DEADLINE: <") (search-forward ":") diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el index bfc4d6c3e..2312b235c 100644 --- a/contrib/lisp/org-velocity.el +++ b/contrib/lisp/org-velocity.el @@ -325,7 +325,7 @@ use it." (save-excursion (when narrow (org-narrow-to-subtree)) - (outline-show-all))) + (org-show-all))) (defun org-velocity-edit-entry/inline (heading) "Edit entry at HEADING in the original buffer." diff --git a/doc/org-manual.org b/doc/org-manual.org index efad195e1..c6f167eac 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and Switch back to the startup visibility of the buffer (see [[*Initial visibility]]). -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) :: +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) :: #+cindex: show all, command #+kindex: C-u C-u C-u TAB - #+findex: outline-show-all + #+findex: org-show-all Show all, including drawers. - {{{kbd(C-c C-r)}}} (~org-reveal~) :: @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and headings. With a double prefix argument, also show the entire subtree of the parent. -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) :: +- {{{kbd(C-c C-k)}}} (~org-show-branches~) :: #+cindex: show branches, command #+kindex: C-c C-k - #+findex: outline-show-branches + #+findex: org-show-branches Expose all the headings of the subtree, but not their bodies. -- {{{kbd(C-c TAB)}}} (~outline-show-children~) :: +- {{{kbd(C-c TAB)}}} (~org-show-children~) :: #+cindex: show children, command #+kindex: C-c TAB - #+findex: outline-show-children + #+findex: org-show-children Expose all direct children of the subtree. With a numeric prefix argument {{{var(N)}}}, expose all children down to level {{{var(N)}}}. @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way: command (see [[*Visibility Cycling]]). You can force cycling archived subtrees with {{{kbd(C-TAB)}}}, or by setting the option ~org-cycle-open-archived-trees~. Also normal outline commands, like - ~outline-show-all~, open archived subtrees. + ~org-show-all~, open archived subtrees. - #+vindex: org-sparse-tree-open-archived-trees diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index 9fbeb2a1e..2f121f743 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -6824,7 +6824,7 @@ and stored in the variable `org-prefix-format-compiled'." (t " %-12:c%?-12t% s"))) (start 0) varform vars var e c f opt) - (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)" + (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)" s start) (setq var (or (cdr (assoc (match-string 4 s) '(("c" . category) ("t" . time) ("l" . level) ("s" . extra) @@ -9136,20 +9136,20 @@ if it was hidden in the outline." ((and (called-interactively-p 'any) (= more 1)) (message "Remote: show with default settings")) ((= more 2) - (outline-show-entry) + (org-show-entry) (org-show-children) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'children)) (message "Remote: CHILDREN")) ((= more 3) - (outline-show-subtree) + (org-show-subtree) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'subtree)) (message "Remote: SUBTREE")) ((> more 3) - (outline-show-subtree) + (org-show-subtree) (message "Remote: SUBTREE AND ALL DRAWERS"))) (select-window win))) diff --git a/lisp/org-archive.el b/lisp/org-archive.el index d3e12d17b..d864dad8a 100644 --- a/lisp/org-archive.el +++ b/lisp/org-archive.el @@ -330,7 +330,7 @@ direct children of this heading." (insert (if datetree-date "" "\n") heading "\n") (end-of-line 0)) ;; Make the subtree visible - (outline-show-subtree) + (org-show-subtree) (if org-archive-reversed-order (progn (org-back-to-heading t) diff --git a/lisp/org-colview.el b/lisp/org-colview.el index e50a4d7c8..e656df555 100644 --- a/lisp/org-colview.el +++ b/lisp/org-colview.el @@ -699,7 +699,7 @@ FUN is a function called with no argument." (move-beginning-of-line 2) (org-at-heading-p t))))) (unwind-protect (funcall fun) - (when hide-body (outline-hide-entry))))) + (when hide-body (org-hide-entry))))) (defun org-columns-previous-allowed-value () "Switch to the previous allowed value for this column." diff --git a/lisp/org-compat.el b/lisp/org-compat.el index 5953f89d2..09a09472a 100644 --- a/lisp/org-compat.el +++ b/lisp/org-compat.el @@ -138,12 +138,8 @@ This is a floating point number if the size is too large for an integer." ;;; Emacs < 25.1 compatibility (when (< emacs-major-version 25) - (defalias 'outline-hide-entry 'hide-entry) - (defalias 'outline-hide-sublevels 'hide-sublevels) - (defalias 'outline-hide-subtree 'hide-subtree) (defalias 'outline-show-branches 'show-branches) (defalias 'outline-show-children 'show-children) - (defalias 'outline-show-entry 'show-entry) (defalias 'outline-show-subtree 'show-subtree) (defalias 'xref-find-definitions 'find-tag) (defalias 'format-message 'format) @@ -644,7 +640,7 @@ When optional argument ELEMENT is a parsed drawer, as returned by When buffer positions BEG and END are provided, hide or show that region as a drawer without further ado." (declare (obsolete "use `org-hide-drawer-toggle' instead." "Org 9.4")) - (if (and beg end) (org-flag-region beg end flag 'outline) + (if (and beg end) (org-flag-region beg end flag 'org-hide-drawer) (let ((drawer (or element (and (save-excursion @@ -658,7 +654,7 @@ region as a drawer without further ado." (save-excursion (goto-char (org-element-property :end drawer)) (skip-chars-backward " \t\n") (line-end-position)) - flag 'outline) + flag 'org-hide-drawer) ;; When the drawer is hidden away, make sure point lies in ;; a visible part of the buffer. (when (invisible-p (max (1- (point)) (point-min))) diff --git a/lisp/org-element.el b/lisp/org-element.el index ac41b7650..2d5c8d771 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -4320,7 +4320,7 @@ element or object. Meaningful values are `first-section', TYPE is the type of the current element or object. If PARENT? is non-nil, assume the next element or object will be -located inside the current one. " +located inside the current one." (if parent? (pcase type (`headline 'section) diff --git a/lisp/org-keys.el b/lisp/org-keys.el index 37df29983..a714dec0f 100644 --- a/lisp/org-keys.el +++ b/lisp/org-keys.el @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names." #'org-next-visible-heading) (define-key org-mode-map [remap outline-previous-visible-heading] #'org-previous-visible-heading) -(define-key org-mode-map [remap show-children] #'org-show-children) +(define-key org-mode-map [remap outline-show-children] #'org-show-children) ;;;; Make `C-c C-x' a prefix key (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap)) diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..b17c0cb4d 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." \f -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,26 +705,126 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org-remove-text-properties (start end properties &optional object) + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. +Do not remove invisible text properties specified by 'outline, +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this +is needed to keep outlines, drawers, and blocks hidden unless they are +toggled by user. +Note: The below may be too specific and create troubles if more +invisibility specs are added to org in future" + (when (plist-member properties 'invisible) + (let ((pos start) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer + 'outline)) + (remove-text-properties pos next '(invisible nil) object)) + (setq pos next)))) + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) + (remove-text-properties start end properties-stripped object))) + +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + +(defun org--add-to-list-text-property (from to prop element) + "Add element to text property PROP, whos value should be a list." + (add-text-properties from to `(,prop ,(list element))) ; create if none + ;; add to existing + (alter-text-property from to + prop + (lambda (val) + (if (member element val) + val + (cons element val))))) + +(defun org--remove-from-list-text-property (from to prop element) + "Remove ELEMENT from text propery PROP, whos value should be a list." + (let ((pos from)) + (while (< pos to) + (when-let ((val (get-text-property pos prop))) + (if (equal val (list element)) + (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil)) + (put-text-property pos (next-single-char-property-change pos prop nil to) + prop (remove element (get-text-property pos prop))))) + (setq pos (next-single-char-property-change pos prop nil to))))) + +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block) + "Priority of invisibility specs.") + +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only) + "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC. +If the buffer already have buffer-local setup in `char-property-alias-alist' +and the setup appears to be created for different buffer, +copy the old invisibility state into new buffer-local text properties, +unless RETURN-ONLY is non-nil." + (if (not (member spec org--invisible-spec-priority-list)) + (user-error "%s should be a valid invisibility spec" spec) + (let* ((buf (or buffer (current-buffer)))) + (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S" + (symbol-name spec) + ;; (sxhash buf) appears to be not constant over time. + ;; Using buffer-name is safe, since the only place where + ;; buffer-local text property actually matters is an indirect + ;; buffer, where the name cannot be same anyway. + (sxhash (buffer-name buf)))))) + (prog1 + local-prop + (unless return-only + (with-current-buffer buf + (unless (member local-prop (alist-get 'invisible char-property-alias-alist)) + ;; copy old property + (dolist (old-prop (alist-get 'invisible char-property-alias-alist)) + (org-with-wide-buffer + (let* ((pos (point-min)) + (spec (seq-find (lambda (spec) + (string-match-p (symbol-name spec) + (symbol-name old-prop))) + org--invisible-spec-priority-list)) + (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))) + (while (< pos (point-max)) + (when-let (val (get-text-property pos old-prop)) + (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val)) + (setq pos (next-single-char-property-change pos old-prop)))))) + (setq-local char-property-alias-alist + (cons (cons 'invisible + (mapcar (lambda (spec) + (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)) + org--invisible-spec-priority-list)) + (remove (assq 'invisible char-property-alias-alist) + char-property-alias-alist))))))))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + (with-silent-modifications + (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil)) + (when flag + (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec)))) \f ;;; Regexp matching (defsubst org-pos-in-match-range (pos n) - (and (match-beginning n) - (<= (match-beginning n) pos) - (>= (match-end n) pos))) +(and (match-beginning n) + (<= (match-beginning n) pos) + (>= (match-end n) pos))) (defun org-skip-whitespace () "Skip over space, tabs and newline characters." diff --git a/lisp/org-src.el b/lisp/org-src.el index 6f6c544dc..9e8a50044 100644 --- a/lisp/org-src.el +++ b/lisp/org-src.el @@ -529,8 +529,8 @@ Leave point in edit buffer." (org-src-switch-to-buffer buffer 'edit) ;; Insert contents. (insert contents) - (remove-text-properties (point-min) (point-max) - '(display nil invisible nil intangible nil)) + (org-remove-text-properties (point-min) (point-max) + '(display nil invisible nil intangible nil)) (unless preserve-ind (org-do-remove-indentation)) (set-buffer-modified-p nil) (setq buffer-file-name nil) diff --git a/lisp/org-table.el b/lisp/org-table.el index 6462b99c4..75801161b 100644 --- a/lisp/org-table.el +++ b/lisp/org-table.el @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'." (arg (let ((b (save-excursion (skip-chars-backward "^|") (point))) (e (save-excursion (skip-chars-forward "^|\r\n") (point)))) - (remove-text-properties b e '(invisible t intangible t)) + (org-remove-text-properties b e '(invisible t intangible t)) (if (and (boundp 'font-lock-mode) font-lock-mode) (font-lock-fontify-block)))) (t @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'." (setq word-wrap t) (goto-char (setq p (point-max))) (insert (org-trim field)) - (remove-text-properties p (point-max) '(invisible t intangible t)) + (org-remove-text-properties p (point-max) '(invisible t intangible t)) (goto-char p) (setq-local org-finish-function 'org-table-finish-edit-field) (setq-local org-window-configuration cw) diff --git a/lisp/org.el b/lisp/org.el index e5cea04c6..3d4a7b072 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") (defvar ffap-url-regexp) (defvar org-element-paragraph-separate) +(defvar org-element-all-objects) +(defvar org-element-all-elements) +(defvar org-element-greater-elements) (defvar org-indent-indentation-per-level) (defvar org-radio-target-regexp) (defvar org-target-link-regexp) @@ -4734,9 +4738,153 @@ This is for getting out of special buffers like capture.") ;;;; Define the Org mode +;;; Handling buffer modifications + (defun org-before-change-function (_beg _end) "Every change indicates that a table might need an update." (setq org-table-may-need-update t)) + +(defun org-after-change-function (from to len) + "Process changes in folded elements. +If a text was inserted into invisible region, hide the inserted text. +If the beginning/end line of a folded drawer/block was changed, unfold it. +If a valid end line was inserted in the middle of the folded drawer/block, unfold it." + + ;; re-hide text inserted in the middle of a folded region + (dolist (spec org--invisible-spec-priority-list) + (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec))) + (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec)))) + (when (eq spec-to spec-from) + (org-flag-region from to 't spec-to)))) + + ;; Process all the folded text between `from' and `to' + (org-with-wide-buffer + + (if (< to from) + (let ((tmp from)) + (setq from to) + (setq to tmp))) + + ;; Include next/previous line into the changed region. + ;; This is needed to catch edits in beginning line of a folded + ;; element. + (setq to (save-excursion (goto-char to) (forward-line) (point))) + (setq from (save-excursion (goto-char from) (forward-line -1) (point))) + + ;; Expand the considered region to include partially present folded + ;; drawer/block. + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) + + ;; check folded drawers + (let ((pos from)) + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) + (setq pos (next-single-char-property-change pos + (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) + (while (< pos to) + (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) + pos)) + (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) + + (let (unfold?) + ;; the line before folded text should be beginning of the drawer + (save-excursion + (goto-char drawer-begin) + (backward-char) + (beginning-of-line) + (unless (looking-at-p org-drawer-regexp) + (setq unfold? t))) + ;; the last line of the folded text should be :END: + (save-excursion + (goto-char drawer-end) + (beginning-of-line) + (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re)) + (setq unfold? t))) + ;; there should be no :END: anywhere in the drawer body + (save-excursion + (goto-char drawer-begin) + (when (save-excursion + (let ((case-fold-search t)) + (re-search-forward org-property-end-re + (max (point) + (1- (save-excursion + (goto-char drawer-end) + (line-beginning-position)))) + 't))) + (setq unfold? t))) + ;; there should be no new entry anywhere in the drawer body + (save-excursion + (goto-char drawer-begin) + (when (save-excursion + (let ((case-fold-search t)) + (re-search-forward org-outline-regexp-bol + (max (point) + (1- (save-excursion + (goto-char drawer-end) + (line-beginning-position)))) + 't))) + (setq unfold? t))) + + (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer)))) + + (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))) + + ;; check folded blocks + (let ((pos from)) + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) + (setq pos (next-single-char-property-change pos + (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) + (while (< pos to) + (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) + pos)) + (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) + + (let (unfold?) + ;; the line before folded text should be beginning of the block + (save-excursion + (goto-char block-begin) + (backward-char) + (beginning-of-line) + (unless (looking-at-p org-dblock-start-re) + (setq unfold? t))) + ;; the last line of the folded text should be end of the block + (save-excursion + (goto-char block-end) + (beginning-of-line) + (unless (looking-at-p org-dblock-end-re) + (setq unfold? t))) + ;; there should be no #+end anywhere in the block body + (save-excursion + (goto-char block-begin) + (when (save-excursion + (re-search-forward org-dblock-end-re + (max (point) + (1- (save-excursion + (goto-char block-end) + (line-beginning-position)))) + 't)) + (setq unfold? t))) + ;; there should be no new entry anywhere in the block body + (save-excursion + (goto-char block-begin) + (when (save-excursion + (let ((case-fold-search t)) + (re-search-forward org-outline-regexp-bol + (max (point) + (1- (save-excursion + (goto-char block-end) + (line-beginning-position)))) + 't))) + (setq unfold? t))) + + (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block)))) + + (setq pos + (next-single-char-property-change pos + (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))))) + (defvar org-mode-map) (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. @@ -4789,6 +4937,7 @@ The following commands are available: (org-install-agenda-files-menu) (when org-link-descriptive (add-to-invisibility-spec '(org-link))) (add-to-invisibility-spec '(org-hide-block . t)) + (add-to-invisibility-spec '(org-hide-drawer . t)) (setq-local outline-regexp org-outline-regexp) (setq-local outline-level 'org-outline-level) (setq bidi-paragraph-direction 'left-to-right) @@ -4817,6 +4966,8 @@ The following commands are available: ;; Activate before-change-function (setq-local org-table-may-need-update t) (add-hook 'before-change-functions 'org-before-change-function nil 'local) + ;; Activate after-change-function + (add-hook 'after-change-functions 'org-after-change-function nil 'local) ;; Check for running clock before killing a buffer (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) ;; Initialize macros templates. @@ -4868,6 +5019,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5049,8 +5204,8 @@ stacked delimiters is N. Escaping delimiters is not possible." (when verbatim? (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 2) (match-end 2) - '(display t invisible t intangible t))) + (org-remove-text-properties (match-beginning 2) (match-end 2) + '(display t invisible t intangible t))) (add-text-properties (match-beginning 2) (match-end 2) '(font-lock-multiline t org-emphasis t)) (when (and org-hide-emphasis-markers @@ -5165,7 +5320,7 @@ This includes angle, plain, and bracket links." (if (not (eq 'bracket style)) (add-text-properties start end properties) ;; Handle invisible parts in bracket links. - (remove-text-properties start end '(invisible nil)) + (org-remove-text-properties start end '(invisible nil)) (let ((hidden (append `(invisible ,(or (org-link-get-parameter type :display) @@ -5185,8 +5340,8 @@ This includes angle, plain, and bracket links." (defun org-activate-code (limit) (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) t)) (defcustom org-src-fontify-natively t @@ -5257,8 +5412,8 @@ by a #." (setq block-end (match-beginning 0)) ; includes the final newline. (when quoting (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline) - (remove-text-properties beg end-of-endline - '(display t invisible t intangible t))) + (org-remove-text-properties beg end-of-endline + '(display t invisible t intangible t))) (add-text-properties beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) (org-remove-flyspell-overlays-in beg bol-after-beginline) @@ -5312,9 +5467,9 @@ by a #." '(font-lock-fontified t face org-document-info)))) ((string-prefix-p "+caption" dc1) (org-remove-flyspell-overlays-in (match-end 2) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) - ;; Handle short captions + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) + ;; Handle short captions. (save-excursion (beginning-of-line) (looking-at (rx (group (zero-or-more blank) @@ -5335,8 +5490,8 @@ by a #." '(font-lock-fontified t face font-lock-comment-face))) (t ;; Just any other in-buffer setting, but not indented (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) (add-text-properties beg (match-end 0) '(font-lock-fontified t face org-meta-line)) t)))))) @@ -5721,35 +5876,59 @@ needs to be inserted at a specific position in the font-lock sequence.") (decompose-region (point-min) (point-max)) (message "Entities are now displayed as plain text")))) -(defvar-local org-custom-properties-overlays nil - "List of overlays used for custom properties.") +(defvar-local org-custom-properties-hidden-p nil + "Non-nil when custom properties are hidden.") + +(defcustom org-custom-properties-hide-emptied-drawers nil + "Non-nil means that drawers containing only `org-custom-properties' will be hidden together with the properties." + :group 'org + :type '(choice + (const :tag "Don't hide emptied drawers" nil) + (const :tag "Hide emptied drawers" t))) (defun org-toggle-custom-properties-visibility () "Display or hide properties in `org-custom-properties'." (interactive) - (if org-custom-properties-overlays - (progn (mapc #'delete-overlay org-custom-properties-overlays) - (setq org-custom-properties-overlays nil)) + (require 'org-macs) + (add-to-invisibility-spec 'org-hide-custom-property) + (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property) + (if org-custom-properties-hidden-p + (let (match) + (setq org-custom-properties-hidden-p nil) + (org-with-wide-buffer + (goto-char (point-min)) + (with-silent-modifications + (while (setq match (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t)) + (org-flag-region (prop-match-beginning match) + (prop-match-end match) + nil 'org-hide-custom-property))))) (when org-custom-properties + (setq org-custom-properties-hidden-p t) (org-with-wide-buffer - (goto-char (point-min)) - (let ((regexp (org-re-property (regexp-opt org-custom-properties) t t))) + (let* ((regexp (org-re-property (regexp-opt org-custom-properties) t t)) + (regexp-drawer (format "%s\n\\(?:%s\\)+\n%s" + (replace-regexp-in-string "\\$$" "" org-drawer-regexp) + (replace-regexp-in-string "\\(^\\^\\|\\$$\\)" "" regexp) + (replace-regexp-in-string "^\\^" "" org-property-end-re)))) + + (when org-custom-properties-hide-emptied-drawers + (goto-char (point-min)) + (while (re-search-forward regexp-drawer nil t) + (with-silent-modifications + (org-flag-region (1- (match-beginning 0)) (match-end 0) t 'org-hide-custom-property)))) + + (goto-char (point-min)) (while (re-search-forward regexp nil t) (let ((end (cdr (save-match-data (org-get-property-block))))) (when (and end (< (point) end)) ;; Hide first custom property in current drawer. - (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0))))) - (overlay-put o 'invisible t) - (overlay-put o 'org-custom-property t) - (push o org-custom-properties-overlays)) - ;; Hide additional custom properties in the same drawer. - (while (re-search-forward regexp end t) - (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0))))) - (overlay-put o 'invisible t) - (overlay-put o 'org-custom-property t) - (push o org-custom-properties-overlays))))) - ;; Each entry is limited to a single property drawer. - (outline-next-heading))))))) + (with-silent-modifications + (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property) + ;; Hide additional custom properties in the same drawer. + (while (re-search-forward regexp end t) + (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property)))))) + ;; Each entry is limited to a single property drawer. + (outline-next-heading)))))) (defun org-fontify-entities (limit) "Find an entity to fontify." @@ -5858,10 +6037,11 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) - (remove-text-properties beg end - '(mouse-face t keymap t org-linked-text t - invisible t intangible t - org-emphasis t)) + (org-remove-text-properties beg end + '(mouse-face t keymap t org-linked-text t + invisible t + intangible t + org-emphasis t)) (org-remove-font-lock-display-properties beg end))) (defconst org-script-display '(((raise -0.3) (height 0.7)) @@ -5969,6 +6149,29 @@ open and agenda-wise Org files." ;;;; Headlines visibility +(defun org-hide-entry () + "Hide the body directly following this heading." + (interactive) + (save-excursion + (outline-back-to-heading) + (outline-end-of-heading) + (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline))) + +(defun org-hide-subtree () + "Hide everything after this heading at deeper levels." + (interactive) + (org-flag-subtree t)) + +(defun org-hide-sublevels (levels) + "Hide everything but the top LEVELS levels of headers, in whole buffer. +This also unhides the top heading-less body, if any. + +Interactively, the prefix argument supplies the value of LEVELS. +When invoked without a prefix argument, LEVELS defaults to the level +of the current heading, or to 1 if the current line is not a heading." + (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region)) + (org-hide-sublevels levels))) + (defun org-show-entry () "Show the body directly following this heading. Show the heading too, if it is currently invisible." @@ -5980,13 +6183,24 @@ Show the heading too, if it is currently invisible." (line-end-position 0) (save-excursion (if (re-search-forward - (concat "[\r\n]\\(" org-outline-regexp "\\)") nil t) + (concat "[\r\n]\\(" (org-get-limited-outline-regexp) "\\)") nil t) (match-beginning 1) (point-max))) nil 'outline) (org-cycle-hide-drawers 'children)))) +(defun org-show-heading () + "Show the current heading and move to its end." + (org-flag-region (- (point) + (if (bobp) 0 + (if (and outline-blank-line + (eq (char-before (1- (point))) ?\n)) + 2 1))) + (progn (outline-end-of-heading) (point)) + nil + 'outline)) + (defun org-show-children (&optional level) "Show all direct subheadings of this heading. Prefix arg LEVEL is how many levels below the current level @@ -6030,6 +6244,11 @@ heading to appear." (org-flag-region (point) (save-excursion (org-end-of-subtree t t)) nil 'outline)) +(defun org-show-branches () + "Show all subheadings of this heading, but not their bodies." + (interactive) + (org-show-children 1000)) + ;;;; Blocks and drawers visibility (defun org--hide-wrapper-toggle (element category force no-error) @@ -6062,7 +6281,9 @@ Return a non-nil value when toggling is successful." ;; at the block closing line. (unless (let ((eol (line-end-position))) (and (> eol start) (/= eol end))) - (let* ((spec (if (eq category 'block) 'org-hide-block 'outline)) + (let* ((spec (cond ((eq category 'block) 'org-hide-block) + ((eq category 'drawer) 'org-hide-drawer) + (t 'outline))) (flag (cond ((eq force 'off) nil) (force t) @@ -6115,24 +6336,24 @@ Return a non-nil value when toggling is successful." (defun org-hide-drawer-all () "Fold all drawers in the current buffer." + (org-show-all '(drawers)) (save-excursion (goto-char (point-min)) (while (re-search-forward org-drawer-regexp nil t) - (let* ((pair (get-char-property-and-overlay (line-beginning-position) - 'invisible)) - (o (cdr-safe pair))) - (if (overlayp o) (goto-char (overlay-end o)) ;invisible drawer - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) (goto-char (overlay-end o))) ;already folded - (_ - (let* ((drawer (org-element-at-point)) - (type (org-element-type drawer))) - (when (memq type '(drawer property-drawer)) - (org-hide-drawer-toggle t nil drawer) - ;; Make sure to skip drawer entirely or we might flag it - ;; another time when matching its ending line with - ;; `org-drawer-regexp'. - (goto-char (org-element-property :end drawer))))))))))) + (let* ((drawer (org-element-at-point)) + (type (org-element-type drawer))) + (when (memq type '(drawer property-drawer)) + ;; We are sure regular drawers are unfolded because of + ;; `org-show-all' call above. However, property drawers may + ;; be folded, or in a folded headline. In that case, do not + ;; re-hide it. + (unless (and (eq type 'property-drawer) + (eq 'org-hide-drawer (get-char-property (point) 'invisible))) + (org-hide-drawer-toggle t nil drawer)) + ;; Make sure to skip drawer entirely or we might flag it + ;; another time when matching its ending line with + ;; `org-drawer-regexp'. + (goto-char (org-element-property :end drawer))))))) (defun org-cycle-hide-drawers (state) "Re-hide all drawers after a visibility state change. @@ -6147,9 +6368,10 @@ STATE should be one of the symbols listed in the docstring of (t (save-excursion (org-end-of-subtree t t)))))) (org-with-point-at beg (while (re-search-forward org-drawer-regexp end t) - (pcase (get-char-property-and-overlay (point) 'invisible) + (pcase (get-char-property (point) 'invisible) ;; Do not fold already folded drawers. - (`(outline . ,o) (goto-char (overlay-end o))) + ('outline + (goto-char (min end (next-single-char-property-change (point) 'invisible)))) (_ (let ((drawer (org-element-at-point))) (when (memq (org-element-type drawer) '(drawer property-drawer)) @@ -6172,31 +6394,13 @@ By default, the function expands headings, blocks and drawers. When optional argument TYPE is a list of symbols among `blocks', `drawers' and `headings', to only expand one specific type." (interactive) - (let ((types (or types '(blocks drawers headings)))) - (when (memq 'blocks types) - (org-flag-region (point-min) (point-max) nil 'org-hide-block)) - (cond - ;; Fast path. Since headings and drawers share the same - ;; invisible spec, clear everything in one go. - ((and (memq 'headings types) - (memq 'drawers types)) - (org-flag-region (point-min) (point-max) nil 'outline)) - ((memq 'headings types) - (org-flag-region (point-min) (point-max) nil 'outline) - (org-cycle-hide-drawers 'all)) - ((memq 'drawers types) - (save-excursion - (goto-char (point-min)) - (while (re-search-forward org-drawer-regexp nil t) - (let* ((pair (get-char-property-and-overlay (line-beginning-position) - 'invisible)) - (o (cdr-safe pair))) - (if (overlayp o) (goto-char (overlay-end o)) - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (goto-char (overlay-end o)) - (delete-overlay o)) - (_ nil)))))))))) + (dolist (type (or types '(blocks drawers headings))) + (org-flag-region (point-min) (point-max) nil + (pcase type + (`blocks 'org-hide-block) + (`drawers 'org-hide-drawer) + (`headings 'outline) + (_ (error "Invalid type: %S" type)))))) ;;;###autoload (defun org-cycle (&optional arg) @@ -6552,7 +6756,7 @@ With a numeric prefix, show all headlines up to that level." (org-narrow-to-subtree) (org-content)))) ((or "all" "showall") - (outline-show-subtree)) + (org-show-subtree)) (_ nil))) (org-end-of-subtree))))))) @@ -6625,7 +6829,7 @@ This function is the default value of the hook `org-cycle-hook'." (while (re-search-forward re nil t) (when (and (not (org-invisible-p)) (org-invisible-p (line-end-position))) - (outline-hide-entry)))) + (org-hide-entry)))) (org-cycle-hide-drawers 'all) (org-cycle-show-empty-lines 'overview))))) @@ -6697,10 +6901,11 @@ information." (org-show-entry) ;; If point is hidden within a drawer or a block, make sure to ;; expose it. - (dolist (o (overlays-at (point))) - (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) - (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -6916,9 +7121,10 @@ unconditionally." ;; When INVISIBLE-OK is non-nil, ensure newly created headline ;; is visible. (unless invisible-ok - (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (move-overlay o (overlay-start o) (line-end-position 0))) + (pcase (get-char-property (point) 'invisible) + ('outline + (let ((region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (line-end-position 0) (cdr region) nil 'outline))) (_ nil)))) ;; At a headline... ((org-at-heading-p) @@ -7515,7 +7721,6 @@ case." (setq txt (buffer-substring beg end)) (org-save-markers-in-region beg end) (delete-region beg end) - (org-remove-empty-overlays-at beg) (unless (= beg (point-min)) (org-flag-region (1- beg) beg nil 'outline)) (unless (bobp) (org-flag-region (1- (point)) (point) nil 'outline)) (and (not (bolp)) (looking-at "\n") (forward-char 1)) @@ -7677,7 +7882,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard." (skip-chars-forward " \t\n\r") (setq beg (point)) (when (and (org-invisible-p) visp) - (save-excursion (outline-show-heading))) + (save-excursion (org-show-heading))) ;; Shift if necessary. (unless (= shift 0) (save-restriction @@ -8119,7 +8324,7 @@ function is being called interactively." (point)) what "children") (goto-char start) - (outline-show-subtree) + (org-show-subtree) (outline-next-heading)) (t ;; we will sort the top-level entries in this file @@ -10736,7 +10941,8 @@ narrowing." (let ((beg (point))) (insert ":" drawer ":\n:END:\n") (org-indent-region beg (point)) - (org-flag-region (line-end-position -1) (1- (point)) t 'outline)) + (org-flag-region + (line-end-position -1) (1- (point)) t 'org-hide-drawer)) (end-of-line -1))))) (t (org-end-of-meta-data org-log-state-notes-insert-after-drawers) @@ -13173,7 +13379,7 @@ drawer is immediately hidden." (inhibit-read-only t)) (unless (bobp) (insert "\n")) (insert ":PROPERTIES:\n:END:") - (org-flag-region (line-end-position 0) (point) t 'outline) + (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer) (when (or (eobp) (= begin (point-min))) (insert "\n")) (org-indent-region begin (point)))))) @@ -16553,7 +16759,7 @@ The detailed reaction depends on the user option `org-catch-invisible-edits'." (when (or invisible-at-point invisible-before-point) (when (eq org-catch-invisible-edits 'error) (user-error "Editing in invisible areas is prohibited, make them visible first")) - (if (and org-custom-properties-overlays + (if (and org-custom-properties-hidden-p (y-or-n-p "Display invisible properties in this buffer? ")) (org-toggle-custom-properties-visibility) ;; Make the area visible @@ -17636,11 +17842,11 @@ Move point to the beginning of first heading or end of buffer." (defun org-show-branches-buffer () "Show all branches in the buffer." (org-flag-above-first-heading) - (outline-hide-sublevels 1) + (org-hide-sublevels 1) (unless (eobp) - (outline-show-branches) + (org-show-branches) (while (outline-get-next-sibling) - (outline-show-branches))) + (org-show-branches))) (goto-char (point-min))) (defun org-kill-note-or-show-branches () @@ -17654,8 +17860,8 @@ Move point to the beginning of first heading or end of buffer." (t (let ((beg (progn (org-back-to-heading) (point))) (end (save-excursion (org-end-of-subtree t t) (point)))) - (outline-hide-subtree) - (outline-show-branches) + (org-hide-subtree) + (org-show-branches) (org-hide-archived-subtrees beg end))))) (defun org-delete-indentation (&optional arg) @@ -17811,9 +18017,9 @@ Otherwise, call `org-show-children'. ARG is the level to hide." (if (org-before-first-heading-p) (progn (org-flag-above-first-heading) - (outline-hide-sublevels (or arg 1)) + (org-hide-sublevels (or arg 1)) (goto-char (point-min))) - (outline-hide-subtree) + (org-hide-subtree) (org-show-children arg)))) (defun org-ctrl-c-star () @@ -20489,20 +20695,20 @@ With ARG, repeats or can move backward if negative." (end-of-line)) (while (and (< arg 0) (re-search-backward regexp nil :move)) (unless (bobp) - (while (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (goto-char (overlay-start o)) - (re-search-backward regexp nil :move)) - (_ nil)))) + (pcase (get-char-property (point) 'invisible) + ('outline + (goto-char (car (org--find-text-property-region (point) 'invisible))) + (beginning-of-line)) + (_ nil))) (cl-incf arg)) - (while (and (> arg 0) (re-search-forward regexp nil t)) - (while (pcase (get-char-property-and-overlay (point) 'invisible) - (`(outline . ,o) - (goto-char (overlay-end o)) - (re-search-forward regexp nil :move)) - (_ - (end-of-line) - nil))) ;leave the loop + (while (and (> arg 0) (re-search-forward regexp nil :move)) + (pcase (get-char-property (point) 'invisible) + ('outline + (goto-char (cdr (org--find-text-property-region (point) 'invisible))) + (skip-chars-forward " \t\n") + (end-of-line)) + (_ + (end-of-line))) (cl-decf arg)) (if (> arg 0) (goto-char (point-max)) (beginning-of-line)))) @@ -20957,6 +21163,80 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) \f + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + (setq spec (get-text-property pos 'invisible)) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (org-flag-region (car region) (cdr region) nil spec))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (org-flag-region (overlay-start ov) (overlay-end ov) t spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + +\f + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode [-- Attachment #3: Type: text/plain, Size: 12921 bytes --] Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > > [The patch itself will be provided in the following email] > > I have four more updates from the previous version of the patch: > > 1. All the code handling modifications in folded drawers/blocks is moved > to after-change-function. It works as follows: > - if any text is inserted in the middle of hidden region, that text > is also hidden; > - if BEGIN/END line of a folded drawer do not match org-drawer-regexp > and org-property-end-re, unfold it; > - if org-property-end-re or new org-outline-regexp-bol is inserted in > the middle of the drawer, unfold it; > - the same logic for blocks. > > 2. The text property stack is rewritten using char-property-alias-alist. > This is faster in comparison with previous approach, which involved > modifying all the text properties every timer org-flag-region was > called. > > 3. org-toggle-custom-properties-visibility is rewritten using text > properties. I also took a freedom to implement a new feature here. > Now, setting new `org-custom-properties-hide-emptied-drawers' to > non-nil will result in hiding the whole property drawer if it > contains only org-custom-properties. > > 4. This patch should work against 1aa095ccf. However, the merge was not > trivial here. Recent commits actively used the fact that drawers and > outlines are hidden via 'outline invisibility spec, which is not the > case in this branch. I am not confident that I did not break anything > during the merge, especially 1aa095ccf. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > >> I gave you a few ideas to quickly check if a change requires expansion, >> in an earlier mail. I suggest to start out from that. Let me know if you >> have questions about it. > > All the code lives in org-after-change-function. I tried to incorporate > the earlier Nicholas' suggestions, except the parts related to > intersecting blocks and drawers. I am not sure if I understand the > parsing priority of blocks vs. drawers. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the text property stack: > > The earlier version of the code literally used stack to save > pre-existing 'invisibility specs in org-flag-region. This was done on > every invocation of org-flag-region, which made org-flag-region > significantly slower. I re-implemented the same feature using > char-property-alias-alist. Now, different invisibility specs live in > separate text properties and can be safely modified independently. The > specs are applied according to org--invisible-spec-priority-list. A side > effect of current implementation is that char-property-alias-alist is > fully controlled by org. All the pre-existing settings for 'invisible > text property will be overwritten by org. > >> `gensym' is just a shorter, and somewhat standard way, to create a new >> uninterned symbol with a given prefix. You seem to re-invent it. What >> you do with that new symbol is orthogonal to that suggestion, of course. > > I do not think that `gensym' is suitable here. We don't want a new > symbol every time org--get-buffer-local-invisible-property-symbol is > called. It should return the same symbol if it is called from the same > buffer multiple times. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the org-toggle-custom-properties-visibility: > > The implementation showcases how to introduce new invisibility specs to > org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) > one also needs to add the spec into org--invisible-spec-priority-list: > > (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property) > > Searching for text with the given invisibility spec is done as > follows: > > (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t) > > This last piece of code is probably not the most elegant. I am thinking > if creating some higher-level interface would be more reasonable here. > What do you think? > > > The new customisation `org-custom-properties-hide-emptied-drawers' > sounds logical for me since empty property drawers left after invoking > org-toggle-custom-properties-visibility are rather useless according to > my experience. If one already wants to hide parts of property drawers, I > do not see a reason to show leftover > > :PROPERTIES: > :END: > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the merge with the latest master: > > I tried my best to not break anything. However, I am not sure if I > understand all the recent commits. Could someone take a look if there is > anything suspicious in org-next-visible-heading? > > Also, I have seen some optimisations making use of the fact that drawers > and headlines both use 'outline invisibility spec. This change in the > implementation details supposed to improve performance and should not be > necessary if this patch is going to be merged. Would it be possible to > refrain from abusing this particular implementation detail in the > nearest commits on master (unless really necessary)? > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > I would like to finalise the current patch and work on other code using > overlays separately. This patch is already quite complicated as is. I do > not want to introduce even more potential bugs by working on things not > directly affected by this version of the patch. > > Best, > Ihor > > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>>> See also `gensym'. Do we really need to use it for something else than >>>> `invisible'? If not, the tool doesn't need to be generic. >>> >>> For now, I also use it for buffer-local 'invisible stack. The stack is >>> needed to preserve folding state of drawers/blocks inside folded >>> outline. Though I am thinking about replacing the stack with separate >>> text properties, like 'invisible-outline-buffer-local + >>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local. >>> Maintaining stack takes a noticeable percentage of CPU time in profiler. >>> >>> org--get-buffer-local-text-property-symbol must take care about >>> situation with indirect buffers. When an indirect buffer is created from >>> some org buffer, the old value of char-property-alias-alist is carried >>> over. We need to detect this case and create new buffer-local symbol, >>> which is unique to the newly created buffer (but not create it if the >>> buffer-local property is already there). Then, the new symbol must >>> replace the old alias in char-property-alias-alist + old folding state >>> must be preserved (via copying the old invisibility specs into the new >>> buffer-local text property). I do not see how gensym can benefit this >>> logic. >> >> `gensym' is just a shorter, and somewhat standard way, to create a new >> uninterned symbol with a given prefix. You seem to re-invent it. What >> you do with that new symbol is orthogonal to that suggestion, of course. >> >>>> OK, but this may not be sufficient if we want to do slightly better than >>>> overlays in that area. This is not mandatory, though. >>> >>> Could you elaborate on what can be "slightly better"? >> >> IIRC, I gave examples of finer control of folding state after a change. >> Consider this _folded_ drawer: >> >> :BEGIN: >> Foo >> :END: >> >> Inserting ":END" in it should not unfold it, as it is currently the case >> with overlays, >> >> :BEGIN >> Foo >> :END >> :END: >> >> but a soon as the last ":" is inserted, the initial drawer could be >> expanded. >> >> :BEGIN >> Foo >> :END: >> :END: >> >> The latter case is not currently handled by overlays. This is what >> I call "slightly better". >> >> Also, note that this change is not related to opening and closing lines >> of the initial drawer, so sticking text properties on them would not >> help here. >> >> Another case is modifying those borders, e.g., >> >> >> :BEGIN: :BEGIN: >> Foo ------> Foo >> :END: :ND: >> >> which should expand the drawer. Your implementation catches this, but >> I'm pointing out that current implementation with overlays does not. >> Even though that's not strictly required for compatibility with >> overlays, it is a welcome slight improvement. >> >>>> As discussed before, I don't think you need to use `modification-hooks' >>>> or `insert-behind-hooks' if you already use `after-change-functions'. >>>> >>>> `after-change-functions' are also triggered upon text properties >>>> changes. So, what is the use case for the other hooks? >>> >>> The problem is that `after-change-functions' cannot be a text property. >>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a >>> valid text property. If we use `after-change-functions', they will >>> always be triggered, regardless if the change was made inside or outside >>> folded region. >> >> As discussed, text properties are local to the change, but require extra >> care when moving text around. You also observed serious overhead when >> using them. >> >> OTOH, even if `a-c-f' is not local, you can quickly determine if the >> change altered a folded element, so the overhead is limited, i.e., >> mostly checking for a text property at a given buffer position. >> >> To be clear, I initially thought that text properties were a superior >> choice, but I changed my mind a while ago, and I thought you had, too. >> IOW, `after-change-functions' is the way to go, since you have no strong >> reason to stick to text properties for this kind of function. >> >>>>> :asd: >>>>> :drawer: >>>>> lksjdfksdfjl >>>>> sdfsdfsdf >>>>> :end: >>>>> >>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >>>>> line of the new folded :asd: drawer would reveal the text between >>>>> :drawer: and :end:. >>>>> >>>>> Let me know what you think on this. >>> >>>> I have first to understand the use case for `modification-hook'. But >>>> I think unfolding is the right thing to do in this situation, isn't it? >>> >>> That situation arises because the modification-hooks from ":drawer:" >>> (they are set via text properties) only have information about the >>> :drawer:...:end: drawer before the modifications (they were set when >>> :drawer: was folded last time). So, they will only unfold a part of the >>> new :asd: drawer. I do not see a simple way to unfold everything without >>> re-parsing the drawer around the changed text. >> >> Oh! I misread your message. I withdraw what I wrote. In this case, we >> don't want to unfold anything. The situation is not worse than what we >> have now, and trying to fix it would have repercussions down in the >> buffer, e.g., expanding drawers screen below. >> >> As a rule of thumb, I think we can pay attention to changes in the >> folded text, and its immediate surroundings (e.g., the opening line, >> which is not folded), but no further. >> >> As written above, slight changes are welcome, but let's not go overboard >> and parse a whole section just to know if we can expand a drawer. >> >>> Actually, I am quite unhappy with the performance of modification-hooks >>> set via text properties (I am using this patch on my Emacs during this >>> week). It appears that setting the text properties costs a significant >>> CPU time in practice, even though running the hooks is pretty fast. >>> I will think about a way to handle modifications using global >>> after-change-functions. >> >> That's better, IMO. >> >> I gave you a few ideas to quickly check if a change requires expansion, >> in an earlier mail. I suggest to start out from that. Let me know if you >> have questions about it. > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply related [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-08 5:05 ` Ihor Radchenko 2020-06-08 5:06 ` Ihor Radchenko @ 2020-06-08 5:08 ` Ihor Radchenko 2020-06-10 17:14 ` Nicolas Goaziou 2 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-06-08 5:08 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Github link to the patch: https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > > [The patch itself will be provided in the following email] > > I have four more updates from the previous version of the patch: > > 1. All the code handling modifications in folded drawers/blocks is moved > to after-change-function. It works as follows: > - if any text is inserted in the middle of hidden region, that text > is also hidden; > - if BEGIN/END line of a folded drawer do not match org-drawer-regexp > and org-property-end-re, unfold it; > - if org-property-end-re or new org-outline-regexp-bol is inserted in > the middle of the drawer, unfold it; > - the same logic for blocks. > > 2. The text property stack is rewritten using char-property-alias-alist. > This is faster in comparison with previous approach, which involved > modifying all the text properties every timer org-flag-region was > called. > > 3. org-toggle-custom-properties-visibility is rewritten using text > properties. I also took a freedom to implement a new feature here. > Now, setting new `org-custom-properties-hide-emptied-drawers' to > non-nil will result in hiding the whole property drawer if it > contains only org-custom-properties. > > 4. This patch should work against 1aa095ccf. However, the merge was not > trivial here. Recent commits actively used the fact that drawers and > outlines are hidden via 'outline invisibility spec, which is not the > case in this branch. I am not confident that I did not break anything > during the merge, especially 1aa095ccf. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > >> I gave you a few ideas to quickly check if a change requires expansion, >> in an earlier mail. I suggest to start out from that. Let me know if you >> have questions about it. > > All the code lives in org-after-change-function. I tried to incorporate > the earlier Nicholas' suggestions, except the parts related to > intersecting blocks and drawers. I am not sure if I understand the > parsing priority of blocks vs. drawers. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the text property stack: > > The earlier version of the code literally used stack to save > pre-existing 'invisibility specs in org-flag-region. This was done on > every invocation of org-flag-region, which made org-flag-region > significantly slower. I re-implemented the same feature using > char-property-alias-alist. Now, different invisibility specs live in > separate text properties and can be safely modified independently. The > specs are applied according to org--invisible-spec-priority-list. A side > effect of current implementation is that char-property-alias-alist is > fully controlled by org. All the pre-existing settings for 'invisible > text property will be overwritten by org. > >> `gensym' is just a shorter, and somewhat standard way, to create a new >> uninterned symbol with a given prefix. You seem to re-invent it. What >> you do with that new symbol is orthogonal to that suggestion, of course. > > I do not think that `gensym' is suitable here. We don't want a new > symbol every time org--get-buffer-local-invisible-property-symbol is > called. It should return the same symbol if it is called from the same > buffer multiple times. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the org-toggle-custom-properties-visibility: > > The implementation showcases how to introduce new invisibility specs to > org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) > one also needs to add the spec into org--invisible-spec-priority-list: > > (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property) > > Searching for text with the given invisibility spec is done as > follows: > > (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t) > > This last piece of code is probably not the most elegant. I am thinking > if creating some higher-level interface would be more reasonable here. > What do you think? > > > The new customisation `org-custom-properties-hide-emptied-drawers' > sounds logical for me since empty property drawers left after invoking > org-toggle-custom-properties-visibility are rather useless according to > my experience. If one already wants to hide parts of property drawers, I > do not see a reason to show leftover > > :PROPERTIES: > :END: > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the merge with the latest master: > > I tried my best to not break anything. However, I am not sure if I > understand all the recent commits. Could someone take a look if there is > anything suspicious in org-next-visible-heading? > > Also, I have seen some optimisations making use of the fact that drawers > and headlines both use 'outline invisibility spec. This change in the > implementation details supposed to improve performance and should not be > necessary if this patch is going to be merged. Would it be possible to > refrain from abusing this particular implementation detail in the > nearest commits on master (unless really necessary)? > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > I would like to finalise the current patch and work on other code using > overlays separately. This patch is already quite complicated as is. I do > not want to introduce even more potential bugs by working on things not > directly affected by this version of the patch. > > Best, > Ihor > > > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>>> See also `gensym'. Do we really need to use it for something else than >>>> `invisible'? If not, the tool doesn't need to be generic. >>> >>> For now, I also use it for buffer-local 'invisible stack. The stack is >>> needed to preserve folding state of drawers/blocks inside folded >>> outline. Though I am thinking about replacing the stack with separate >>> text properties, like 'invisible-outline-buffer-local + >>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local. >>> Maintaining stack takes a noticeable percentage of CPU time in profiler. >>> >>> org--get-buffer-local-text-property-symbol must take care about >>> situation with indirect buffers. When an indirect buffer is created from >>> some org buffer, the old value of char-property-alias-alist is carried >>> over. We need to detect this case and create new buffer-local symbol, >>> which is unique to the newly created buffer (but not create it if the >>> buffer-local property is already there). Then, the new symbol must >>> replace the old alias in char-property-alias-alist + old folding state >>> must be preserved (via copying the old invisibility specs into the new >>> buffer-local text property). I do not see how gensym can benefit this >>> logic. >> >> `gensym' is just a shorter, and somewhat standard way, to create a new >> uninterned symbol with a given prefix. You seem to re-invent it. What >> you do with that new symbol is orthogonal to that suggestion, of course. >> >>>> OK, but this may not be sufficient if we want to do slightly better than >>>> overlays in that area. This is not mandatory, though. >>> >>> Could you elaborate on what can be "slightly better"? >> >> IIRC, I gave examples of finer control of folding state after a change. >> Consider this _folded_ drawer: >> >> :BEGIN: >> Foo >> :END: >> >> Inserting ":END" in it should not unfold it, as it is currently the case >> with overlays, >> >> :BEGIN >> Foo >> :END >> :END: >> >> but a soon as the last ":" is inserted, the initial drawer could be >> expanded. >> >> :BEGIN >> Foo >> :END: >> :END: >> >> The latter case is not currently handled by overlays. This is what >> I call "slightly better". >> >> Also, note that this change is not related to opening and closing lines >> of the initial drawer, so sticking text properties on them would not >> help here. >> >> Another case is modifying those borders, e.g., >> >> >> :BEGIN: :BEGIN: >> Foo ------> Foo >> :END: :ND: >> >> which should expand the drawer. Your implementation catches this, but >> I'm pointing out that current implementation with overlays does not. >> Even though that's not strictly required for compatibility with >> overlays, it is a welcome slight improvement. >> >>>> As discussed before, I don't think you need to use `modification-hooks' >>>> or `insert-behind-hooks' if you already use `after-change-functions'. >>>> >>>> `after-change-functions' are also triggered upon text properties >>>> changes. So, what is the use case for the other hooks? >>> >>> The problem is that `after-change-functions' cannot be a text property. >>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a >>> valid text property. If we use `after-change-functions', they will >>> always be triggered, regardless if the change was made inside or outside >>> folded region. >> >> As discussed, text properties are local to the change, but require extra >> care when moving text around. You also observed serious overhead when >> using them. >> >> OTOH, even if `a-c-f' is not local, you can quickly determine if the >> change altered a folded element, so the overhead is limited, i.e., >> mostly checking for a text property at a given buffer position. >> >> To be clear, I initially thought that text properties were a superior >> choice, but I changed my mind a while ago, and I thought you had, too. >> IOW, `after-change-functions' is the way to go, since you have no strong >> reason to stick to text properties for this kind of function. >> >>>>> :asd: >>>>> :drawer: >>>>> lksjdfksdfjl >>>>> sdfsdfsdf >>>>> :end: >>>>> >>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer: >>>>> line of the new folded :asd: drawer would reveal the text between >>>>> :drawer: and :end:. >>>>> >>>>> Let me know what you think on this. >>> >>>> I have first to understand the use case for `modification-hook'. But >>>> I think unfolding is the right thing to do in this situation, isn't it? >>> >>> That situation arises because the modification-hooks from ":drawer:" >>> (they are set via text properties) only have information about the >>> :drawer:...:end: drawer before the modifications (they were set when >>> :drawer: was folded last time). So, they will only unfold a part of the >>> new :asd: drawer. I do not see a simple way to unfold everything without >>> re-parsing the drawer around the changed text. >> >> Oh! I misread your message. I withdraw what I wrote. In this case, we >> don't want to unfold anything. The situation is not worse than what we >> have now, and trying to fix it would have repercussions down in the >> buffer, e.g., expanding drawers screen below. >> >> As a rule of thumb, I think we can pay attention to changes in the >> folded text, and its immediate surroundings (e.g., the opening line, >> which is not folded), but no further. >> >> As written above, slight changes are welcome, but let's not go overboard >> and parse a whole section just to know if we can expand a drawer. >> >>> Actually, I am quite unhappy with the performance of modification-hooks >>> set via text properties (I am using this patch on my Emacs during this >>> week). It appears that setting the text properties costs a significant >>> CPU time in practice, even though running the hooks is pretty fast. >>> I will think about a way to handle modifications using global >>> after-change-functions. >> >> That's better, IMO. >> >> I gave you a few ideas to quickly check if a change requires expansion, >> in an earlier mail. I suggest to start out from that. Let me know if you >> have questions about it. > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-08 5:05 ` Ihor Radchenko 2020-06-08 5:06 ` Ihor Radchenko 2020-06-08 5:08 ` Ihor Radchenko @ 2020-06-10 17:14 ` Nicolas Goaziou 2020-06-21 9:52 ` Ihor Radchenko 2020-08-11 6:45 ` Ihor Radchenko 2 siblings, 2 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-06-10 17:14 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > [The patch itself will be provided in the following email] Thank you! I'll first make some generic remarks, then comment the diff in more details. > I have four more updates from the previous version of the patch: > > 1. All the code handling modifications in folded drawers/blocks is moved > to after-change-function. It works as follows: > - if any text is inserted in the middle of hidden region, that text > is also hidden; > - if BEGIN/END line of a folded drawer do not match org-drawer-regexp > and org-property-end-re, unfold it; > - if org-property-end-re or new org-outline-regexp-bol is inserted in > the middle of the drawer, unfold it; > - the same logic for blocks. This sounds good, barring a minor error in the regexp for blocks, and missing optimizations. More on this in the detailed comments. > 2. The text property stack is rewritten using char-property-alias-alist. > This is faster in comparison with previous approach, which involved > modifying all the text properties every timer org-flag-region was > called. I'll need information about this, as I'm not sure to fully understand all the consequences of this. But more importantly, this needs to be copiously documented somewhere for future hackers. > 3. org-toggle-custom-properties-visibility is rewritten using text > properties. I also took a freedom to implement a new feature here. > Now, setting new `org-custom-properties-hide-emptied-drawers' to > non-nil will result in hiding the whole property drawer if it > contains only org-custom-properties. I don't think this is a good idea. AFAIR, we always refused to hide completely anything, including empty drawers. The reason is that if the drawer is completely hidden, you cannot expand it easily, or even know there is one. In any case, this change shouldn't belong to this patch set, and should be discussed separately. > 4. This patch should work against 1aa095ccf. However, the merge was not > trivial here. Recent commits actively used the fact that drawers and > outlines are hidden via 'outline invisibility spec, which is not the > case in this branch. I am not confident that I did not break anything > during the merge, especially 1aa095ccf. [...] > Also, I have seen some optimisations making use of the fact that drawers > and headlines both use 'outline invisibility spec. This change in the > implementation details supposed to improve performance and should not be > necessary if this patch is going to be merged. Would it be possible to > refrain from abusing this particular implementation detail in the > nearest commits on master (unless really necessary)? To be clear, I didn't intend to make your life miserable. However, I had to fix regression on drawers visibility before Org 9.4 release. Also, merging invisibility properties for drawers and outline was easier for me. So, I had the opportunity to kill two birds with one stone. As a reminder, Org 9.4 is about to be released, but Org 9.5 will take months to go out. So, even though I hope your changes will land into Org, there is no reason for us to refrain from improving (actually fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such changes are not expected to happen anymore. I hope you understand. > I would like to finalise the current patch and work on other code using > overlays separately. This patch is already quite complicated as is. I do > not want to introduce even more potential bugs by working on things not > directly affected by this version of the patch. The patch is technically mostly good, but needs more work for integration into Org. First, it includes a few unrelated changes that should be removed (e.g., white space fixes in unrelated parts of the code). Also, as written above, the changes about `org-custom-properties-hide-emptied-drawers' should be removed for the time being. Once done, I think we should move (or copy, first) _all_ folding-related functions into a new "org-fold.el" library. Functions and variables included there should have a proper "org-fold-" prefix. More on this in the detailed report. The functions `org-find-text-property-region', `org-add-to-list-text-property', and `org-remove-from-list-text-property' can be left in "org-macs.el", since they do not directly depend on the `invisible' property. Note the last two functions I mentioned are not used throughout your patch. They might be removed. This first patch can coexist with overlay folding since functions in both mechanisms are named differently. Then, another patch can integrate "org-fold.el" into Org folding. I also suggest to move the Outline -> Org transition to yet another patch. I think there's more work to do on this part. Now, into the details of your patch. The first remarks are: 1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not sure), so some functions cannot be used. 2. we don't use "subr-x.el" in the code base. In particular, it would be nice to replace `when-let' with `when' + `let'. This change costs only one loc. 3. Some docstrings need more work. In particular, Emacs documentation expects all arguments to be explained in the docstring, if possible in the order in which they appear. There are exceptions, though. For example, in a function like `org-remove-text-properties', you can mention arguments are simply the same as in `remove-text-properties'. 4. Some refactorization is needed in some places. I mentioned them in the report below. 5. I didn't dive much into the Isearch code so far. I tested it a bit and seems to work nicely. I noticed one bug though. In the following document: #+begin: foo :FOO: bar :END: #+end bar when both the drawer and the block are folded (i.e., you fold the drawer first, then the block), searching for "bar" first find the last one, then overwraps and find the first one. 6. Since we're rewriting folding code, we might as well rename folding properties: org-hide-drawer -> org-fold-drawer, outline -> org-fold-headline… Now, here are more comments about the code. ----- > +(defun org-remove-text-properties (start end properties &optional object) IMO, this generic name doesn't match the specialized nature of the function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library. > + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. Line is too long. Suggestion: Remove text properties except folding-related ones. > +Do not remove invisible text properties specified by 'outline, > +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this > +is needed to keep outlines, drawers, and blocks hidden unless they are > +toggled by user. Said properties should be moved into a defconst, e.g., `org-fold-properties', then: Remove text properties as in `remove-text-properties'. See the function for the description of the arguments. However, do not remove invisible text properties defined in `org-fold-properties'. Those are required to keep headlines, drawers and blocks folded. > +Note: The below may be too specific and create troubles if more > +invisibility specs are added to org in future" You can remove the note. If you think the note is important, it should put a comment in the code instead. > + (when (plist-member properties 'invisible) > + (let ((pos start) > + next spec) > + (while (< pos end) > + (setq next (next-single-property-change pos 'invisible nil end) > + spec (get-text-property pos 'invisible)) > + (unless (memq spec (list 'org-hide-block > + 'org-hide-drawer > + 'outline)) The (list ...) should be moved outside the `while' loop. Better, this should be a constant defined somewhere. I also suggest to move `outline' to `org-outline' since we differ from Outline mode. > + (remove-text-properties pos next '(invisible nil) object)) > + (setq pos next)))) > + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) Typo here. There should a single pair of parenthesis, but see above about `when-let'. > + (remove-text-properties start end properties-stripped object))) > + > +(defun org--find-text-property-region (pos prop) I think this is a function useful enough to have a name without double dashes. It can be left in "org-macs.el". It would be nice to have a wrapper for `invisible' property in "org-fold.el", tho. > + "Find a region containing PROP text property around point POS." Reverse the order of arguments in the docstring: Find a region around POS containing PROP text property. > + (let* ((beg (and (get-text-property pos prop) pos)) > + (end beg)) > + (when beg BEG can only be nil if arguments are wrong. In this case, you can throw an error (assuming this is no longer an internal function): (unless beg (user-error "...")) > + ;; when beg is the first point in the region, `previous-single-property-change' > + ;; will return nil. when -> When > + (setq beg (or (previous-single-property-change pos prop) > + beg)) > + ;; when end is the last point in the region, `next-single-property-change' > + ;; will return nil. Ditto. > + (setq end (or (next-single-property-change pos prop) > + end)) > + (unless (= beg end) ; this should not happen I assume this will be the case in an empty buffer. Anyway, (1 . 1) sounds more regular than a nil return value, not specified in the docstring. IOW, I suggest to remove this check. > + (cons beg end))))) > + > +(defun org--add-to-list-text-property (from to prop element) > + "Add element to text property PROP, whos value should be a list." The docstring is incomplete. All arguments need to be described. Also, I suggest: Append ELEMENT to the value of text property PROP. > + (add-text-properties from to `(,prop ,(list element))) ; create if none Here, you are resetting all the properties before adding anything, aren't you? IOW, there might be a bug there. > + ;; add to existing > + (alter-text-property from to > + prop > + (lambda (val) > + (if (member element val) > + val > + (cons element val))))) > +(defun org--remove-from-list-text-property (from to prop element) > + "Remove ELEMENT from text propery PROP, whos value should be a list." The docstring needs to be improved. > + (let ((pos from)) > + (while (< pos to) > + (when-let ((val (get-text-property pos prop))) > + (if (equal val (list element)) (list element) needs to be moved out of the `while' loop. > + (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil)) > + (put-text-property pos (next-single-char-property-change pos prop nil to) > + prop (remove element (get-text-property pos prop))))) If we specialize the function, `remove' -> `remq' > + (setq pos (next-single-char-property-change pos prop nil to))))) Please factor out `next-single-char-property-change'. Note that `org--remove-from-list-text-property' and `org--add-to-list-text-property' do not seem to be used throughout your patch. > +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block) > + "Priority of invisibility specs.") This should be the constant I wrote about earlier. Note that those are not "specs", just properties. I suggest to rename it. > +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only) This name is waaaaaaay too long. > + "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC. Maybe: Return a unique symbol suitable for `invisible' property. Then: Return value is meant to be used as a buffer-local variable in current buffer, or BUFFER if this is non-nil. > +If the buffer already have buffer-local setup in `char-property-alias-alist' > +and the setup appears to be created for different buffer, > +copy the old invisibility state into new buffer-local text properties, > +unless RETURN-ONLY is non-nil." > + (if (not (member spec org--invisible-spec-priority-list)) > + (user-error "%s should be a valid invisibility spec" spec) No need to waste an indentation level for that: (unless (member …) (user-error "%S should be …" spec)) Also, this is a property, not a "spec". > + (let* ((buf (or buffer (current-buffer)))) > + (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S" This clearly needs a shorter name. In particular, "buffer-local" can be removed. > + (symbol-name spec) > + ;; (sxhash buf) appears to be not constant over time. > + ;; Using buffer-name is safe, since the only place where > + ;; buffer-local text property actually matters is an indirect > + ;; buffer, where the name cannot be same anyway. > + (sxhash (buffer-name buf)))))) > + (prog1 > + local-prop Please move LOCAL-PROP after the (unless return-only ...) sexp. > + (unless return-only > + (with-current-buffer buf > + (unless (member local-prop (alist-get 'invisible char-property-alias-alist)) > + ;; copy old property "Copy old property." > + (dolist (old-prop (alist-get 'invisible char-property-alias-alist)) We cannot use `alist-get', which was added in Emacs 25.3 only. > + (org-with-wide-buffer > + (let* ((pos (point-min)) > + (spec (seq-find (lambda (spec) > + (string-match-p (symbol-name spec) > + (symbol-name old-prop))) > + org--invisible-spec-priority-list)) Likewise, we cannot use `seq-find'. > + (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))) > + (while (< pos (point-max)) > + (when-let (val (get-text-property pos old-prop)) > + (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val)) > + (setq pos (next-single-char-property-change pos old-prop)))))) > + (setq-local char-property-alias-alist > + (cons (cons 'invisible > + (mapcar (lambda (spec) > + (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)) > + org--invisible-spec-priority-list)) > + (remove (assq 'invisible char-property-alias-alist) > + char-property-alias-alist))))))))))) This begs for explainations in the docstring or as comments. In particular, just by reading the code, I have no clue about how this is going to be used, how it is going to solve issues with indirect buffers, with invisibility stacking, etc. I don't mind if there are more comment lines than lines of code in that area. > - (remove-overlays from to 'invisible spec) > - ;; Use `front-advance' since text right before to the beginning of > - ;; the overlay belongs to the visible line than to the contents. > - (when flag > - (let ((o (make-overlay from to nil 'front-advance))) > - (overlay-put o 'evaporate t) > - (overlay-put o 'invisible spec) > - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > - > + (with-silent-modifications > + (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil)) > + (when flag > + (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec)))) I don't think there is a need for `remove-text-properties' in every case. Also, (org--get-buffer-local-invisible-property-symbol spec) should be factored out. I suggest: (with-silent-modifications (let ((property (org--get-buffer-local-invisible-property-symbol spec))) (if flag (put-text-property from to property spec) (remove-text-properties from to (list property nil))))) > +(defun org-after-change-function (from to len) This is a terrible name. Org may add different functions in a-c-f, they cannot all be called like this. Assuming the "org-fold" prefix, it could be: org-fold--fix-folded-region > + "Process changes in folded elements. > +If a text was inserted into invisible region, hide the inserted text. > +If the beginning/end line of a folded drawer/block was changed, unfold it. > +If a valid end line was inserted in the middle of the folded drawer/block, unfold it." Nitpick: please do not skip lines amidst a function. Empty lines are used to separate functions, so this is distracting. If a part of the function should stand out, a comment explaining what the part is doing is enough. > + ;; re-hide text inserted in the middle of a folded region Re-hide … folded region. > + (dolist (spec org--invisible-spec-priority-list) > + (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec))) > + (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec)))) > + (when (eq spec-to spec-from) > + (org-flag-region from to 't spec-to)))) This part should first check if we're really after an insertion, e.g., if FROM is different from TO, and exit early if that's not the case. Also, no need to quote t. > + ;; Process all the folded text between `from' and `to' > + (org-with-wide-buffer > + > + (if (< to from) > + (let ((tmp from)) > + (setq from to) > + (setq to tmp))) I'm surprised you need to do that. Did you encounter a case where a-c-f was called with boundaries in reverse order? > + ;; Include next/previous line into the changed region. > + ;; This is needed to catch edits in beginning line of a folded > + ;; element. > + (setq to (save-excursion (goto-char to) (forward-line) (point))) (forward-line) (point) ---> (line-beginning-position 2) > + (setq from (save-excursion (goto-char from) (forward-line -1) (point))) (forward-line -1) (point) ---> (line-beginning-position 0) Anyway, I have the feeling this is not a good idea to extend it now, without first checking that we are in a folded drawer or block. It may also catch unwanted parts, e.g., a folded drawer ending on the line above. What about first finding the whole region with property (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer) then extending the initial part to include the drawer opening? I don't think we need to extend past the ending part, because drawer closing line is always included in the invisible part of the drawer. > + ;; Expand the considered region to include partially present folded > + ;; drawer/block. > + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) > + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) > + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) > + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) Please factor out (org--get-buffer-local-invisible-property-symbol XXX), this is difficult to read. > + ;; check folded drawers Check folded drawers. > + (let ((pos from)) > + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) > + (setq pos (next-single-char-property-change pos > + (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) > + (while (< pos to) > + (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) > + pos)) > + (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) > + > + (let (unfold?) > + ;; the line before folded text should be beginning of the drawer > + (save-excursion > + (goto-char drawer-begin) > + (backward-char) Why `backward-char'? > + (beginning-of-line) > + (unless (looking-at-p org-drawer-regexp) looking-at-p ---> looking-at However, you must wrap this function within `save-match-data'. > + (setq unfold? t))) > + ;; the last line of the folded text should be :END: > + (save-excursion > + (goto-char drawer-end) > + (beginning-of-line) > + (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re)) > + (setq unfold? t))) > + ;; there should be no :END: anywhere in the drawer body > + (save-excursion > + (goto-char drawer-begin) > + (when (save-excursion > + (let ((case-fold-search t)) > + (re-search-forward org-property-end-re > + (max (point) > + (1- (save-excursion > + (goto-char drawer-end) > + (line-beginning-position)))) > + 't))) > (max (point) > (save-excursion (goto-char drawer-end) (line-end-position 0)) > + (setq unfold? t))) > + ;; there should be no new entry anywhere in the drawer body > + (save-excursion > + (goto-char drawer-begin) > + (when (save-excursion > + (let ((case-fold-search t)) > + (re-search-forward org-outline-regexp-bol > + (max (point) > + (1- (save-excursion > + (goto-char drawer-end) > + (line-beginning-position)))) > + 't))) > + (setq unfold? t))) In the phase above, you need to bail out as soon as unfold? is non-nil: (catch :exit ... (throw :exit (setq unfold? t)) ...) Also last two checks should be lumped together, with an appropriate regexp. Finally, I have the feeling we're missing out some early exits when nothing is folded around point (e.g., most of the case). > + > + (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer)))) > + > + (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))) > + > + ;; check folded blocks > + (let ((pos from)) > + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) > + (setq pos (next-single-char-property-change pos > + (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) > + (while (< pos to) > + (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) > + pos)) > + (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) > + > + (let (unfold?) > + ;; the line before folded text should be beginning of the block > + (save-excursion > + (goto-char block-begin) > + (backward-char) > + (beginning-of-line) > + (unless (looking-at-p org-dblock-start-re) > + (setq unfold? t))) > + ;; the last line of the folded text should be end of the block > + (save-excursion > + (goto-char block-end) > + (beginning-of-line) > + (unless (looking-at-p org-dblock-end-re) > + (setq unfold? t))) > + ;; there should be no #+end anywhere in the block body > + (save-excursion > + (goto-char block-begin) > + (when (save-excursion > + (re-search-forward org-dblock-end-re > + (max (point) > + (1- (save-excursion > + (goto-char block-end) > + (line-beginning-position)))) > + 't)) > + (setq unfold? t))) > + ;; there should be no new entry anywhere in the block body > + (save-excursion > + (goto-char block-begin) > + (when (save-excursion > + (let ((case-fold-search t)) > + (re-search-forward org-outline-regexp-bol > + (max (point) > + (1- (save-excursion > + (goto-char block-end) > + (line-beginning-position)))) > + 't))) > + (setq unfold? t))) > + > + (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block)))) > + > + (setq pos > + (next-single-char-property-change pos > + (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))))) See remarks above. The parts related to drawers and blocks are so similar they should be factorized out. Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we want here. The correct regexps would be: (rx bol (zero-or-more (any " " "\t")) "#+begin" (or ":" (seq "_" (group (one-or-more (not (syntax whitespace))))))) and closing line should match match-group 1 from the regexp above, e.g.: (concat (rx bol (zero-or-more (any " " "\t")) "#+end") (if block-type (concat "_" (regexp-quote block-type) (rx (zero-or-more (any " " "\t")) eol)) (rx (opt ":") (zero-or-more (any " " "\t")) eol))) assuming `block-type' is the type of the block, or nil, i.e., (match-string 1) in the previous regexp. > - (pcase (get-char-property-and-overlay (point) 'invisible) > + (pcase (get-char-property (point) 'invisible) > ;; Do not fold already folded drawers. > - (`(outline . ,o) (goto-char (overlay-end o))) > + ('outline 'outline --> `outline > (end-of-line)) > (while (and (< arg 0) (re-search-backward regexp nil :move)) > (unless (bobp) > - (while (pcase (get-char-property-and-overlay (point) 'invisible) > - (`(outline . ,o) > - (goto-char (overlay-start o)) > - (re-search-backward regexp nil :move)) > - (_ nil)))) > + (pcase (get-char-property (point) 'invisible) > + ('outline > + (goto-char (car (org--find-text-property-region (point) 'invisible))) > + (beginning-of-line)) > + (_ nil))) Does this move to the beginning of the widest invisible part around point? If that's not the case, we need a function in "org-fold.el" doing just that. Or we need to nest `while' loops as it was the case in the code you reverted. ----- Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-10 17:14 ` Nicolas Goaziou @ 2020-06-21 9:52 ` Ihor Radchenko 2020-06-21 15:01 ` Nicolas Goaziou 2020-08-11 6:45 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-06-21 9:52 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode > Once done, I think we should move (or copy, first) _all_ folding-related > functions into a new "org-fold.el" library. Functions and variables > included there should have a proper "org-fold-" prefix. More on this in > the detailed report. I am currently working on org-fold.el. However, I am not sure if it is acceptable to move some of the existing functions from org.el to org-fold.el. Specifically, functions from the following sections of org.el might be moved to org-fold.el: > ;;; Visibility (headlines, blocks, drawers) > ;;;; Reveal point location > ;;;; Visibility cycling Should I do it? Best, Ihor Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> [The patch itself will be provided in the following email] > > Thank you! I'll first make some generic remarks, then comment the diff > in more details. > >> I have four more updates from the previous version of the patch: >> >> 1. All the code handling modifications in folded drawers/blocks is moved >> to after-change-function. It works as follows: >> - if any text is inserted in the middle of hidden region, that text >> is also hidden; >> - if BEGIN/END line of a folded drawer do not match org-drawer-regexp >> and org-property-end-re, unfold it; >> - if org-property-end-re or new org-outline-regexp-bol is inserted in >> the middle of the drawer, unfold it; >> - the same logic for blocks. > > This sounds good, barring a minor error in the regexp for blocks, and > missing optimizations. More on this in the detailed comments. > >> 2. The text property stack is rewritten using char-property-alias-alist. >> This is faster in comparison with previous approach, which involved >> modifying all the text properties every timer org-flag-region was >> called. > > I'll need information about this, as I'm not sure to fully understand > all the consequences of this. But more importantly, this needs to be > copiously documented somewhere for future hackers. > >> 3. org-toggle-custom-properties-visibility is rewritten using text >> properties. I also took a freedom to implement a new feature here. >> Now, setting new `org-custom-properties-hide-emptied-drawers' to >> non-nil will result in hiding the whole property drawer if it >> contains only org-custom-properties. > > I don't think this is a good idea. AFAIR, we always refused to hide > completely anything, including empty drawers. The reason is that if the > drawer is completely hidden, you cannot expand it easily, or even know > there is one. > > In any case, this change shouldn't belong to this patch set, and should > be discussed separately. > >> 4. This patch should work against 1aa095ccf. However, the merge was not >> trivial here. Recent commits actively used the fact that drawers and >> outlines are hidden via 'outline invisibility spec, which is not the >> case in this branch. I am not confident that I did not break anything >> during the merge, especially 1aa095ccf. > > [...] > >> Also, I have seen some optimisations making use of the fact that drawers >> and headlines both use 'outline invisibility spec. This change in the >> implementation details supposed to improve performance and should not be >> necessary if this patch is going to be merged. Would it be possible to >> refrain from abusing this particular implementation detail in the >> nearest commits on master (unless really necessary)? > > To be clear, I didn't intend to make your life miserable. > > However, I had to fix regression on drawers visibility before Org 9.4 > release. Also, merging invisibility properties for drawers and outline > was easier for me. So, I had the opportunity to kill two birds with one > stone. > > As a reminder, Org 9.4 is about to be released, but Org 9.5 will take > months to go out. So, even though I hope your changes will land into > Org, there is no reason for us to refrain from improving (actually > fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such > changes are not expected to happen anymore. > > I hope you understand. > >> I would like to finalise the current patch and work on other code using >> overlays separately. This patch is already quite complicated as is. I do >> not want to introduce even more potential bugs by working on things not >> directly affected by this version of the patch. > > The patch is technically mostly good, but needs more work for > integration into Org. > > First, it includes a few unrelated changes that should be removed (e.g., > white space fixes in unrelated parts of the code). Also, as written > above, the changes about `org-custom-properties-hide-emptied-drawers' > should be removed for the time being. > > Once done, I think we should move (or copy, first) _all_ folding-related > functions into a new "org-fold.el" library. Functions and variables > included there should have a proper "org-fold-" prefix. More on this in > the detailed report. > > The functions `org-find-text-property-region', > `org-add-to-list-text-property', and > `org-remove-from-list-text-property' can be left in "org-macs.el", since > they do not directly depend on the `invisible' property. Note the last > two functions I mentioned are not used throughout your patch. They might > be removed. > > This first patch can coexist with overlay folding since functions in > both mechanisms are named differently. > > Then, another patch can integrate "org-fold.el" into Org folding. I also > suggest to move the Outline -> Org transition to yet another patch. > I think there's more work to do on this part. > > Now, into the details of your patch. The first remarks are: > > 1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not > sure), so some functions cannot be used. > > 2. we don't use "subr-x.el" in the code base. In particular, it would be > nice to replace `when-let' with `when' + `let'. This change costs > only one loc. > > 3. Some docstrings need more work. In particular, Emacs documentation > expects all arguments to be explained in the docstring, if possible > in the order in which they appear. There are exceptions, though. For > example, in a function like `org-remove-text-properties', you can > mention arguments are simply the same as in `remove-text-properties'. > > 4. Some refactorization is needed in some places. I mentioned them in > the report below. > > 5. I didn't dive much into the Isearch code so far. I tested it a bit > and seems to work nicely. I noticed one bug though. In the following > document: > > #+begin: foo > :FOO: > bar > :END: > #+end > bar > > when both the drawer and the block are folded (i.e., you fold the > drawer first, then the block), searching for "bar" first find the > last one, then overwraps and find the first one. > > 6. Since we're rewriting folding code, we might as well rename folding > properties: org-hide-drawer -> org-fold-drawer, outline -> > org-fold-headline… > > Now, here are more comments about the code. > > ----- > >> +(defun org-remove-text-properties (start end properties &optional object) > > IMO, this generic name doesn't match the specialized nature of the > function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library. > >> + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. > > Line is too long. Suggestion: > > Remove text properties except folding-related ones. > >> +Do not remove invisible text properties specified by 'outline, >> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this >> +is needed to keep outlines, drawers, and blocks hidden unless they are >> +toggled by user. > > Said properties should be moved into a defconst, e.g., > `org-fold-properties', then: > > Remove text properties as in `remove-text-properties'. See the > function for the description of the arguments. > > However, do not remove invisible text properties defined in > `org-fold-properties'. Those are required to keep headlines, drawers > and blocks folded. > >> +Note: The below may be too specific and create troubles if more >> +invisibility specs are added to org in future" > > You can remove the note. If you think the note is important, it should > put a comment in the code instead. > >> + (when (plist-member properties 'invisible) >> + (let ((pos start) >> + next spec) >> + (while (< pos end) >> + (setq next (next-single-property-change pos 'invisible nil end) >> + spec (get-text-property pos 'invisible)) >> + (unless (memq spec (list 'org-hide-block >> + 'org-hide-drawer >> + 'outline)) > > The (list ...) should be moved outside the `while' loop. Better, this > should be a constant defined somewhere. I also suggest to move > `outline' to `org-outline' since we differ from Outline mode. > >> + (remove-text-properties pos next '(invisible nil) object)) >> + (setq pos next)))) >> + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) > > Typo here. There should a single pair of parenthesis, but see above > about `when-let'. > >> + (remove-text-properties start end properties-stripped object))) >> + >> +(defun org--find-text-property-region (pos prop) > > I think this is a function useful enough to have a name without double > dashes. It can be left in "org-macs.el". It would be nice to have > a wrapper for `invisible' property in "org-fold.el", tho. > >> + "Find a region containing PROP text property around point POS." > > Reverse the order of arguments in the docstring: > > Find a region around POS containing PROP text property. > >> + (let* ((beg (and (get-text-property pos prop) pos)) >> + (end beg)) >> + (when beg > > BEG can only be nil if arguments are wrong. In this case, you can > throw an error (assuming this is no longer an internal function): > > (unless beg (user-error "...")) > >> + ;; when beg is the first point in the region, `previous-single-property-change' >> + ;; will return nil. > > when -> When > >> + (setq beg (or (previous-single-property-change pos prop) >> + beg)) >> + ;; when end is the last point in the region, `next-single-property-change' >> + ;; will return nil. > > Ditto. > >> + (setq end (or (next-single-property-change pos prop) >> + end)) >> + (unless (= beg end) ; this should not happen > > I assume this will be the case in an empty buffer. Anyway, (1 . 1) > sounds more regular than a nil return value, not specified in the > docstring. IOW, I suggest to remove this check. > >> + (cons beg end))))) >> + >> +(defun org--add-to-list-text-property (from to prop element) >> + "Add element to text property PROP, whos value should be a list." > > The docstring is incomplete. All arguments need to be described. Also, > I suggest: > > Append ELEMENT to the value of text property PROP. > >> + (add-text-properties from to `(,prop ,(list element))) ; create if none > > Here, you are resetting all the properties before adding anything, > aren't you? IOW, there might be a bug there. > >> + ;; add to existing >> + (alter-text-property from to >> + prop >> + (lambda (val) >> + (if (member element val) >> + val >> + (cons element val))))) > >> +(defun org--remove-from-list-text-property (from to prop element) >> + "Remove ELEMENT from text propery PROP, whos value should be a list." > > The docstring needs to be improved. > >> + (let ((pos from)) >> + (while (< pos to) >> + (when-let ((val (get-text-property pos prop))) >> + (if (equal val (list element)) > > (list element) needs to be moved out of the `while' loop. > >> + (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil)) >> + (put-text-property pos (next-single-char-property-change pos prop nil to) >> + prop (remove element (get-text-property pos prop))))) > > If we specialize the function, `remove' -> `remq' > >> + (setq pos (next-single-char-property-change pos prop nil to))))) > > Please factor out `next-single-char-property-change'. > > Note that `org--remove-from-list-text-property' and > `org--add-to-list-text-property' do not seem to be used throughout > your patch. > >> +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block) >> + "Priority of invisibility specs.") > > This should be the constant I wrote about earlier. Note that those are > not "specs", just properties. I suggest to rename it. > >> +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only) > > This name is waaaaaaay too long. > >> + "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC. > > Maybe: > > > Return a unique symbol suitable for `invisible' property. > > Then: > > Return value is meant to be used as a buffer-local variable in > current buffer, or BUFFER if this is non-nil. > >> +If the buffer already have buffer-local setup in `char-property-alias-alist' >> +and the setup appears to be created for different buffer, >> +copy the old invisibility state into new buffer-local text properties, >> +unless RETURN-ONLY is non-nil." >> + (if (not (member spec org--invisible-spec-priority-list)) >> + (user-error "%s should be a valid invisibility spec" spec) > > No need to waste an indentation level for that: > > (unless (member …) > (user-error "%S should be …" spec)) > > Also, this is a property, not a "spec". > >> + (let* ((buf (or buffer (current-buffer)))) >> + (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S" > > This clearly needs a shorter name. In particular, "buffer-local" can be removed. > >> + (symbol-name spec) >> + ;; (sxhash buf) appears to be not constant over time. >> + ;; Using buffer-name is safe, since the only place where >> + ;; buffer-local text property actually matters is an indirect >> + ;; buffer, where the name cannot be same anyway. >> + (sxhash (buffer-name buf)))))) > > >> + (prog1 >> + local-prop > > Please move LOCAL-PROP after the (unless return-only ...) sexp. > >> + (unless return-only >> + (with-current-buffer buf >> + (unless (member local-prop (alist-get 'invisible char-property-alias-alist)) >> + ;; copy old property > > "Copy old property." > >> + (dolist (old-prop (alist-get 'invisible char-property-alias-alist)) > > We cannot use `alist-get', which was added in Emacs 25.3 only. > >> + (org-with-wide-buffer >> + (let* ((pos (point-min)) >> + (spec (seq-find (lambda (spec) >> + (string-match-p (symbol-name spec) >> + (symbol-name old-prop))) >> + org--invisible-spec-priority-list)) > > Likewise, we cannot use `seq-find'. > >> + (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))) >> + (while (< pos (point-max)) >> + (when-let (val (get-text-property pos old-prop)) >> + (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val)) >> + (setq pos (next-single-char-property-change pos old-prop)))))) >> + (setq-local char-property-alias-alist >> + (cons (cons 'invisible >> + (mapcar (lambda (spec) >> + (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)) >> + org--invisible-spec-priority-list)) >> + (remove (assq 'invisible char-property-alias-alist) >> + char-property-alias-alist))))))))))) > > This begs for explainations in the docstring or as comments. In > particular, just by reading the code, I have no clue about how this is > going to be used, how it is going to solve issues with indirect > buffers, with invisibility stacking, etc. > > I don't mind if there are more comment lines than lines of code in > that area. > >> - (remove-overlays from to 'invisible spec) >> - ;; Use `front-advance' since text right before to the beginning of >> - ;; the overlay belongs to the visible line than to the contents. >> - (when flag >> - (let ((o (make-overlay from to nil 'front-advance))) >> - (overlay-put o 'evaporate t) >> - (overlay-put o 'invisible spec) >> - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) >> - >> + (with-silent-modifications >> + (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil)) >> + (when flag >> + (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec)))) > > I don't think there is a need for `remove-text-properties' in every > case. Also, (org--get-buffer-local-invisible-property-symbol spec) > should be factored out. > > I suggest: > > (with-silent-modifications > (let ((property (org--get-buffer-local-invisible-property-symbol spec))) > (if flag > (put-text-property from to property spec) > (remove-text-properties from to (list property nil))))) > >> +(defun org-after-change-function (from to len) > > This is a terrible name. Org may add different functions in a-c-f, > they cannot all be called like this. Assuming the "org-fold" prefix, > it could be: > > org-fold--fix-folded-region > >> + "Process changes in folded elements. >> +If a text was inserted into invisible region, hide the inserted text. >> +If the beginning/end line of a folded drawer/block was changed, unfold it. >> +If a valid end line was inserted in the middle of the folded drawer/block, unfold it." > > Nitpick: please do not skip lines amidst a function. Empty lines are > used to separate functions, so this is distracting. > > If a part of the function should stand out, a comment explaining what > the part is doing is enough. > >> + ;; re-hide text inserted in the middle of a folded region > > Re-hide … folded region. > >> + (dolist (spec org--invisible-spec-priority-list) >> + (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec))) >> + (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec)))) >> + (when (eq spec-to spec-from) >> + (org-flag-region from to 't spec-to)))) > > This part should first check if we're really after an insertion, e.g., > if FROM is different from TO, and exit early if that's not the case. > > Also, no need to quote t. > >> + ;; Process all the folded text between `from' and `to' >> + (org-with-wide-buffer >> + >> + (if (< to from) >> + (let ((tmp from)) >> + (setq from to) >> + (setq to tmp))) > > I'm surprised you need to do that. Did you encounter a case where > a-c-f was called with boundaries in reverse order? > >> + ;; Include next/previous line into the changed region. >> + ;; This is needed to catch edits in beginning line of a folded >> + ;; element. >> + (setq to (save-excursion (goto-char to) (forward-line) (point))) > > (forward-line) (point) ---> (line-beginning-position 2) > >> + (setq from (save-excursion (goto-char from) (forward-line -1) (point))) > > (forward-line -1) (point) ---> (line-beginning-position 0) > > Anyway, I have the feeling this is not a good idea to extend it now, > without first checking that we are in a folded drawer or block. It may > also catch unwanted parts, e.g., a folded drawer ending on the line > above. > > What about first finding the whole region with property > > (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer) > > then extending the initial part to include the drawer opening? I don't > think we need to extend past the ending part, because drawer closing > line is always included in the invisible part of the drawer. > >> + ;; Expand the considered region to include partially present folded >> + ;; drawer/block. >> + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) > > Please factor out (org--get-buffer-local-invisible-property-symbol > XXX), this is difficult to read. > >> + ;; check folded drawers > > Check folded drawers. > >> + (let ((pos from)) >> + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + (setq pos (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + (while (< pos to) >> + (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + pos)) >> + (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + >> + (let (unfold?) >> + ;; the line before folded text should be beginning of the drawer >> + (save-excursion >> + (goto-char drawer-begin) >> + (backward-char) > > Why `backward-char'? > >> + (beginning-of-line) >> + (unless (looking-at-p org-drawer-regexp) > > looking-at-p ---> looking-at > > However, you must wrap this function within `save-match-data'. > >> + (setq unfold? t))) >> + ;; the last line of the folded text should be :END: >> + (save-excursion >> + (goto-char drawer-end) >> + (beginning-of-line) >> + (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re)) >> + (setq unfold? t))) >> + ;; there should be no :END: anywhere in the drawer body >> + (save-excursion >> + (goto-char drawer-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-property-end-re >> + (max (point) >> + (1- (save-excursion >> + (goto-char drawer-end) >> + (line-beginning-position)))) >> + 't))) > >> (max (point) >> (save-excursion (goto-char drawer-end) (line-end-position 0)) > >> + (setq unfold? t))) >> + ;; there should be no new entry anywhere in the drawer body >> + (save-excursion >> + (goto-char drawer-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-outline-regexp-bol >> + (max (point) >> + (1- (save-excursion >> + (goto-char drawer-end) >> + (line-beginning-position)))) >> + 't))) >> + (setq unfold? t))) > > In the phase above, you need to bail out as soon as unfold? is non-nil: > > (catch :exit > ... > (throw :exit (setq unfold? t)) > ...) > > Also last two checks should be lumped together, with an appropriate > regexp. > > Finally, I have the feeling we're missing out some early exits when > nothing is folded around point (e.g., most of the case). > >> + >> + (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer)))) >> + >> + (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))) >> + >> + ;; check folded blocks >> + (let ((pos from)) >> + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + (setq pos (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) >> + (while (< pos to) >> + (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + pos)) >> + (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) >> + >> + (let (unfold?) >> + ;; the line before folded text should be beginning of the block >> + (save-excursion >> + (goto-char block-begin) >> + (backward-char) >> + (beginning-of-line) >> + (unless (looking-at-p org-dblock-start-re) >> + (setq unfold? t))) >> + ;; the last line of the folded text should be end of the block >> + (save-excursion >> + (goto-char block-end) >> + (beginning-of-line) >> + (unless (looking-at-p org-dblock-end-re) >> + (setq unfold? t))) >> + ;; there should be no #+end anywhere in the block body >> + (save-excursion >> + (goto-char block-begin) >> + (when (save-excursion >> + (re-search-forward org-dblock-end-re >> + (max (point) >> + (1- (save-excursion >> + (goto-char block-end) >> + (line-beginning-position)))) >> + 't)) >> + (setq unfold? t))) >> + ;; there should be no new entry anywhere in the block body >> + (save-excursion >> + (goto-char block-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-outline-regexp-bol >> + (max (point) >> + (1- (save-excursion >> + (goto-char block-end) >> + (line-beginning-position)))) >> + 't))) >> + (setq unfold? t))) >> + >> + (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block)))) >> + >> + (setq pos >> + (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))))) > > See remarks above. The parts related to drawers and blocks are so > similar they should be factorized out. > > Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we > want here. The correct regexps would be: > > (rx bol > (zero-or-more (any " " "\t")) > "#+begin" > (or ":" > (seq "_" > (group (one-or-more (not (syntax whitespace))))))) > > and closing line should match match-group 1 from the regexp above, e.g.: > > (concat (rx bol (zero-or-more (any " " "\t")) "#+end") > (if block-type > (concat "_" > (regexp-quote block-type) > (rx (zero-or-more (any " " "\t")) eol)) > (rx (opt ":") (zero-or-more (any " " "\t")) eol))) > > assuming `block-type' is the type of the block, or nil, i.e., > (match-string 1) in the previous regexp. > >> - (pcase (get-char-property-and-overlay (point) 'invisible) >> + (pcase (get-char-property (point) 'invisible) >> ;; Do not fold already folded drawers. >> - (`(outline . ,o) (goto-char (overlay-end o))) >> + ('outline > > 'outline --> `outline > >> (end-of-line)) >> (while (and (< arg 0) (re-search-backward regexp nil :move)) >> (unless (bobp) >> - (while (pcase (get-char-property-and-overlay (point) 'invisible) >> - (`(outline . ,o) >> - (goto-char (overlay-start o)) >> - (re-search-backward regexp nil :move)) >> - (_ nil)))) >> + (pcase (get-char-property (point) 'invisible) >> + ('outline >> + (goto-char (car (org--find-text-property-region (point) 'invisible))) >> + (beginning-of-line)) >> + (_ nil))) > > Does this move to the beginning of the widest invisible part around > point? If that's not the case, we need a function in "org-fold.el" > doing just that. Or we need to nest `while' loops as it was the case > in the code you reverted. > > ----- > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-21 9:52 ` Ihor Radchenko @ 2020-06-21 15:01 ` Nicolas Goaziou 0 siblings, 0 replies; 192+ messages in thread From: Nicolas Goaziou @ 2020-06-21 15:01 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Hello, Ihor Radchenko <yantar92@gmail.com> writes: > I am currently working on org-fold.el. However, I am not sure if it is > acceptable to move some of the existing functions from org.el to > org-fold.el. > > Specifically, functions from the following sections of org.el might be > moved to org-fold.el: >> ;;; Visibility (headlines, blocks, drawers) >> ;;;; Reveal point location >> ;;;; Visibility cycling > > Should I do it? That makes sense, yes. Note that you can first copy and rename most functions to make the transition easier. As a second step, we can plug new functions into the main system. Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-06-10 17:14 ` Nicolas Goaziou 2020-06-21 9:52 ` Ihor Radchenko @ 2020-08-11 6:45 ` Ihor Radchenko 2020-08-11 23:07 ` Kyle Meyer 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-08-11 6:45 UTC (permalink / raw) To: Nicolas Goaziou; +Cc: emacs-orgmode Hello, [The patch itself will be provided in the following email or can be accessed via Github [1]] I have finally finished the suggested edits. Most importantly: - All the folding-related code lives in =org-fold.el= and =org-cycle.el= now. - =org-fold.el= have commentary section explaining how folding works and exposing API for external code using folding. - I wrote a patch for =isearch.el= adding support searching inside text hidden via text properties [2] and the relevant support of the patch in the =org-fold.el=. The current =isearch= behaviour is also supported. Hope the patch will go through eventually. The patch is fairly stable on my system. Any feedback or bug reports are welcome. There are still known problems though. The patch currently breaks many org-mode tests when running =make test=. It is partially because some tests assume overlays to be used for folding and partially because the patch appears to break certain folding conventions. I am still investigating this (and learning =ert=). More details: >> 2. The text property stack is rewritten using char-property-alias-alist. >> This is faster in comparison with previous approach, which involved >> modifying all the text properties every timer org-flag-region was >> called. > I'll need information about this, as I'm not sure to fully understand > all the consequences of this. But more importantly, this needs to be > copiously documented somewhere for future hackers. See commentary section in =org-fold.el= and comments in =org-fold--property-symbol-get-create=. > As a reminder, Org 9.4 is about to be released, but Org 9.5 will take > months to go out. So, even though I hope your changes will land into > Org, there is no reason for us to refrain from improving (actually > fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such > changes are not expected to happen anymore. > > I hope you understand. Probably my message sounded harsher than it should. I totally understand why such changes are needed, but wanted to make people aware that old folding implementation will be likely changed. > First, it includes a few unrelated changes that should be removed (e.g., > white space fixes in unrelated parts of the code). Also, as written > above, the changes about `org-custom-properties-hide-emptied-drawers' > should be removed for the time being. Let's leave this until the patch is ready to be pushed. I want to focus on handling bugs first without a need to check for the whitespace changes. > Once done, I think we should move (or copy, first) _all_ folding-related > functions into a new "org-fold.el" library. Functions and variables > included there should have a proper "org-fold-" prefix. More on this in > the detailed report. I decided to create =org-fold.el= and =org-cycle.el= and move all the relevant functions there. The org-cycle code seems to be so frequently used that I did not want to break the org-fold prefix to org-fold-cycle and decided to separate the cycle code into standalone file. > Then, another patch can integrate "org-fold.el" into Org folding. I also > suggest to move the Outline -> Org transition to yet another patch. > I think there's more work to do on this part. Agree. For the time being, I will still provide the full patch if anyone wants to test the whole thing on their system. > 1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not > sure), so some functions cannot be used. I tried my best to cleanup the functions as you suggested, but I do not know a good way to check which functions are not supported by old Emacs versions. > 2. we don't use "subr-x.el" in the code base. In particular, it would be > nice to replace `when-let' with `when' + `let'. This change costs > only one loc. Done. > 3. Some docstrings need more work. In particular, Emacs documentation > expects all arguments to be explained in the docstring, if possible > in the order in which they appear. There are exceptions, though. For > example, in a function like `org-remove-text-properties', you can > mention arguments are simply the same as in `remove-text-properties'. Done. > 5. I didn't dive much into the Isearch code so far. I tested it a bit > and seems to work nicely. I noticed one bug though. In the following > document: > > #+begin: foo > :FOO: > bar > :END: > #+end > bar > > when both the drawer and the block are folded (i.e., you fold the > drawer first, then the block), searching for "bar" first find the > last one, then overwraps and find the first one. Fixed now. > 6. Since we're rewriting folding code, we might as well rename folding > properties: org-hide-drawer -> org-fold-drawer, outline -> > org-fold-headline… Done. See =org-fold-get-folding-spec-for-element=. >> +(defun org-remove-text-properties (start end properties &optional object) > > IMO, this generic name doesn't match the specialized nature of the > function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library. This function is unused. I simply removed the function altogether. >> +(defun org--find-text-property-region (pos prop) > > I think this is a function useful enough to have a name without double > dashes. It can be left in "org-macs.el". It would be nice to have > a wrapper for `invisible' property in "org-fold.el", tho. Done. See org-find-text-property-region and org-fold-get-region-at-point. >> + "Find a region containing PROP text property around point POS." > > Reverse the order of arguments in the docstring: Done >> + (let* ((beg (and (get-text-property pos prop) pos)) >> + (end beg)) >> + (when beg > > BEG can only be nil if arguments are wrong. In this case, you can > throw an error (assuming this is no longer an internal function): I added "Return nil when PROP is not set at POS." to the docstring. I believe it is better not to force the user to check the property at point before calling this function or catch errors in the code. > I assume this will be the case in an empty buffer. Anyway, (1 . 1) > sounds more regular than a nil return value, not specified in the > docstring. IOW, I suggest to remove this check. Removed. >> +(defun org--add-to-list-text-property (from to prop element) >> + "Add element to text property PROP, whos value should be a list." > > The docstring is incomplete. All arguments need to be described. Also, This functions is unused. I removed it completely. >> +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block) >> + "Priority of invisibility specs.") > > This should be the constant I wrote about earlier. Note that those are > not "specs", just properties. I suggest to rename it. Please note that 'outline, 'out-hide-drawer, and 'org-hide-block (now renamed to 'org-fold-outline, 'org-fold-drawer, and 'org-fold-block) are not text property names. They are values stored in text properties used to fold the text. That's why I call them "folding specs", similarly to =buffer-invisibility-spec= in Emacs. Internally, they are exactly used as members of =buffer-invisibility-spec=. >> +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only) > > This name is waaaaaaay too long. Changed to org-fold--property-symbol-get-create. It is still long, but it don't need to (and should not) be used outside org-fold.el from now. > Maybe: > > > Return a unique symbol suitable for `invisible' property. > > Then: > > Return value is meant to be used as a buffer-local variable in > current buffer, or BUFFER if this is non-nil. Changed the docstring in similar manner. > No need to waste an indentation level for that: > > (unless (member …) > (user-error "%S should be …" spec)) Done >> + (let* ((buf (or buffer (current-buffer)))) >> + (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S" > > This clearly needs a shorter name. In particular, "buffer-local" can be removed. Changed to "org-fold--spec-%s-%S". >> + (prog1 >> + local-prop > > Please move LOCAL-PROP after the (unless return-only ...) sexp. I am not sure I understand why this needs to be changed. I feel that listing the return value will be more clear while reading the code. The remaining part of the =prog1= is optional logic. Moving =local-prop= to the end may reduce readability. > We cannot use `alist-get', which was added in Emacs 25.3 only. Changed to =assq=. > Likewise, we cannot use `seq-find'. Changed to =dolist=. > This begs for explainations in the docstring or as comments. In > particular, just by reading the code, I have no clue about how this is > going to be used, how it is going to solve issues with indirect > buffers, with invisibility stacking, etc. > > I don't mind if there are more comment lines than lines of code in > that area. Done. > I don't think there is a need for `remove-text-properties' in every > case. Also, (org--get-buffer-local-invisible-property-symbol spec) > should be factored out. Done. >> +(defun org-after-change-function (from to len) > > This is a terrible name. Org may add different functions in a-c-f, > they cannot all be called like this. Assuming the "org-fold" prefix, > it could be: > > org-fold--fix-folded-region Changed as you suggested. > Nitpick: please do not skip lines amidst a function. Empty lines are > used to separate functions, so this is distracting. > > If a part of the function should stand out, a comment explaining what > the part is doing is enough. Done. Though many docstrings in org have empty lines creating the same problem. > This part should first check if we're really after an insertion, e.g., > if FROM is different from TO, and exit early if that's not the case. Done. >> + (if (< to from) >> + (let ((tmp from)) >> + (setq from to) >> + (setq to tmp))) > > I'm surprised you need to do that. Did you encounter a case where > a-c-f was called with boundaries in reverse order? I removed it and saw no issues. You are right, it does not seem to happen at all. > (forward-line) (point) ---> (line-beginning-position 2) > (forward-line -1) (point) ---> (line-beginning-position 0) Done. > Anyway, I have the feeling this is not a good idea to extend it now, > without first checking that we are in a folded drawer or block. It may > also catch unwanted parts, e.g., a folded drawer ending on the line > above. This code is specifically written for cases when we are outside folded text, but the edit can still affect folded text right before/after the edited region. Consider two examples: 1. We can change the first visible line of a folded drawer :DRAWER:<begin fold> text inside folded drawer :END:<end-fold> <deleted first : in drawer header> ---- DRAWER:<begin fold, which should be unfolded> text inside folded drawer :END:<end-fold> The edited text was not folded, but must affected the following drawer. 2. We modify :END: of a folded drawer :DRAWER:<begin fold> text inside folded drawer :END:<end-fold> <deleted : in :END:> --- :DRAWER:<begin fold> text inside folded drawer :END<end-fold><changed text region is after the fold> Again, the effected region is not folded, anymore, but it should affect the preceding drawer. > What about first finding the whole region with property > > (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer) > > then extending the initial part to include the drawer opening? I don't > think we need to extend past the ending part, because drawer closing > line is always included in the invisible part of the drawer. As I just showed, we may not really have any folded text in the modified region and thus cannot know if we need to update nearby drawers without looking at them. This code allow handling the described cases and also correctly keep folded drawers folded if they were not really modified. >> + (let (unfold?) >> + ;; the line before folded text should be beginning of the drawer >> + (save-excursion >> + (goto-char drawer-begin) >> + (backward-char) > > Why `backward-char'? =drawer-begin= is pointing to the beginning of folded part of the drawer, so we need to move the line containing the :drawer: > looking-at-p ---> looking-at > > However, you must wrap this function within `save-match-data'. Is there any particular reason to use looking-at in favour of looking-at-p? I have seen looking-at-p many times in org-mode code. > In the phase above, you need to bail out as soon as unfold? is non-nil: > > (catch :exit > ... > (throw :exit (setq unfold? t)) > ...) > > Also last two checks should be lumped together, with an appropriate > regexp. > > Finally, I have the feeling we're missing out some early exits when > nothing is folded around point (e.g., most of the case). Done. > Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we > want here. The correct regexps would be: > > (rx bol > (zero-or-more (any " " "\t")) > "#+begin" > (or ":" > (seq "_" > (group (one-or-more (not (syntax whitespace))))))) > > and closing line should match match-group 1 from the regexp above, e.g.: > > (concat (rx bol (zero-or-more (any " " "\t")) "#+end") > (if block-type > (concat "_" > (regexp-quote block-type) > (rx (zero-or-more (any " " "\t")) eol)) > (rx (opt ":") (zero-or-more (any " " "\t")) eol))) > > assuming `block-type' is the type of the block, or nil, i.e., > (match-string 1) in the previous regexp. Fixed. > 'outline --> `outline Could you explain why? > Does this move to the beginning of the widest invisible part around > point? If that's not the case, we need a function in "org-fold.el" > doing just that. Or we need to nest `while' loops as it was the case > in the code you reverted. See org-fold-next-visibility-change. Best, Ihor [1] Full patch: https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef org-fold.el: https://gist.github.com/yantar92/ffc1fc11550c58dae71de06700e7e4c1 org-cycle.el: https://gist.github.com/yantar92/2be75c0e11968c0bbacc0d22dbca97fd [2] https://lists.gnu.org/archive/html/emacs-devel/2020-07/msg00679.html Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> [The patch itself will be provided in the following email] > > Thank you! I'll first make some generic remarks, then comment the diff > in more details. > >> I have four more updates from the previous version of the patch: >> >> 1. All the code handling modifications in folded drawers/blocks is moved >> to after-change-function. It works as follows: >> - if any text is inserted in the middle of hidden region, that text >> is also hidden; >> - if BEGIN/END line of a folded drawer do not match org-drawer-regexp >> and org-property-end-re, unfold it; >> - if org-property-end-re or new org-outline-regexp-bol is inserted in >> the middle of the drawer, unfold it; >> - the same logic for blocks. > > This sounds good, barring a minor error in the regexp for blocks, and > missing optimizations. More on this in the detailed comments. > >> 2. The text property stack is rewritten using char-property-alias-alist. >> This is faster in comparison with previous approach, which involved >> modifying all the text properties every timer org-flag-region was >> called. > > I'll need information about this, as I'm not sure to fully understand > all the consequences of this. But more importantly, this needs to be > copiously documented somewhere for future hackers. > >> 3. org-toggle-custom-properties-visibility is rewritten using text >> properties. I also took a freedom to implement a new feature here. >> Now, setting new `org-custom-properties-hide-emptied-drawers' to >> non-nil will result in hiding the whole property drawer if it >> contains only org-custom-properties. > > I don't think this is a good idea. AFAIR, we always refused to hide > completely anything, including empty drawers. The reason is that if the > drawer is completely hidden, you cannot expand it easily, or even know > there is one. > > In any case, this change shouldn't belong to this patch set, and should > be discussed separately. > >> 4. This patch should work against 1aa095ccf. However, the merge was not >> trivial here. Recent commits actively used the fact that drawers and >> outlines are hidden via 'outline invisibility spec, which is not the >> case in this branch. I am not confident that I did not break anything >> during the merge, especially 1aa095ccf. > > [...] > >> Also, I have seen some optimisations making use of the fact that drawers >> and headlines both use 'outline invisibility spec. This change in the >> implementation details supposed to improve performance and should not be >> necessary if this patch is going to be merged. Would it be possible to >> refrain from abusing this particular implementation detail in the >> nearest commits on master (unless really necessary)? > > To be clear, I didn't intend to make your life miserable. > > However, I had to fix regression on drawers visibility before Org 9.4 > release. Also, merging invisibility properties for drawers and outline > was easier for me. So, I had the opportunity to kill two birds with one > stone. > > As a reminder, Org 9.4 is about to be released, but Org 9.5 will take > months to go out. So, even though I hope your changes will land into > Org, there is no reason for us to refrain from improving (actually > fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such > changes are not expected to happen anymore. > > I hope you understand. > >> I would like to finalise the current patch and work on other code using >> overlays separately. This patch is already quite complicated as is. I do >> not want to introduce even more potential bugs by working on things not >> directly affected by this version of the patch. > > The patch is technically mostly good, but needs more work for > integration into Org. > > First, it includes a few unrelated changes that should be removed (e.g., > white space fixes in unrelated parts of the code). Also, as written > above, the changes about `org-custom-properties-hide-emptied-drawers' > should be removed for the time being. > > Once done, I think we should move (or copy, first) _all_ folding-related > functions into a new "org-fold.el" library. Functions and variables > included there should have a proper "org-fold-" prefix. More on this in > the detailed report. > > The functions `org-find-text-property-region', > `org-add-to-list-text-property', and > `org-remove-from-list-text-property' can be left in "org-macs.el", since > they do not directly depend on the `invisible' property. Note the last > two functions I mentioned are not used throughout your patch. They might > be removed. > > This first patch can coexist with overlay folding since functions in > both mechanisms are named differently. > > Then, another patch can integrate "org-fold.el" into Org folding. I also > suggest to move the Outline -> Org transition to yet another patch. > I think there's more work to do on this part. > > Now, into the details of your patch. The first remarks are: > > 1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not > sure), so some functions cannot be used. > > 2. we don't use "subr-x.el" in the code base. In particular, it would be > nice to replace `when-let' with `when' + `let'. This change costs > only one loc. > > 3. Some docstrings need more work. In particular, Emacs documentation > expects all arguments to be explained in the docstring, if possible > in the order in which they appear. There are exceptions, though. For > example, in a function like `org-remove-text-properties', you can > mention arguments are simply the same as in `remove-text-properties'. > > 4. Some refactorization is needed in some places. I mentioned them in > the report below. > > 5. I didn't dive much into the Isearch code so far. I tested it a bit > and seems to work nicely. I noticed one bug though. In the following > document: > > #+begin: foo > :FOO: > bar > :END: > #+end > bar > > when both the drawer and the block are folded (i.e., you fold the > drawer first, then the block), searching for "bar" first find the > last one, then overwraps and find the first one. > > 6. Since we're rewriting folding code, we might as well rename folding > properties: org-hide-drawer -> org-fold-drawer, outline -> > org-fold-headline… > > Now, here are more comments about the code. > > ----- > >> +(defun org-remove-text-properties (start end properties &optional object) > > IMO, this generic name doesn't match the specialized nature of the > function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library. > >> + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. > > Line is too long. Suggestion: > > Remove text properties except folding-related ones. > >> +Do not remove invisible text properties specified by 'outline, >> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this >> +is needed to keep outlines, drawers, and blocks hidden unless they are >> +toggled by user. > > Said properties should be moved into a defconst, e.g., > `org-fold-properties', then: > > Remove text properties as in `remove-text-properties'. See the > function for the description of the arguments. > > However, do not remove invisible text properties defined in > `org-fold-properties'. Those are required to keep headlines, drawers > and blocks folded. > >> +Note: The below may be too specific and create troubles if more >> +invisibility specs are added to org in future" > > You can remove the note. If you think the note is important, it should > put a comment in the code instead. > >> + (when (plist-member properties 'invisible) >> + (let ((pos start) >> + next spec) >> + (while (< pos end) >> + (setq next (next-single-property-change pos 'invisible nil end) >> + spec (get-text-property pos 'invisible)) >> + (unless (memq spec (list 'org-hide-block >> + 'org-hide-drawer >> + 'outline)) > > The (list ...) should be moved outside the `while' loop. Better, this > should be a constant defined somewhere. I also suggest to move > `outline' to `org-outline' since we differ from Outline mode. > >> + (remove-text-properties pos next '(invisible nil) object)) >> + (setq pos next)))) >> + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) > > Typo here. There should a single pair of parenthesis, but see above > about `when-let'. > >> + (remove-text-properties start end properties-stripped object))) >> + >> +(defun org--find-text-property-region (pos prop) > > I think this is a function useful enough to have a name without double > dashes. It can be left in "org-macs.el". It would be nice to have > a wrapper for `invisible' property in "org-fold.el", tho. > >> + "Find a region containing PROP text property around point POS." > > Reverse the order of arguments in the docstring: > > Find a region around POS containing PROP text property. > >> + (let* ((beg (and (get-text-property pos prop) pos)) >> + (end beg)) >> + (when beg > > BEG can only be nil if arguments are wrong. In this case, you can > throw an error (assuming this is no longer an internal function): > > (unless beg (user-error "...")) > >> + ;; when beg is the first point in the region, `previous-single-property-change' >> + ;; will return nil. > > when -> When > >> + (setq beg (or (previous-single-property-change pos prop) >> + beg)) >> + ;; when end is the last point in the region, `next-single-property-change' >> + ;; will return nil. > > Ditto. > >> + (setq end (or (next-single-property-change pos prop) >> + end)) >> + (unless (= beg end) ; this should not happen > > I assume this will be the case in an empty buffer. Anyway, (1 . 1) > sounds more regular than a nil return value, not specified in the > docstring. IOW, I suggest to remove this check. > >> + (cons beg end))))) >> + >> +(defun org--add-to-list-text-property (from to prop element) >> + "Add element to text property PROP, whos value should be a list." > > The docstring is incomplete. All arguments need to be described. Also, > I suggest: > > Append ELEMENT to the value of text property PROP. > >> + (add-text-properties from to `(,prop ,(list element))) ; create if none > > Here, you are resetting all the properties before adding anything, > aren't you? IOW, there might be a bug there. > >> + ;; add to existing >> + (alter-text-property from to >> + prop >> + (lambda (val) >> + (if (member element val) >> + val >> + (cons element val))))) > >> +(defun org--remove-from-list-text-property (from to prop element) >> + "Remove ELEMENT from text propery PROP, whos value should be a list." > > The docstring needs to be improved. > >> + (let ((pos from)) >> + (while (< pos to) >> + (when-let ((val (get-text-property pos prop))) >> + (if (equal val (list element)) > > (list element) needs to be moved out of the `while' loop. > >> + (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil)) >> + (put-text-property pos (next-single-char-property-change pos prop nil to) >> + prop (remove element (get-text-property pos prop))))) > > If we specialize the function, `remove' -> `remq' > >> + (setq pos (next-single-char-property-change pos prop nil to))))) > > Please factor out `next-single-char-property-change'. > > Note that `org--remove-from-list-text-property' and > `org--add-to-list-text-property' do not seem to be used throughout > your patch. > >> +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block) >> + "Priority of invisibility specs.") > > This should be the constant I wrote about earlier. Note that those are > not "specs", just properties. I suggest to rename it. > >> +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only) > > This name is waaaaaaay too long. > >> + "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC. > > Maybe: > > > Return a unique symbol suitable for `invisible' property. > > Then: > > Return value is meant to be used as a buffer-local variable in > current buffer, or BUFFER if this is non-nil. > >> +If the buffer already have buffer-local setup in `char-property-alias-alist' >> +and the setup appears to be created for different buffer, >> +copy the old invisibility state into new buffer-local text properties, >> +unless RETURN-ONLY is non-nil." >> + (if (not (member spec org--invisible-spec-priority-list)) >> + (user-error "%s should be a valid invisibility spec" spec) > > No need to waste an indentation level for that: > > (unless (member …) > (user-error "%S should be …" spec)) > > Also, this is a property, not a "spec". > >> + (let* ((buf (or buffer (current-buffer)))) >> + (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S" > > This clearly needs a shorter name. In particular, "buffer-local" can be removed. > >> + (symbol-name spec) >> + ;; (sxhash buf) appears to be not constant over time. >> + ;; Using buffer-name is safe, since the only place where >> + ;; buffer-local text property actually matters is an indirect >> + ;; buffer, where the name cannot be same anyway. >> + (sxhash (buffer-name buf)))))) > > >> + (prog1 >> + local-prop > > Please move LOCAL-PROP after the (unless return-only ...) sexp. > >> + (unless return-only >> + (with-current-buffer buf >> + (unless (member local-prop (alist-get 'invisible char-property-alias-alist)) >> + ;; copy old property > > "Copy old property." > >> + (dolist (old-prop (alist-get 'invisible char-property-alias-alist)) > > We cannot use `alist-get', which was added in Emacs 25.3 only. > >> + (org-with-wide-buffer >> + (let* ((pos (point-min)) >> + (spec (seq-find (lambda (spec) >> + (string-match-p (symbol-name spec) >> + (symbol-name old-prop))) >> + org--invisible-spec-priority-list)) > > Likewise, we cannot use `seq-find'. > >> + (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))) >> + (while (< pos (point-max)) >> + (when-let (val (get-text-property pos old-prop)) >> + (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val)) >> + (setq pos (next-single-char-property-change pos old-prop)))))) >> + (setq-local char-property-alias-alist >> + (cons (cons 'invisible >> + (mapcar (lambda (spec) >> + (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)) >> + org--invisible-spec-priority-list)) >> + (remove (assq 'invisible char-property-alias-alist) >> + char-property-alias-alist))))))))))) > > This begs for explainations in the docstring or as comments. In > particular, just by reading the code, I have no clue about how this is > going to be used, how it is going to solve issues with indirect > buffers, with invisibility stacking, etc. > > I don't mind if there are more comment lines than lines of code in > that area. > >> - (remove-overlays from to 'invisible spec) >> - ;; Use `front-advance' since text right before to the beginning of >> - ;; the overlay belongs to the visible line than to the contents. >> - (when flag >> - (let ((o (make-overlay from to nil 'front-advance))) >> - (overlay-put o 'evaporate t) >> - (overlay-put o 'invisible spec) >> - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) >> - >> + (with-silent-modifications >> + (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil)) >> + (when flag >> + (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec)))) > > I don't think there is a need for `remove-text-properties' in every > case. Also, (org--get-buffer-local-invisible-property-symbol spec) > should be factored out. > > I suggest: > > (with-silent-modifications > (let ((property (org--get-buffer-local-invisible-property-symbol spec))) > (if flag > (put-text-property from to property spec) > (remove-text-properties from to (list property nil))))) > >> +(defun org-after-change-function (from to len) > > This is a terrible name. Org may add different functions in a-c-f, > they cannot all be called like this. Assuming the "org-fold" prefix, > it could be: > > org-fold--fix-folded-region > >> + "Process changes in folded elements. >> +If a text was inserted into invisible region, hide the inserted text. >> +If the beginning/end line of a folded drawer/block was changed, unfold it. >> +If a valid end line was inserted in the middle of the folded drawer/block, unfold it." > > Nitpick: please do not skip lines amidst a function. Empty lines are > used to separate functions, so this is distracting. > > If a part of the function should stand out, a comment explaining what > the part is doing is enough. > >> + ;; re-hide text inserted in the middle of a folded region > > Re-hide … folded region. > >> + (dolist (spec org--invisible-spec-priority-list) >> + (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec))) >> + (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec)))) >> + (when (eq spec-to spec-from) >> + (org-flag-region from to 't spec-to)))) > > This part should first check if we're really after an insertion, e.g., > if FROM is different from TO, and exit early if that's not the case. > > Also, no need to quote t. > >> + ;; Process all the folded text between `from' and `to' >> + (org-with-wide-buffer >> + >> + (if (< to from) >> + (let ((tmp from)) >> + (setq from to) >> + (setq to tmp))) > > I'm surprised you need to do that. Did you encounter a case where > a-c-f was called with boundaries in reverse order? > >> + ;; Include next/previous line into the changed region. >> + ;; This is needed to catch edits in beginning line of a folded >> + ;; element. >> + (setq to (save-excursion (goto-char to) (forward-line) (point))) > > (forward-line) (point) ---> (line-beginning-position 2) > >> + (setq from (save-excursion (goto-char from) (forward-line -1) (point))) > > (forward-line -1) (point) ---> (line-beginning-position 0) > > Anyway, I have the feeling this is not a good idea to extend it now, > without first checking that we are in a folded drawer or block. It may > also catch unwanted parts, e.g., a folded drawer ending on the line > above. > > What about first finding the whole region with property > > (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer) > > then extending the initial part to include the drawer opening? I don't > think we need to extend past the ending part, because drawer closing > line is always included in the invisible part of the drawer. > >> + ;; Expand the considered region to include partially present folded >> + ;; drawer/block. >> + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) > > Please factor out (org--get-buffer-local-invisible-property-symbol > XXX), this is difficult to read. > >> + ;; check folded drawers > > Check folded drawers. > >> + (let ((pos from)) >> + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + (setq pos (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + (while (< pos to) >> + (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)) >> + pos)) >> + (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))) >> + >> + (let (unfold?) >> + ;; the line before folded text should be beginning of the drawer >> + (save-excursion >> + (goto-char drawer-begin) >> + (backward-char) > > Why `backward-char'? > >> + (beginning-of-line) >> + (unless (looking-at-p org-drawer-regexp) > > looking-at-p ---> looking-at > > However, you must wrap this function within `save-match-data'. > >> + (setq unfold? t))) >> + ;; the last line of the folded text should be :END: >> + (save-excursion >> + (goto-char drawer-end) >> + (beginning-of-line) >> + (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re)) >> + (setq unfold? t))) >> + ;; there should be no :END: anywhere in the drawer body >> + (save-excursion >> + (goto-char drawer-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-property-end-re >> + (max (point) >> + (1- (save-excursion >> + (goto-char drawer-end) >> + (line-beginning-position)))) >> + 't))) > >> (max (point) >> (save-excursion (goto-char drawer-end) (line-end-position 0)) > >> + (setq unfold? t))) >> + ;; there should be no new entry anywhere in the drawer body >> + (save-excursion >> + (goto-char drawer-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-outline-regexp-bol >> + (max (point) >> + (1- (save-excursion >> + (goto-char drawer-end) >> + (line-beginning-position)))) >> + 't))) >> + (setq unfold? t))) > > In the phase above, you need to bail out as soon as unfold? is non-nil: > > (catch :exit > ... > (throw :exit (setq unfold? t)) > ...) > > Also last two checks should be lumped together, with an appropriate > regexp. > > Finally, I have the feeling we're missing out some early exits when > nothing is folded around point (e.g., most of the case). > >> + >> + (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer)))) >> + >> + (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))) >> + >> + ;; check folded blocks >> + (let ((pos from)) >> + (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + (setq pos (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) >> + (while (< pos to) >> + (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)) >> + pos)) >> + (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))) >> + >> + (let (unfold?) >> + ;; the line before folded text should be beginning of the block >> + (save-excursion >> + (goto-char block-begin) >> + (backward-char) >> + (beginning-of-line) >> + (unless (looking-at-p org-dblock-start-re) >> + (setq unfold? t))) >> + ;; the last line of the folded text should be end of the block >> + (save-excursion >> + (goto-char block-end) >> + (beginning-of-line) >> + (unless (looking-at-p org-dblock-end-re) >> + (setq unfold? t))) >> + ;; there should be no #+end anywhere in the block body >> + (save-excursion >> + (goto-char block-begin) >> + (when (save-excursion >> + (re-search-forward org-dblock-end-re >> + (max (point) >> + (1- (save-excursion >> + (goto-char block-end) >> + (line-beginning-position)))) >> + 't)) >> + (setq unfold? t))) >> + ;; there should be no new entry anywhere in the block body >> + (save-excursion >> + (goto-char block-begin) >> + (when (save-excursion >> + (let ((case-fold-search t)) >> + (re-search-forward org-outline-regexp-bol >> + (max (point) >> + (1- (save-excursion >> + (goto-char block-end) >> + (line-beginning-position)))) >> + 't))) >> + (setq unfold? t))) >> + >> + (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block)))) >> + >> + (setq pos >> + (next-single-char-property-change pos >> + (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))))) > > See remarks above. The parts related to drawers and blocks are so > similar they should be factorized out. > > Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we > want here. The correct regexps would be: > > (rx bol > (zero-or-more (any " " "\t")) > "#+begin" > (or ":" > (seq "_" > (group (one-or-more (not (syntax whitespace))))))) > > and closing line should match match-group 1 from the regexp above, e.g.: > > (concat (rx bol (zero-or-more (any " " "\t")) "#+end") > (if block-type > (concat "_" > (regexp-quote block-type) > (rx (zero-or-more (any " " "\t")) eol)) > (rx (opt ":") (zero-or-more (any " " "\t")) eol))) > > assuming `block-type' is the type of the block, or nil, i.e., > (match-string 1) in the previous regexp. > >> - (pcase (get-char-property-and-overlay (point) 'invisible) >> + (pcase (get-char-property (point) 'invisible) >> ;; Do not fold already folded drawers. >> - (`(outline . ,o) (goto-char (overlay-end o))) >> + ('outline > > 'outline --> `outline > >> (end-of-line)) >> (while (and (< arg 0) (re-search-backward regexp nil :move)) >> (unless (bobp) >> - (while (pcase (get-char-property-and-overlay (point) 'invisible) >> - (`(outline . ,o) >> - (goto-char (overlay-start o)) >> - (re-search-backward regexp nil :move)) >> - (_ nil)))) >> + (pcase (get-char-property (point) 'invisible) >> + ('outline >> + (goto-char (car (org--find-text-property-region (point) 'invisible))) >> + (beginning-of-line)) >> + (_ nil))) > > Does this move to the beginning of the widest invisible part around > point? If that's not the case, we need a function in "org-fold.el" > doing just that. Or we need to nest `while' loops as it was the case > in the code you reverted. > > ----- > > Regards, > > -- > Nicolas Goaziou ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-08-11 6:45 ` Ihor Radchenko @ 2020-08-11 23:07 ` Kyle Meyer 2020-08-12 6:29 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kyle Meyer @ 2020-08-11 23:07 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko writes: >> 'outline --> `outline > > Could you explain why? Compatibility. pcase learned that in Emacs 25, IIRC. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-08-11 23:07 ` Kyle Meyer @ 2020-08-12 6:29 ` Ihor Radchenko 2020-09-20 5:53 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-08-12 6:29 UTC (permalink / raw) To: Kyle Meyer; +Cc: emacs-orgmode >>> 'outline --> `outline >> >> Could you explain why? > > Compatibility. pcase learned that in Emacs 25, IIRC. Thanks for the explanation. Fixed now in my local branch. I will send the updated version of the patch after more edits unless someone specifically need to fix this change to make patch work on their system. Best, Ihor Kyle Meyer <kyle@kyleam.com> writes: > Ihor Radchenko writes: > >>> 'outline --> `outline >> >> Could you explain why? > > Compatibility. pcase learned that in Emacs 25, IIRC. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-08-12 6:29 ` Ihor Radchenko @ 2020-09-20 5:53 ` Ihor Radchenko 2020-09-20 11:45 ` Kévin Le Gouguec 2020-12-04 5:58 ` [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko 0 siblings, 2 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-09-20 5:53 UTC (permalink / raw) To: Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, Bastien Cc: emacs-orgmode Hello, > There are still known problems though. The patch currently breaks many > org-mode tests when running =make test=. It is partially because some > tests assume overlays to be used for folding and partially because the > patch appears to break certain folding conventions. I am still > investigating this (and learning =ert=). All the tests are passing now. The current version of the patch (against master) is in https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef The patch is stable on my system for last several months. There are still some minor issues here and there, but it is getting harder for me to find any problems by myself. I need help from interested users to review and/or test the patch. Best, Ihor Ihor Radchenko <yantar92@gmail.com> writes: >>>> 'outline --> `outline >>> >>> Could you explain why? >> >> Compatibility. pcase learned that in Emacs 25, IIRC. > > Thanks for the explanation. Fixed now in my local branch. > > I will send the updated version of the patch after more edits unless > someone specifically need to fix this change to make patch work on their > system. > > Best, > Ihor > > > Kyle Meyer <kyle@kyleam.com> writes: > >> Ihor Radchenko writes: >> >>>> 'outline --> `outline >>> >>> Could you explain why? >> >> Compatibility. pcase learned that in Emacs 25, IIRC. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-20 5:53 ` Ihor Radchenko @ 2020-09-20 11:45 ` Kévin Le Gouguec 2020-09-22 9:05 ` Ihor Radchenko 2020-12-04 5:58 ` [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-09-20 11:45 UTC (permalink / raw) To: emacs-orgmode Hi! Ihor Radchenko <yantar92@gmail.com> writes: > The current version of the patch (against master) is in > https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef I'm probably missing something obvious, but when applying your patch on top of master[1], make fails when generating manuals: > emacs -Q -batch --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ > --eval '(add-to-list '"'"'load-path "../lisp")' \ > --eval '(load "../mk/org-fixup.el")' \ > --eval '(org-make-manuals)' > Loading /home/peniblec/Downloads/sources/emacs-meta/org-mode/mk/org-fixup.el (source)... > Before first headline at position 760959 in buffer org-manual.org<2> > make[1]: *** [Makefile:31: org.texi] Error 255 I've tried going to doc/, running emacs -Q --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ --eval '(add-to-list '"'"'load-path "../lisp")' \ --eval '(load "../mk/org-fixup.el")' then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't get a stacktrace. I'm guessing this is because this error (which IIUC originates from org-back-to-heading in org.el) is a user-error; however, if I change the function to raise a "regular error", then everything compiles fine… 😕 [1] git apply --3way, on top of commit b64ba64fe. I get a conflict in org.el, on the hunk where org-reveal-location and org-show-context-detail are defined; since your patch just deletes them, I resolve this with: git checkout --theirs -- lisp/org.el ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-20 11:45 ` Kévin Le Gouguec @ 2020-09-22 9:05 ` Ihor Radchenko 2020-09-22 10:00 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-09-22 9:05 UTC (permalink / raw) To: Kévin Le Gouguec, emacs-orgmode > I get a conflict in org.el, on the hunk where org-reveal-location > and org-show-context-detail are defined; since your patch just > deletes them, I resolve this with: That's because the patch was against 0afef17e1. The new version of the patch (same URL) is against aea1109ef now. > I've tried going to doc/, running > > emacs -Q --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ > --eval '(add-to-list '"'"'load-path "../lisp")' \ > --eval '(load "../mk/org-fixup.el")' > > then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't > get a stacktrace. I'm guessing this is because this error (which IIUC > originates from org-back-to-heading in org.el) is a user-error; however, > if I change the function to raise a "regular error", then everything > compiles fine… 😕 I suspect that you forgot to run =make clean= (to remove old untracked .elc files). Best, Ihor Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Hi! > > Ihor Radchenko <yantar92@gmail.com> writes: > >> The current version of the patch (against master) is in >> https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef > > I'm probably missing something obvious, but when applying your patch on > top of master[1], make fails when generating manuals: > >> emacs -Q -batch --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ >> --eval '(add-to-list '"'"'load-path "../lisp")' \ >> --eval '(load "../mk/org-fixup.el")' \ >> --eval '(org-make-manuals)' >> Loading /home/peniblec/Downloads/sources/emacs-meta/org-mode/mk/org-fixup.el (source)... >> Before first headline at position 760959 in buffer org-manual.org<2> >> make[1]: *** [Makefile:31: org.texi] Error 255 > > I've tried going to doc/, running > > emacs -Q --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ > --eval '(add-to-list '"'"'load-path "../lisp")' \ > --eval '(load "../mk/org-fixup.el")' > > then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't > get a stacktrace. I'm guessing this is because this error (which IIUC > originates from org-back-to-heading in org.el) is a user-error; however, > if I change the function to raise a "regular error", then everything > compiles fine… 😕 > > > [1] git apply --3way, on top of commit b64ba64fe. > > I get a conflict in org.el, on the hunk where org-reveal-location > and org-show-context-detail are defined; since your patch just > deletes them, I resolve this with: > > git checkout --theirs -- lisp/org.el ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-22 9:05 ` Ihor Radchenko @ 2020-09-22 10:00 ` Ihor Radchenko 2020-09-23 6:16 ` Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-09-22 10:00 UTC (permalink / raw) To: Kévin Le Gouguec, emacs-orgmode >> then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't >> get a stacktrace. I'm guessing this is because this error (which IIUC >> originates from org-back-to-heading in org.el) is a user-error; however, >> if I change the function to raise a "regular error", then everything >> compiles fine… 😕 > > I suspect that you forgot to run =make clean= (to remove old untracked > .elc files). I was wrong. It was actually a problem with org-back-to-heading. Should be fixed now. Best, Ihor Ihor Radchenko <yantar92@gmail.com> writes: >> I get a conflict in org.el, on the hunk where org-reveal-location >> and org-show-context-detail are defined; since your patch just >> deletes them, I resolve this with: > > That's because the patch was against 0afef17e1. The new version of the > patch (same URL) is against aea1109ef now. > >> I've tried going to doc/, running >> >> emacs -Q --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ >> --eval '(add-to-list '"'"'load-path "../lisp")' \ >> --eval '(load "../mk/org-fixup.el")' >> >> then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't >> get a stacktrace. I'm guessing this is because this error (which IIUC >> originates from org-back-to-heading in org.el) is a user-error; however, >> if I change the function to raise a "regular error", then everything >> compiles fine… 😕 > > I suspect that you forgot to run =make clean= (to remove old untracked > .elc files). > > Best, > Ihor > > Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > >> Hi! >> >> Ihor Radchenko <yantar92@gmail.com> writes: >> >>> The current version of the patch (against master) is in >>> https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef >> >> I'm probably missing something obvious, but when applying your patch on >> top of master[1], make fails when generating manuals: >> >>> emacs -Q -batch --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ >>> --eval '(add-to-list '"'"'load-path "../lisp")' \ >>> --eval '(load "../mk/org-fixup.el")' \ >>> --eval '(org-make-manuals)' >>> Loading /home/peniblec/Downloads/sources/emacs-meta/org-mode/mk/org-fixup.el (source)... >>> Before first headline at position 760959 in buffer org-manual.org<2> >>> make[1]: *** [Makefile:31: org.texi] Error 255 >> >> I've tried going to doc/, running >> >> emacs -Q --eval '(setq vc-handled-backends nil org-startup-folded nil)' \ >> --eval '(add-to-list '"'"'load-path "../lisp")' \ >> --eval '(load "../mk/org-fixup.el")' >> >> then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't >> get a stacktrace. I'm guessing this is because this error (which IIUC >> originates from org-back-to-heading in org.el) is a user-error; however, >> if I change the function to raise a "regular error", then everything >> compiles fine… 😕 >> >> >> [1] git apply --3way, on top of commit b64ba64fe. >> >> I get a conflict in org.el, on the hunk where org-reveal-location >> and org-show-context-detail are defined; since your patch just >> deletes them, I resolve this with: >> >> git checkout --theirs -- lisp/org.el ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-22 10:00 ` Ihor Radchenko @ 2020-09-23 6:16 ` Kévin Le Gouguec 2020-09-23 6:48 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-09-23 6:16 UTC (permalink / raw) To: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: >>> then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't >>> get a stacktrace. I'm guessing this is because this error (which IIUC >>> originates from org-back-to-heading in org.el) is a user-error; however, >>> if I change the function to raise a "regular error", then everything >>> compiles fine… 😕 >> >> I suspect that you forgot to run =make clean= (to remove old untracked >> .elc files). > > I was wrong. It was actually a problem with org-back-to-heading. Should > be fixed now. Thanks! The new patch applies cleanly (to aea1109ef), and "make" runs to completion. I have seen no obvious breakage so far; I'll make sure to report if anything funny shows up. Apologies for maybe changing the subject, but earlier this summer you mentioned[1] you were working on a patch to the folding system that would fix an issue I have[2] with LOGBOOKs since 9.4. AFAICT the patch you are sharing now does not fix that; is this issue still on your radar? At any rate, thank you for your work! [1] https://orgmode.org/list/87r1ts3s8r.fsf@localhost/ [2] https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/ tl;dr even with #+STARTUP: overview, isearching opens all logbooks near search results, even though there are no matches inside logbooks themselves. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-23 6:16 ` Kévin Le Gouguec @ 2020-09-23 6:48 ` Ihor Radchenko 2020-09-23 7:09 ` Bastien 2020-09-24 18:07 ` Kévin Le Gouguec 0 siblings, 2 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-09-23 6:48 UTC (permalink / raw) To: Kévin Le Gouguec, emacs-orgmode > Apologies for maybe changing the subject, but earlier this summer you > mentioned[1] you were working on a patch to the folding system that > would fix an issue I have[2] with LOGBOOKs since 9.4. AFAICT the patch > you are sharing now does not fix that; is this issue still on your > radar? Thanks for reporting! I accidentally reintroduced the bug because of mistake when converting org-hide-drawers to new folding library. (:facepalm:). Should be fixed in the gist now. Best, Ihor Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >>>> then M-x toggle-debug-on-error and M-: (org-make-manuals), but I can't >>>> get a stacktrace. I'm guessing this is because this error (which IIUC >>>> originates from org-back-to-heading in org.el) is a user-error; however, >>>> if I change the function to raise a "regular error", then everything >>>> compiles fine… 😕 >>> >>> I suspect that you forgot to run =make clean= (to remove old untracked >>> .elc files). >> >> I was wrong. It was actually a problem with org-back-to-heading. Should >> be fixed now. > > Thanks! The new patch applies cleanly (to aea1109ef), and "make" runs > to completion. > > I have seen no obvious breakage so far; I'll make sure to report if > anything funny shows up. > > > Apologies for maybe changing the subject, but earlier this summer you > mentioned[1] you were working on a patch to the folding system that > would fix an issue I have[2] with LOGBOOKs since 9.4. AFAICT the patch > you are sharing now does not fix that; is this issue still on your > radar? > > > At any rate, thank you for your work! > > > [1] https://orgmode.org/list/87r1ts3s8r.fsf@localhost/ > [2] https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/ > > tl;dr even with #+STARTUP: overview, isearching opens all logbooks > near search results, even though there are no matches inside > logbooks themselves. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-23 6:48 ` Ihor Radchenko @ 2020-09-23 7:09 ` Bastien 2020-09-23 7:30 ` Ihor Radchenko 2020-09-24 18:07 ` Kévin Le Gouguec 1 sibling, 1 reply; 192+ messages in thread From: Bastien @ 2020-09-23 7:09 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, Kévin Le Gouguec Hi Ihor, Ihor Radchenko <yantar92@gmail.com> writes: > Thanks for reporting! I accidentally reintroduced the bug because of > mistake when converting org-hide-drawers to new folding library. > (:facepalm:). > > Should be fixed in the gist now. Can you share this gist as a patch against Org's current master? -- Bastien ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-23 7:09 ` Bastien @ 2020-09-23 7:30 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-09-23 7:30 UTC (permalink / raw) To: Bastien; +Cc: emacs-orgmode, Kévin Le Gouguec > Can you share this gist as a patch against Org's current master? That is not possible. The underlying reason of the bug in the patch is different from master. On master, the overlays for folded drawers and headlines are merged together - when folded headline is opened by isearch, everything is revealed. The fix would involve special logic re-hiding drawers when necessary. On the org-fold feature branch, the drawers and headlines are folded independently. The reason why the bug persisted was my mistake in org-hide-drawers - I skipped drawers inside folded headlines, even when the drawers themselves were not folded. In my case the fix was trivial - I replaced condition when to skip drawer at point: [any fold is present at point] -> [drawer fold is present at point] (org-fold-get-folding-spec) -> (org-fold-get-folding-spec (org-fold-get-folding-spec-for-element 'drawer)) So, the fix is only relevant to the whole org-fold branch. Best, Ihor Bastien <bzg@gnu.org> writes: > Hi Ihor, > > Ihor Radchenko <yantar92@gmail.com> writes: > >> Thanks for reporting! I accidentally reintroduced the bug because of >> mistake when converting org-hide-drawers to new folding library. >> (:facepalm:). >> >> Should be fixed in the gist now. > > Can you share this gist as a patch against Org's current master? > > -- > Bastien ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-23 6:48 ` Ihor Radchenko 2020-09-23 7:09 ` Bastien @ 2020-09-24 18:07 ` Kévin Le Gouguec 2020-09-25 2:16 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-09-24 18:07 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Thanks for reporting! I accidentally reintroduced the bug because of > mistake when converting org-hide-drawers to new folding library. > (:facepalm:). > > Should be fixed in the gist now. Can confirm, thanks! I understand from your answer to Bastien's query that this fix is specific to your branch; would it be hard to backport it to Org's maint branch? Otherwise IIUC Org 9.4 will keep this regression, and users will have to wait until Org 9.5 for a fix. Also, just in case there's been a misunderstanding: Bastien <bzg@gnu.org> writes: > Can you share this gist as a patch against Org's current master? Bastien asked for the /gist/ as a patch against master, whereas your answer explained why you couldn't share the /fix/ as a patch against master. If Bastien did mean the whole gist, here is the corresponding patch against master: https://gist.githubusercontent.com/yantar92/6447754415457927293acda43a7fcaef/raw/7e43948e6c21220661534b79770bc1a6784b7893/featuredrawertextprop.patch Apologies if I'm the one misunderstanding, and thank you for all your efforts! ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-24 18:07 ` Kévin Le Gouguec @ 2020-09-25 2:16 ` Ihor Radchenko 2020-12-15 17:38 ` [9.4] Fixing logbook visibility during isearch Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-09-25 2:16 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode > I understand from your answer to Bastien's query that this fix is > specific to your branch; would it be hard to backport it to Org's maint > branch? Otherwise IIUC Org 9.4 will keep this regression, and users > will have to wait until Org 9.5 for a fix. The problem is that fix in my branch has nothing to do with main branch. The bugs were inherently different even though looked same from user point of view. If one wants to make the fix work on master, the whole branch must be applied. However, I can try to suggest a way to fix the issue on master. The way isearch handles folded text in org is set from org-flag-region (org-macs.el): (overlay-put o 'isearch-open-invisible (lambda (&rest _) (org-show-context 'isearch))) It means that isearch calls org-show-context (org.el) to reveal hidden text. Then, it calls org-show-set-visibility with argument defined in org-show-context-detail (now, it is 'lineage). With current defaults, the searched text is revealed using org-flag-heading, which reveals both heading body and drawers. The easiest way to write the fix would be changing org-flag-heading directly, but there might be unforeseen consequences on other folding commands. Another way would be changing the way org-show-set-visibility handles 'lineage argument. Again, it may affect other things. Finally, one can add an extra possible argument to org-show-set-visibility and alter default value of org-show-context-detail accordingly. The last way will have least risk to break something else. I guess, patches welcome ;) > Bastien asked for the /gist/ as a patch against master, whereas your > answer explained why you couldn't share the /fix/ as a patch against > master. If Bastien did mean the whole gist, here is the corresponding > patch against master: Well. The gist is a patch applying the whole feature/org-fold branch to master. That's not yet something we can do. The plan is to apply the org-fold feature in several steps, as discussed in earlier messages. So, I thought that it would just create confusion if I share the gist as is. Sorry if I was not clear. Best, Ihor Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> Thanks for reporting! I accidentally reintroduced the bug because of >> mistake when converting org-hide-drawers to new folding library. >> (:facepalm:). >> >> Should be fixed in the gist now. > > Can confirm, thanks! > > I understand from your answer to Bastien's query that this fix is > specific to your branch; would it be hard to backport it to Org's maint > branch? Otherwise IIUC Org 9.4 will keep this regression, and users > will have to wait until Org 9.5 for a fix. > > Also, just in case there's been a misunderstanding: > > Bastien <bzg@gnu.org> writes: > >> Can you share this gist as a patch against Org's current master? > > Bastien asked for the /gist/ as a patch against master, whereas your > answer explained why you couldn't share the /fix/ as a patch against > master. If Bastien did mean the whole gist, here is the corresponding > patch against master: > > https://gist.githubusercontent.com/yantar92/6447754415457927293acda43a7fcaef/raw/7e43948e6c21220661534b79770bc1a6784b7893/featuredrawertextprop.patch > > Apologies if I'm the one misunderstanding, and thank you for all your > efforts! ^ permalink raw reply [flat|nested] 192+ messages in thread
* [9.4] Fixing logbook visibility during isearch 2020-09-25 2:16 ` Ihor Radchenko @ 2020-12-15 17:38 ` Kévin Le Gouguec 2020-12-16 3:15 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-15 17:38 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > However, I can try to suggest a way to fix the issue on master. The way > isearch handles folded text in org is set from org-flag-region > (org-macs.el): > > (overlay-put o > 'isearch-open-invisible > (lambda (&rest _) (org-show-context 'isearch))) > > It means that isearch calls org-show-context (org.el) to reveal hidden > text. Then, it calls org-show-set-visibility with argument defined in > org-show-context-detail (now, it is 'lineage). With current defaults, > the searched text is revealed using org-flag-heading, which reveals both > heading body and drawers. > > The easiest way to write the fix would be changing org-flag-heading > directly, but there might be unforeseen consequences on other folding > commands. > > Another way would be changing the way org-show-set-visibility handles > 'lineage argument. Again, it may affect other things. > > Finally, one can add an extra possible argument to > org-show-set-visibility and alter default value of > org-show-context-detail accordingly. > > The last way will have least risk to break something else. > > I guess, patches welcome ;) Since Org 9.4 has landed in the emacs-27 branch, I have renewed interest in finding a fix for this before 27.2 is released (… and more selfishly, before emacs-27 is merged into master 😉). I'm a bit confused, because AFAICT org-show-context is called *after* exiting isearch, so IIUC by the time org-show-set-visibility is called it's too late to undo the damage. Recipe using my repro file[1]: - C-x C-f logbooks.org - M-x toggle-debug-on-entry org-show-context - C-s bug The debugger only fires *after* we exit isearch, and by that time it's too late: my issue comes from all those logbooks cluttering the screen while I'm mashing C-s to iterate through matches. I can try to dig deeper into this, but before doing so: would you have any insight as to what's going on here? [1] wget https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/2-logbooks.org -O tmp/logbooks.org ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-15 17:38 ` [9.4] Fixing logbook visibility during isearch Kévin Le Gouguec @ 2020-12-16 3:15 ` Ihor Radchenko 2020-12-16 18:05 ` Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-12-16 3:15 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > The debugger only fires *after* we exit isearch, and by that time it's > too late: my issue comes from all those logbooks cluttering the screen > while I'm mashing C-s to iterate through matches. > > I can try to dig deeper into this, but before doing so: would you have > any insight as to what's going on here? org-mode is relying on default isearch behaviour during interactive C-s session. By default, isearch simply makes all the overlays at match visible and re-hide them once we move to the next match. In case of org-mode, this reveals drawers as well, since they are in the same overlay with the rest of the folded heading. The way to change default isearch behaviour *during* isearch session is setting undocumented 'isearch-open-invisible-temporary property of the overlay (see isearch-open-overlay-temporary). The function must accept two arguments: overlay and flag. If flag is non-nil, the function should re-hide the overlay text and it should reveal the overlay when flag is nil. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-16 3:15 ` Ihor Radchenko @ 2020-12-16 18:05 ` Kévin Le Gouguec 2020-12-17 3:18 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-16 18:05 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > >> The debugger only fires *after* we exit isearch, and by that time it's >> too late: my issue comes from all those logbooks cluttering the screen >> while I'm mashing C-s to iterate through matches. >> >> I can try to dig deeper into this, but before doing so: would you have >> any insight as to what's going on here? > > org-mode is relying on default isearch behaviour during interactive C-s > session. By default, isearch simply makes all the overlays at match > visible and re-hide them once we move to the next match. In case of > org-mode, this reveals drawers as well, since they are in the same > overlay with the rest of the folded heading. > > The way to change default isearch behaviour *during* isearch session is > setting undocumented 'isearch-open-invisible-temporary property of the > overlay (see isearch-open-overlay-temporary). Thanks for taking the time to explain this. I can't find any reference to this property in Org <9.4 (e.g. 9.3 as shipped in 27.1, where the bug does not happen) so do I understand correctly that the root cause ("since [drawers] are in the same overlay with the rest of the folded heading") dates from Org 9.4? (Just trying to understand if I should keep looking at Org 9.3 for inspiration, or if your proposed solution based on isearch-open-invisible-temporary should be implemented from scratch) ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-16 18:05 ` Kévin Le Gouguec @ 2020-12-17 3:18 ` Ihor Radchenko 2020-12-17 14:50 ` Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-12-17 3:18 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > I can't find any reference to this property in Org <9.4 (e.g. 9.3 as > shipped in 27.1, where the bug does not happen) so do I understand > correctly that the root cause ("since [drawers] are in the same overlay > with the rest of the folded heading") dates from Org 9.4? Yes, the root cause is that overlays used to hide drawers now automatically merge with outline overlays. This was introduced in Org 9.4 to improve performance (too many overlays are handled badly by Emacs). > (Just trying to understand if I should keep looking at Org 9.3 for > inspiration, or if your proposed solution based on > isearch-open-invisible-temporary should be implemented from scratch) You will probably need to implement this from scratch (or use the feature/org-fold branch from github.com/yantar92/org). In Org 9.3 the folded headline looked like the following: * Headline <begin hidden outline overlay> :PROPERTIES:<begin hidden drawer overlay> :PROPERTY1: value1 :PROPERTY2: value2 :END:<end hidden drawer overlay> headline text another line <end hidden outline overlay> When using isearch with "text" search string, the overlay containing "text" is temporarily revealed by isearch (via setting 'invisible property of the overlay to nil): * Headline <begin *visible* outline overlay> :PROPERTES:<begin hidden drawer overlay> :PROPERTY1: value1 :PROPERTY2: value2 :END:<end hidden drawer overlay> headline text another line <end *visible* outline overlay> As you can see, the drawer overlay remains unchanged and hidden. In Org 9.4, drawer overlay does not exist when we fold the headline text and isearch reveals everything. To work around this issue, you need to hook into the way isearch reveals hidden match by setting 'isearch-open-invisible-temporary property of the overlays to custom function (you can set the property inside org-flag-region). The function should re-hide the drawers when matching text is not inside the drawer. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-17 3:18 ` Ihor Radchenko @ 2020-12-17 14:50 ` Kévin Le Gouguec 2020-12-18 2:23 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-17 14:50 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > You will probably need to implement this from scratch (or use the > feature/org-fold branch from github.com/yantar92/org). Gotcha. TBH I don't know if I'll have the time to cook up a patch before 27.2 is released; all the same, I appreciate you taking the time to explain all this. Since the changes in Org 9.4 aimed at improving performance, is there a test case somewhere in the "Mitigating the poor Emacs performance on huge org files" thread that could help ensure that a tentative fix will not degrade performance? ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-17 14:50 ` Kévin Le Gouguec @ 2020-12-18 2:23 ` Ihor Radchenko 2020-12-24 23:37 ` Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-12-18 2:23 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Since the changes in Org 9.4 aimed at improving performance, is there a > test case somewhere in the "Mitigating the poor Emacs performance on > huge org files" thread that could help ensure that a tentative fix will > not degrade performance? The first message in the thread ;) I believe it was also used to benchmark the change in 9.4. >> [3] See the attached org file in my Emacs bug report: https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/txte6kQp35VOm.txt Or you can ask me to test. That example file is my stripped someday list, which grew to much larger size since the time I created that example. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-18 2:23 ` Ihor Radchenko @ 2020-12-24 23:37 ` Kévin Le Gouguec 2020-12-25 2:51 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-24 23:37 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > >> Since the changes in Org 9.4 aimed at improving performance, is there a >> test case somewhere in the "Mitigating the poor Emacs performance on >> huge org files" thread that could help ensure that a tentative fix will >> not degrade performance? > > The first message in the thread ;) I believe it was also used to > benchmark the change in 9.4. Thanks for the pointer! I've looked at your branch for inspiration, and my takeaway is that the isearch-open-invisible-temporary route might be too involved for a bugfix, especially if it's going to be reverted wholesale when your branch gets merged. Then again, maybe I'm not smart enough to devise a solution. I wonder if the path of least resistance couldn't be found in org-cycle-hide-drawers: right now this function just skips over drawers which are covered with an invisible overlay, but maybe it should not skip a drawer if the overlay starts before it (i.e. the overlay is not specific to this drawer but covers a whole containing section). ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-24 23:37 ` Kévin Le Gouguec @ 2020-12-25 2:51 ` Ihor Radchenko 2020-12-25 10:59 ` Kévin Le Gouguec 2020-12-25 21:35 ` Kévin Le Gouguec 0 siblings, 2 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-12-25 2:51 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > I've looked at your branch for inspiration, and my takeaway is that the > isearch-open-invisible-temporary route might be too involved for a > bugfix, especially if it's going to be reverted wholesale when your > branch gets merged. Then again, maybe I'm not smart enough to devise a > solution. My current plan is supporting the overlay-based approach even after merging the branch (by default). So, overlays should be around for a while and the issue with drawer visibility will be around as well, unless you fix it. I will probably work on this in distant future, but that's not the priority now. > I wonder if the path of least resistance couldn't be found in > org-cycle-hide-drawers: right now this function just skips over drawers > which are covered with an invisible overlay, but maybe it should not > skip a drawer if the overlay starts before it (i.e. the overlay is not > specific to this drawer but covers a whole containing section). That would defeat the purpose why the number of overlays was reduced in Org 9.4. However, org-cycle-hide-drawers might be called in isearch-open-invisible-temporary. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-25 2:51 ` Ihor Radchenko @ 2020-12-25 10:59 ` Kévin Le Gouguec 2020-12-25 12:32 ` Ihor Radchenko 2020-12-25 21:35 ` Kévin Le Gouguec 1 sibling, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-25 10:59 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > My current plan is supporting the overlay-based approach even after > merging the branch (by default). So, overlays should be around for a > while and the issue with drawer visibility will be around as well, > unless you fix it. I will probably work on this in distant future, but > that's not the priority now. Mmm; is the current state of your branch representative of your plan? If I compile it and run emacs -Q -L $yourbranch/lisp --eval '(setq org-startup-folded t)' $someorgfile Then isearching does not reveal logbook drawers unless matches are found inside, which as far as I am concerned fixes my issue with 9.4. >> I wonder if the path of least resistance couldn't be found in >> org-cycle-hide-drawers: right now this function just skips over drawers >> which are covered with an invisible overlay, but maybe it should not >> skip a drawer if the overlay starts before it (i.e. the overlay is not >> specific to this drawer but covers a whole containing section). > > That would defeat the purpose why the number of overlays was reduced in > Org 9.4. However, org-cycle-hide-drawers might be called in > isearch-open-invisible-temporary. Thanks for the tip. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-25 10:59 ` Kévin Le Gouguec @ 2020-12-25 12:32 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-12-25 12:32 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> My current plan is supporting the overlay-based approach even after >> merging the branch (by default). So, overlays should be around for a >> while and the issue with drawer visibility will be around as well, >> unless you fix it. I will probably work on this in distant future, but >> that's not the priority now. > > Mmm; is the current state of your branch representative of your plan? Not yet. That's rather a big change and I am currently generalising the core org-fold API to support both text properties and overlays. You can see WIP in org-fold-core.el from org-fold-universal-core branch. That branch is not usable yet. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-25 2:51 ` Ihor Radchenko 2020-12-25 10:59 ` Kévin Le Gouguec @ 2020-12-25 21:35 ` Kévin Le Gouguec 2020-12-26 4:14 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-25 21:35 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > However, org-cycle-hide-drawers might be called in > isearch-open-invisible-temporary. This callback receives two arguments: - the overlay which contains a match, - whether we are un-hiding the overlay's span or hiding it back. To get the same behaviour as Org≤9.3, IIUC we want to do the following: 1. When isearch asks us to un-hide, 1. go over all drawers within the overlay, 2. hide those that do not contain a match, by adding an invisible overlay. 2. When isearch asks us to hide back, 1. remove the invisible overlays we have put on these drawers. 1.1. is straightforward: overlay-start and overlay-end tell us where to look for drawers. 1.2. stumps me: is there an isearch API I can use while in the callback to know where the matches are located? For 2.1, I guess we will need to cache the temporary invisible overlays we add during step 1. in a global list; that way when it's time to destroy them, we can simply iterate on the list? (Sorry for being so slow 😕 I never seem to be able to spend more than 10 minutes on this issue before having to switch to something else…) ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-25 21:35 ` Kévin Le Gouguec @ 2020-12-26 4:14 ` Ihor Radchenko 2020-12-26 11:44 ` Kévin Le Gouguec 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-12-26 4:14 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > 1.2. stumps me: is there an isearch API I can use while in the callback > to know where the matches are located? I do not think that there is direct API for this, but the match should be accessible through match-beginning/match-end, as I can see from the isearch.el code. > For 2.1, I guess we will need to cache the temporary invisible overlays > we add during step 1. in a global list; that way when it's time to > destroy them, we can simply iterate on the list? That's what I do in org-fold--isearch-show-temporary. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-26 4:14 ` Ihor Radchenko @ 2020-12-26 11:44 ` Kévin Le Gouguec 2020-12-26 12:22 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Kévin Le Gouguec @ 2020-12-26 11:44 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode Ihor Radchenko <yantar92@gmail.com> writes: > Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > >> 1.2. stumps me: is there an isearch API I can use while in the callback >> to know where the matches are located? > > I do not think that there is direct API for this, but the match should > be accessible through match-beginning/match-end, as I can see from the > isearch.el code. Right, I've seen this too; I wonder if it's a hard guarantee or an implementation detail. I might page help-gnu-emacs about this. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [9.4] Fixing logbook visibility during isearch 2020-12-26 11:44 ` Kévin Le Gouguec @ 2020-12-26 12:22 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2020-12-26 12:22 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: emacs-orgmode Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Ihor Radchenko <yantar92@gmail.com> writes: > >> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: >> >>> 1.2. stumps me: is there an isearch API I can use while in the callback >>> to know where the matches are located? >> >> I do not think that there is direct API for this, but the match should >> be accessible through match-beginning/match-end, as I can see from the >> isearch.el code. > > Right, I've seen this too; I wonder if it's a hard guarantee or an > implementation detail. I might page help-gnu-emacs about this. Another way could by using isearch-filter-predicate. It is given the search region directly. ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-09-20 5:53 ` Ihor Radchenko 2020-09-20 11:45 ` Kévin Le Gouguec @ 2020-12-04 5:58 ` Ihor Radchenko 2021-03-21 9:09 ` Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2020-12-04 5:58 UTC (permalink / raw) To: Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, Bastien Cc: emacs-orgmode Hello, This is an update about the current status of the patch. Since there was not much feedback, I decided to share the up-to-date branch on github, so that people can directly download/clone the whole thing and load it to Emacs without a need to install the patch manually. The github repo is https://github.com/yantar92/org ---- On the progress with the code, I have found many more bugs, which are not critical for me, but should be fixed anyway. I will keep working on them and keep the github repo up to date. One more important thing I wanted to mention is about the way org-fold should be merged on master. I plan to support using overlays within org-fold depending on custom variable. If the variable is set to 'overlay, org fold will use overlays without all the complexity of text property approach. The 'overlay value will be set by default. If a user wants to use text properties, the variable can be customised. The described approach will allow all the users test the text property-based folding as experimental feature (similar to org-element-use-cache). Once we are confident enough that the code is stable, we can just change the default. What do you think? Best, Ihor Ihor Radchenko <yantar92@gmail.com> writes: > Hello, > >> There are still known problems though. The patch currently breaks many >> org-mode tests when running =make test=. It is partially because some >> tests assume overlays to be used for folding and partially because the >> patch appears to break certain folding conventions. I am still >> investigating this (and learning =ert=). > > All the tests are passing now. > The current version of the patch (against master) is in > https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef > > The patch is stable on my system for last several months. There are > still some minor issues here and there, but it is getting harder for me > to find any problems by myself. I need help from interested users to > review and/or test the patch. > > Best, > Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2020-12-04 5:58 ` [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko @ 2021-03-21 9:09 ` Ihor Radchenko 2021-05-03 17:28 ` Bastien 0 siblings, 1 reply; 192+ messages in thread From: Ihor Radchenko @ 2021-03-21 9:09 UTC (permalink / raw) To: Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, Bastien Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 749 bytes --] Hello, This is another update about the status of the patch. I am mostly happy with the current state of the code, got rid of most of the bugs, and did not get any new bug reports in github for a while. I would like to start the process of applying the patch on master. As a first step, I would like to submit the core folding library (org-fold-core) for review. org-fold-core is pretty much independent from org-mode code base and does not affect anything if applied as is. It will be used by org-specific org-fold.el I will finalise and send later. For now, I would like to hear any suggestions about API and implementation of org-fold-core.el. I tried to document all the details in the code. Looking forward for the feedback. Best, Ihor [-- Attachment #2: org-fold-core.el --] [-- Type: application/emacs-lisp, Size: 62759 bytes --] ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2021-03-21 9:09 ` Ihor Radchenko @ 2021-05-03 17:28 ` Bastien 2021-09-21 13:32 ` Timothy 2022-01-29 11:37 ` [PATCH 00/35] Merge org-fold feature branch Ihor Radchenko 0 siblings, 2 replies; 192+ messages in thread From: Bastien @ 2021-05-03 17:28 UTC (permalink / raw) To: Ihor Radchenko Cc: Karl Voit, emacs-orgmode, Kyle Meyer, Christian Heinrich, Nicolas Goaziou Hi Ihor, Ihor Radchenko <yantar92@gmail.com> writes: > This is another update about the status of the patch. Thank you *very much* for this work and sorry for the slow reply. I urge everyone to test this change, as I'd like to include it in Org 9.5 if it's ready. I will test this myself this week and report. Thanks! -- Bastien ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2021-05-03 17:28 ` Bastien @ 2021-09-21 13:32 ` Timothy 2021-10-26 17:25 ` Matt Price 2022-01-29 11:37 ` [PATCH 00/35] Merge org-fold feature branch Ihor Radchenko 1 sibling, 1 reply; 192+ messages in thread From: Timothy @ 2021-09-21 13:32 UTC (permalink / raw) To: Bastien Cc: Karl Voit, Ihor Radchenko, emacs-orgmode, Nicolas Goaziou, Christian Heinrich, Kyle Meyer [-- Attachment #1: Type: text/plain, Size: 445 bytes --] I’m suspect it too short notice for such a large change to make its way into Org 9.5, but Bastien’s release email is certainly a good prompt to bump this. Bastien <bzg@gnu.org> writes: > Thank you *very much* for this work and sorry for the slow reply. > > I urge everyone to test this change, as I’d like to include it in > Org 9.5 if it’s ready. > > I will test this myself this week and report. All the best, Timothy ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2021-09-21 13:32 ` Timothy @ 2021-10-26 17:25 ` Matt Price 2021-10-27 6:27 ` Ihor Radchenko 0 siblings, 1 reply; 192+ messages in thread From: Matt Price @ 2021-10-26 17:25 UTC (permalink / raw) To: Timothy Cc: Karl Voit, Ihor Radchenko, Bastien, Christian Heinrich, Nicolas Goaziou, Org Mode, Kyle Meyer [-- Attachment #1: Type: text/plain, Size: 630 bytes --] On Tue, Sep 21, 2021 at 9:36 AM Timothy <tecosaur@gmail.com> wrote: > I’m suspect it too short notice for such a large change to make its way > into Org > 9.5, but Bastien’s release email is certainly a good prompt to bump this. > > Bastien <bzg@gnu.org> writes: > > > Thank you *very much* for this work and sorry for the slow reply. > > > > I urge everyone to test this change, as I’d like to include it in > > Org 9.5 if it’s ready. > > > > I will test this myself this week and report. > > All the best, > Timothy > Is this code in main now, and do I have to do anything special to test it out? [-- Attachment #2: Type: text/html, Size: 1055 bytes --] ^ permalink raw reply [flat|nested] 192+ messages in thread
* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers 2021-10-26 17:25 ` Matt Price @ 2021-10-27 6:27 ` Ihor Radchenko 0 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2021-10-27 6:27 UTC (permalink / raw) To: Matt Price Cc: Karl Voit, Bastien, Christian Heinrich, Nicolas Goaziou, Org Mode, Kyle Meyer, Timothy Matt Price <moptop99@gmail.com> writes: > Is this code in main now, and do I have to do anything special to test it > out? Not on main yet. I need maintainers to agree about the merge. It is a major change. I plan to prepare a proper patchset and bump the thread again a few weeks later, when the dust settles on the recent org-persist/org-element merge. If you want to test the code, it is available at https://github.com/yantar92/org You can simply clone the github repo and load Org from there. Best, Ihor ^ permalink raw reply [flat|nested] 192+ messages in thread
* [PATCH 00/35] Merge org-fold feature branch 2021-05-03 17:28 ` Bastien 2021-09-21 13:32 ` Timothy @ 2022-01-29 11:37 ` Ihor Radchenko 2022-01-29 11:37 ` [PATCH 01/35] Add org-fold-core: new folding engine Ihor Radchenko ` (35 more replies) 1 sibling, 36 replies; 192+ messages in thread From: Ihor Radchenko @ 2022-01-29 11:37 UTC (permalink / raw) To: Bastien, Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, emacs-orgmode Cc: Ihor Radchenko It took a while, but I am finally done with rebasing the org-fold branch code onto current main. This branch has been tested by me and other volunteers for over a year. Things are basically stable using recent Emacs versions. There were a couple of back-compatibility issues with older Emacs, which I fixed during the cleanup. Those may need to be tested more carefully after merging to main. I would like to thank all the people who helped with bug reporting and provided bugfixes over the testing time. Thank you all - arkhan, HyunggyuJang, Robert Irelan, Alois Janíček, Anders Johansson, Daniel Kraus, Ypot, ntharim, Colin McLear, Yiming Chen, tpeacock19, and Karl Voit. After the cleanup, some patches are not included for the merge. Apart from my patches, 3 patches by Anders Johansson are included here. He has signed FSF copyright paperwork and appears on the Org mode contributor list (https://orgmode.org/worg/contributors.html). Unless there are comments on the patches below, I plan to merge the branch in the coming weeks. Please, let me know if I still need to wait for comments. Anders Johansson (3): Fix typo: delete-duplicates → delete-dups Fix bug in org-get-heading Rename remaining org-force-cycle-archived → org-cycle-force-archived Ihor Radchenko (32): Add org-fold-core: new folding engine Separate folding functions from org.el into new library: org-fold Separate cycling functions from org.el into new library: org-cycle Remove functions from org.el that are now moved elsewhere Disable native-comp in agenda org-macs: New function org-find-text-property-region org-at-heading-p: Accept optional argument org-string-width: Reimplement to work with new folding Rename old function call to use org-fold Implement link folding Implement overlay- and text-property-based versions of some functions org-fold: Handle indirect buffer visibility Fix subtle differences between overlays and invisible text properties Support extra org-fold optimisations for huge buffers Alias new org-fold functions to their old shorter names Obsolete old function names that are now in org-fold org-compat: Work around some third-party packages using outline-* functions Move `org-buffer-list' to org-macs.el Restore old visibility behaviour of org-refile Add org-fold-related tests org-manual: Update to new org-fold function names ORG-NEWS: Add list of changes Backport contributed commits Fix org-fold--hide-drawers--overlays org-string-width: Handle undefined behaviour in older Emacs org-string-width: Work around `window-pixel-width' bug in old Emacs org-fold-show-set-visibility: Fix edge case when folded region is at BOB org-fold-core: Fix fontification inside folded regions test-org/string-width: Add tests for strings with prefix properties org--string-from-props: Fix handling folds in Emacs <28 org-link-make-string: Throw error when both LINK and DESCRIPTION are empty test-ol/org-toggle-link-display: Fix compatibility with old Emacs doc/org-manual.org | 14 +- etc/ORG-NEWS | 104 ++ lisp/ob-core.el | 14 +- lisp/ob-lilypond.el | 4 +- lisp/ob-ref.el | 4 +- lisp/ol.el | 59 +- lisp/org-agenda.el | 50 +- lisp/org-archive.el | 12 +- lisp/org-capture.el | 7 +- lisp/org-clock.el | 126 +- lisp/org-colview.el | 10 +- lisp/org-compat.el | 189 ++- lisp/org-crypt.el | 8 +- lisp/org-cycle.el | 818 +++++++++++ lisp/org-element.el | 55 +- lisp/org-feed.el | 4 +- lisp/org-fold-core.el | 1503 +++++++++++++++++++ lisp/org-fold.el | 1132 +++++++++++++++ lisp/org-footnote.el | 6 +- lisp/org-goto.el | 6 +- lisp/org-id.el | 4 +- lisp/org-inlinetask.el | 26 +- lisp/org-keys.el | 26 +- lisp/org-lint.el | 3 +- lisp/org-list.el | 84 +- lisp/org-macs.el | 290 +++- lisp/org-mobile.el | 2 +- lisp/org-mouse.el | 4 +- lisp/org-refile.el | 3 +- lisp/org-src.el | 6 +- lisp/org-timer.el | 2 +- lisp/org.el | 2552 +++++++++++---------------------- lisp/ox-org.el | 2 +- lisp/ox.el | 4 +- testing/lisp/test-ob.el | 12 +- testing/lisp/test-ol.el | 24 + testing/lisp/test-org-list.el | 75 +- testing/lisp/test-org-macs.el | 6 +- testing/lisp/test-org.el | 258 +++- 39 files changed, 5475 insertions(+), 2033 deletions(-) create mode 100644 lisp/org-cycle.el create mode 100644 lisp/org-fold-core.el create mode 100644 lisp/org-fold.el -- 2.34.1 ^ permalink raw reply [flat|nested] 192+ messages in thread
* [PATCH 01/35] Add org-fold-core: new folding engine 2022-01-29 11:37 ` [PATCH 00/35] Merge org-fold feature branch Ihor Radchenko @ 2022-01-29 11:37 ` Ihor Radchenko 2022-01-29 11:37 ` [PATCH 02/35] Separate folding functions from org.el into new library: org-fold Ihor Radchenko ` (34 subsequent siblings) 35 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2022-01-29 11:37 UTC (permalink / raw) To: Bastien, Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, emacs-orgmode Cc: Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 155 bytes --] --- lisp/org-fold-core.el | 1490 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1490 insertions(+) create mode 100644 lisp/org-fold-core.el [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-Add-org-fold-core-new-folding-engine.patch --] [-- Type: text/x-patch; name="0001-Add-org-fold-core-new-folding-engine.patch", Size: 78656 bytes --] diff --git a/lisp/org-fold-core.el b/lisp/org-fold-core.el new file mode 100644 index 000000000..121c6b5c4 --- /dev/null +++ b/lisp/org-fold-core.el @@ -0,0 +1,1490 @@ +;;; org-fold-core.el --- Folding buffer text -*- lexical-binding: t; -*- +;; +;; Copyright (C) 2020-2020 Free Software Foundation, Inc. +;; +;; Author: Ihor Radchenko <yantar92 at gmail dot com> +;; Keywords: folding, invisible text +;; Homepage: https://orgmode.org +;; +;; This file is part of GNU Emacs. +;; +;; GNU Emacs is free software: you can redistribute it and/or modify +;; it under the terms of the GNU General Public License as published by +;; the Free Software Foundation, either version 3 of the License, or +;; (at your option) any later version. + +;; GNU Emacs is distributed in the hope that it will be useful, +;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;; GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public License +;; along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>. +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; +;;; Commentary: + +;; This file contains library to control temporary invisibility +;; (folding and unfolding) of text in buffers. + +;; The file implements the following functionality: +;; +;; - Folding/unfolding regions of text +;; - Searching and examining boundaries of folded text +;; - Interactive searching in folded text (via isearch) +;; - Handling edits in folded text +;; - Killing/yanking (copying/pasting) of the folded text +;; - Fontification of the folded text + +;; To setup folding in an arbitrary buffer, one must call +;; `org-fold-core-initialize', optionally providing the list of folding specs to be +;; used in the buffer. The specs can be added, removed, or +;; re-configured later. Read below for more details. + +;;; Folding/unfolding regions of text + +;; User can temporarily hide/reveal (fold/unfold) arbitrary regions or +;; text. The folds can be nested. + +;; Internally, nested folds are marked with different folding specs +;; Overlapping folds marked with the same folding spec are +;; automatically merged, while folds with different folding specs can +;; coexist and be folded/unfolded independently. + +;; When multiple folding specs are applied to the same region of text, +;; text visibility is decided according to the folding spec with +;; topmost priority. + +;; By default, we define two types of folding specs: +;; - 'org-fold-visible :: the folded text is not hidden +;; - 'org-fold-hidden :: the folded text is completely hidden +;; +;; The 'org-fold-visible spec has highest priority allowing parts of +;; text folded with 'org-fold-hidden to be shown unconditionally. + +;; Consider the following Org mode link: +;; [[file:/path/to/file/file.ext][description]] +;; Only the word "description" is normally visible in this link. +;; +;; The way this partial visibility is achieved is combining the two +;; folding specs. The whole link is folded using 'org-fold-hidden +;; folding spec, but the visible part is additionally folded using +;; 'org-fold-visible: +;; +;; <begin org-fold-hidden>[[file:/path/to/file/file.ext][<begin org-fold-visible>description<end org-fold-visible>]]<end org-fold-hidden> +;; +;; Because 'org-fold-visible has higher priority than +;; 'org-fold-hidden, it suppresses the 'org-fold-hidden effect and +;; thus reveals the description part of the link. + +;; Similar to 'org-fold-visible, display of any arbitrary folding spec +;; can be configured using folding spec properties. In particular, +;; `:visible' folding spec proprety controls whether the folded text +;; is visible or not. If the `:visible' folding spec property is nil, +;; folded text is hidden or displayed as a constant string (ellipsis) +;; according to the value of `:ellipsis' folding spec property. See +;; docstring of `org-fold-core--specs' for the description of all the available +;; folding spec properties. + +;; Folding spec properties of any valid folding spec can be changed +;; any time using `org-fold-core-set-folding-spec-property'. + +;; If necessary, one can add or remove folding specs using +;; `org-fold-core-add-folding-spec' and `org-fold-core-remove-folding-spec'. + +;; If a buffer initialised with `org-fold-core-initialize' is cloned into indirect +;; buffers, it's folding state is copied to that indirect buffer. +;; The folding states are independent. + +;; When working with indirect buffers that are handled by this +;; library, one has to keep in mind that folding state is preserved on +;; copy when using non-interactive functions. Moreover, the folding +;; states of all the indirect buffers will be copied together. +;; +;; Example of the implications: +;; Consider a base buffer and indirect buffer with the following state: +;; ----- base buffer -------- +;; * Heading<begin fold> +;; Some text folded in the base buffer, but unfolded in the indirect buffer<end fold> +;; * Other heading +;; Heading unfolded in both the buffers. +;; --------------------------- +;; ------ indirect buffer ---- +;; * Heading +;; Some text folded in the base buffer, but unfolded in the indirect buffer +;; * Other heading +;; Heading unfolded in both the buffers. +;; ---------------------------- +;; If some Elisp code copies the whole "Heading" from the indirect +;; buffer with `buffer-substring' or match data and inserts it into +;; the base buffer, the inserted heading will be folded since the +;; internal setting for the folding state is shared between the base +;; and indirect buffers. It's just that the indirect buffer ignores +;; the base buffer folding settings. However, as soon as the text is +;; copied back to the base buffer, the folding state will become +;; respected again. + +;; If the described situation is undesired, Elisp code can use +;; `filter-buffer-substring' instead of `buffer-substring'. All the +;; folding states that do not belong to the currently active buffer +;; will be cleared in the copied text then. See +;; `org-fold-core--buffer-substring-filter' for more details. + +;; Because of details of implementation of the folding, it is also not +;; recommended to set text visibility in buffer directly by setting +;; `invisible' text property to anything other than t. While this +;; should usually work just fine, normal folding can be broken if one +;; sets `invisible' text property to a value not listed in +;; `buffer-invisibility-spec'. + +;;; Searching and examining boundaries of folded text + +;; It is possible to examine folding specs (there may be several) of +;; text at point or search for regions with the same folding spec. +;; See functions defined under ";;;; Searching and examining folded +;; text" below for details. + +;; All the folding specs can be specified by symbol representing their +;; name. However, this is not always convenient, especially if the +;; same spec can be used for fold different syntaxical structures. +;; Any folding spec can be additionally referenced by a symbol listed +;; in the spec's `:alias' folding spec property. For example, Org +;; mode's `org-fold-outline' folding spec can be referened as any +;; symbol from the following list: '(headline heading outline +;; inlinetask plain-list) The list is the value of the spec's `:alias' +;; property. + +;; Most of the functions defined below that require a folding spec +;; symbol as their argument, can also accept any symbol from the +;; `:alias' spec property to reference that folding spec. + +;; If one wants to search invisible text without using the provided +;; functions, it is important to keep in mind that 'invisible text +;; property may have multiple possible values (not just nil and +;; t). Hence, (next-single-char-property-change pos 'invisible) is not +;; guarantied to return the boundary of invisible/visible text. + +;;; Interactive searching inside folded text (via isearch) + +;; The library provides a way to control if the folded text can be +;; searchable using isearch. If the text is searchable, it is also +;; possible to control to unfold it temporarily during interactive +;; isearch session. + +;; The isearch behaviour is controlled on per-folding-spec basis by +;; setting `isearch-open' and `isearch-ignore' folding spec +;; properties. The the docstring of `org-fold-core--specs' for more details. + +;;; Handling edits inside folded text + +;; The visibility of the text inserted in front, rear, or in the +;; middle of a folded region is managed according to `:front-sticky' +;; and `:rear-sticky' folding properties of the corresponding folding +;; spec. The rules are the same with stickyness of text properties in +;; Elisp. + +;; If a text being inserted into the buffer is already folded and +;; invisible (before applying the stickyness rules), then it is +;; revealed. This behaviour can be changed by wrapping the insertion +;; code into `org-fold-core-ignore-modifications' macro. The macro will disable +;; all the processing related to buffer modifications. + +;; The library also provides a way to unfold the text after some +;; destructive changes breaking syntaxical structure of the buffer. +;; For example, Org mode automatically reveals folded drawers when the +;; drawer becomes syntaxically incorrect: +;; ------- before modification ------- +;; :DRAWER:<begin fold> +;; Some folded text inside drawer +;; :END:<end fold> +;; ----------------------------------- +;; If the ":END:" is edited, drawer syntax is not correct anymore and +;; the folded text is automatically unfolded. +;; ------- after modification -------- +;; :DRAWER: +;; Some folded text inside drawer +;; :EN: +;; ----------------------------------- + +;; The described automatic unfolding is controlled by `:fragile' +;; folding spec property. It's value can be a function checking if +;; changes inside (or around) the fold should drigger the unfold. By +;; default, only changes that directly involve folded regions will +;; trigger the check. In addition, `org-fold-core-extend-changed-region-functions' +;; can be set to extend the checks to all folded regions intersecting +;; with the region returned by the functions listed in the variable. + +;; The fragility checks can be bypassed if the code doing +;; modifications is wrapped into `org-fold-core-ignore-fragility-checks' macro. + +;;; Fontification of the folded text + +;; When working with huge buffers, `font-lock' may take a lot of time +;; to fontify all the buffer text during startup. This library +;; provides a way to delay fontification of initially folded text to +;; the time when the text is unfolded. The fontification is +;; controlled on per-folding-spec basis according to `:font-lock-skip' +;; folding spec property. + +;; This library replaces `font-lock-fontify-region-function' to implement the +;; delayed fontification. However, it only does so when +;; `font-lock-fontify-region-function' is not modified at the initialisation +;; time. If one needs to use both delayed fontification and custom +;; `font-lock-fontify-region-function', it is recommended to consult the +;; source code of `org-fold-core-fontify-region'. + +;;; Performance considerations + +;; This library is using text properties to hide text. Text +;; properties are much faster than overlays, that could be used for +;; the same purpose. Overlays are implemented with O(n) complexity in +;; Emacs (as for 2021-03-11). It means that any attempt to move +;; through hidden text in a file with many invisible overlays will +;; require time scaling with the number of folded regions (the problem +;; Overlays note of the manual warns about). For curious, historical +;; reasons why overlays are not efficient can be found in +;; https://www.jwz.org/doc/lemacs.html. + +;; Despite using text properties, the performance is still limited by +;; Emacs display engine. For example, >7Mb of text hidden within +;; visible part of a buffer may cause noticeable lags (which is still +;; orders of magnitude better in comparison with overlays). If the +;; performance issues become critical while using this library, it is +;; recommended to minimise the number of folding specs used in the +;; same buffer at a time. + +;; Alternatively, the library provides `org-fold-core--optimise-for-huge-buffers' +;; for additional speedup. This can be used as a file-local variable +;; in huge buffers. The variable can be set to enable various levels +;; of extra optimisation. See the docstring for detailed information. + +;; It is worth noting that when using `org-fold-core--optimise-for-huge-buffers' +;; with `grab-invisible' option, folded regions copied to other +;; buffers (including buffers that do not use this library) will +;; remain invisible. org-fold-core provides functions to work around +;; this issue: `org-fold-core-remove-optimisation' and `org-fold-core-update-optimisation', but +;; it is unlikely that a random external package will use them. + +;; Another possible bottleneck is the fragility check after the change +;; related to the folded text. The functions used in `:fragile' +;; folding properties must be optimised. Also, +;; `org-fold-core-ignore-fragility-checks' or even `org-fold-core-ignore-modifications' may be +;; used when appropriate in the performance-critical code. When +;; inserting text from within `org-fold-core-ignore-modifications' macro, it is +;; recommended to use `insert-and-inherit' instead of `insert' and +;; `insert-before-markers-and-inherit' instead of +;; `insert-before-markers' to avoid revealing inserted text in the +;; middle of a folded region. + +;; Performance of isearch is currently limited by Emacs isearch +;; implementation. For now, Emacs isearch only supports searching +;; through text hidden using overlays. This library handles isearch +;; by converting folds with matching text to overlays, which may +;; affect performance in case of large number of matches. In the +;; future, Emacs will hopefully accept the relevant patch allowing +;; isearch to work with text hidden via text properties, but the +;; performance hit has to be accepted meanwhile. + +;;; Code: + +(require 'org-macs) +(require 'org-compat) + +(declare-function isearch-filter-visible "isearch" (beg end)) + +;;; Customization + +(defcustom org-fold-core-style 'text-properties + "Internal implementation detail used to hide folded text. +Can be either `text-properties' or `overlays'. +The former is faster on large files, while the latter is generally +less error-prone." + :group 'org + :package-version '(Org . "9.6") + :type '(choice + (const :tag "Overlays" 'overlays) + (const :tag "Text properties" 'text-properties))) + +(defcustom org-fold-core-first-unfold-functions nil + "Functions executed after first unfolding during fontification. +Each function is exectured with two arguments: begin and end points of +the unfolded region." + :group 'org + :package-version '(Org . "9.6") + :type 'hook) + +(defvar-local org-fold-core-isearch-open-function #'org-fold-core--isearch-reveal + "Function used to reveal hidden text found by isearch. +The function is called with a single argument - point where text is to +be revealed.") + +(defvar-local org-fold-core--optimise-for-huge-buffers nil + "Non-nil turns on extra speedup on huge buffers (Mbs of folded text). + +This setting is risky and may cause various artefacts and degraded +functionality, especially when using external packages. It is +recommended to enable it on per-buffer basis as file-local variable. + +When set to non-nil, must be a list containing one or multiple the +following symbols: + +- `grab-invisible': Use `invisible' text property to hide text. This + will reduce the load on Emacs display engine and one may use it if + moving point across folded regions becomes slow. However, as a side + effect, some external packages extracting i.e. headlings from folded + parts of buffer may keep the text invisible. + +- `ignore-fragility-checks': Do not try to detect when user edits + break structure of the folded elements. This will speed up + modifying the folded regions at the cost that some higher-level + functions relying on this package might not be able to unfold the + edited text. For example, removed leading stars from a folded + headline in Org mode will break visibility cycling since Org mode + will not be avare that the following folded text belonged to + headline. + +- `ignore-modification-checks': Do not try to detect insertions in the + middle of the folded regions. This will speed up non-interactive + edits of the folded regions. However, text inserted in the middle + of the folded regions may become visible for some external packages + inserting text using `insert' instead of `insert-and-inherit' (the + latter is rarely used in practice). + +- `ignore-indirect': Do not decouple folding state in the indirect + buffers. This can speed up Emacs display engine (and thus motion of + point), especially when large number of indirect buffers is being + used. + +- `merge-folds': Do not distinguish between different types of folding + specs. This is the most aggressive optimisation with unforseen and + potentially drastic effects.") +(put 'org-fold-core--optimise-for-huge-buffers 'safe-local-variable 'listp) + +;;; Core functionality + +;;;; Folding specs + +(defvar-local org-fold-core--specs '((org-fold-visible + (:visible . t) + (:alias . (visible))) + (org-fold-hidden + (:ellipsis . "...") + (:isearch-open . t) + (:alias . (hidden)))) + "Folding specs defined in current buffer. + +Each spec is a list (SPEC-SYMBOL SPEC-PROPERTIES). +SPEC-SYMBOL is the symbol respresenting the folding spec. +SPEC-PROPERTIES is an alist defining folding spec properties. + +If a text region is folded using multiple specs, only the folding spec +listed earlier is used. + +The following properties are known: +- :ellipsis :: must be nil or string to show when text is folded + using this spec. +- :global :: non-nil means that folding state will be preserved + when copying folded text between buffers. +- :isearch-ignore :: non-nil means that folded text is not searchable + using isearch. +- :isearch-open :: non-nil means that isearch can reveal text hidden + using this spec. This property does nothing + when 'isearch-ignore property is non-nil. +- :front-sticky :: non-nil means that text prepended to the folded text + is automatically folded. +- :rear-sticky :: non-nil means that text appended to the folded text + is folded. +- :visible :: non-nil means that folding spec visibility is not + managed. Instead, visibility settings in + `buffer-invisibility-spec' will be used as is. + Note that changing this property from nil to t may + clear the setting in `buffer-invisibility-spec'. +- :alias :: a list of aliases for the SPEC-SYMBOL. +- :font-lock-skip :: Suppress font-locking in folded text. +- :fragile :: Must be a function accepting two arguments. + Non-nil means that changes in region may cause + the region to be revealed. The region is + revealed after changes if the function returns + non-nil. + The function called after changes are made with + two arguments: cons (beg . end) representing the + folded region and spec symbol.") +(defvar-local org-fold-core--spec-symbols nil + "Alist holding buffer spec symbols and aliases. + +This variable is defined to reduce load on Emacs garbage collector +reducing the number of transiently allocated variables.") +(defvar-local org-fold-core--spec-list nil + "List holding buffer spec symbols, but not aliases. + +This variable is defined to reduce load on Emacs garbage collector +reducing the number of transiently allocated variables.") + +(defvar-local org-fold-core-extend-changed-region-functions nil + "Special hook run just before handling changes in buffer. + +This is used to account changes outside folded regions that still +affect the folded region visibility. For example, removing all stars +at the beginning of a folded Org mode heading should trigger the +folded text to be revealed. Each function is called with two +arguments: beginning and the end of the changed region.") + +;;; Utility functions + +(defsubst org-fold-core-folding-spec-list (&optional buffer) + "Return list of all the folding spec symbols in BUFFER." + (or (buffer-local-value 'org-fold-core--spec-list (or buffer (current-buffer))) + (with-current-buffer (or buffer (current-buffer)) + (setq org-fold-core--spec-list (mapcar #'car org-fold-core--specs))))) + +(defun org-fold-core-get-folding-spec-from-alias (spec-or-alias) + "Return the folding spec symbol for SPEC-OR-ALIAS. +Return nil when there is no matching folding spec." + (when spec-or-alias + (unless org-fold-core--spec-symbols + (dolist (spec (org-fold-core-folding-spec-list)) + (push (cons spec spec) org-fold-core--spec-symbols) + (dolist (alias (assq :alias (assq spec org-fold-core--specs))) + (push (cons alias spec) org-fold-core--spec-symbols)))) + (alist-get spec-or-alias org-fold-core--spec-symbols))) + +(defsubst org-fold-core-folding-spec-p (spec-or-alias) + "Check if SPEC-OR-ALIAS is a registered folding spec." + (org-fold-core-get-folding-spec-from-alias spec-or-alias)) + +(defsubst org-fold-core--check-spec (spec-or-alias) + "Throw an error if SPEC-OR-ALIAS is not in `org-fold-core--spec-priority-list'." + (unless (org-fold-core-folding-spec-p spec-or-alias) + (error "%s is not a valid folding spec" spec-or-alias))) + +(defsubst org-fold-core-get-folding-spec-property (spec-or-alias property) + "Get PROPERTY of a folding SPEC-OR-ALIAS. +Possible properties can be found in `org-fold-core--specs' docstring." + (org-fold-core--check-spec spec-or-alias) + (if (and (memql 'ignore-indirect org-fold-core--optimise-for-huge-buffers) + (eq property :global)) + t + (if (and (memql 'merge-folds org-fold-core--optimise-for-huge-buffers) + (eq property :visible)) + nil + (cdr (assq property (assq (org-fold-core-get-folding-spec-from-alias spec-or-alias) org-fold-core--specs)))))) + +(defconst org-fold-core--spec-property-prefix "org-fold--spec-" + "Prefix used to create property symbol.") + +(defsubst org-fold-core-get-folding-property-symbol (spec &optional buffer global) + "Get folding text property using to store SPEC in current buffer or BUFFER. +If GLOBAL is non-nil, do not make the property unique in the BUFFER." + (if (memql 'merge-folds org-fold-core--optimise-for-huge-buffers) + (intern (format "%s-global" org-fold-core--spec-property-prefix)) + (intern (format (concat org-fold-core--spec-property-prefix "%s-%S") + (symbol-name spec) + ;; (sxhash buf) appears to be not constant over time. + ;; Using buffer-name is safe, since the only place where + ;; buffer-local text property actually matters is an indirect + ;; buffer, where the name cannot be same anyway. + (if global 'global + (sxhash (buffer-name (or buffer (current-buffer))))))))) + +(defsubst org-fold-core-get-folding-spec-from-folding-prop (folding-prop) + "Return folding spec symbol used for folding property with name FOLDING-PROP." + (catch :exit + (dolist (spec (org-fold-core-folding-spec-list)) + ;; We know that folding properties have + ;; folding spec in their name. + (when (string-match-p (symbol-name spec) + (symbol-name folding-prop)) + (throw :exit spec))))) + +(defvar org-fold-core--property-symbol-cache (make-hash-table :test 'equal) + "Saved values of folding properties for (buffer . spec) conses.") +(defvar-local org-fold-core--indirect-buffers nil + "List of indirect buffers created from current buffer. + +The first element of the list is always the current buffer. + +This variable is needed to work around Emacs bug#46982, while Emacs +does not provide a way `after-change-functions' in any other buffer +than the buffer where the change was actually made.") + +(defmacro org-fold-core-cycle-over-indirect-buffers (&rest body) + "Execute BODY in current buffer and all its indirect buffers. + +Also, make sure that folding properties from killed buffers are not +hanging around." + (declare (debug (form body)) (indent 1)) + `(let (buffers dead-properties) + (if (and (not (buffer-base-buffer)) + (not (eq (current-buffer) (car org-fold-core--indirect-buffers)))) + ;; We are in base buffer with `org-fold-core--indirect-buffers' value from + ;; different buffer. This can happen, for example, when + ;; org-capture copies local variables into *Capture* buffer. + (setq buffers (list (current-buffer))) + (dolist (buf (cons (or (buffer-base-buffer) (current-buffer)) + (buffer-local-value 'org-fold-core--indirect-buffers (or (buffer-base-buffer) (current-buffer))))) + (if (buffer-live-p buf) + (push buf buffers) + (dolist (spec (org-fold-core-folding-spec-list)) + (when (and (not (org-fold-core-get-folding-spec-property spec :global)) + (gethash (cons buf spec) org-fold-core--property-symbol-cache)) + ;; Make sure that dead-properties variable can be passed + ;; as argument to `remove-text-properties'. + (push t dead-properties) + (push (gethash (cons buf spec) org-fold-core--property-symbol-cache) + dead-properties)))))) + (dolist (buf buffers) + (with-current-buffer buf + (with-silent-modifications + (save-restriction + (widen) + (remove-text-properties + (point-min) (point-max) + dead-properties))) + ,@body)))) + +;; This is the core function used to fold text in buffers. We use +;; text properties to hide folded text, however 'invisible property is +;; not directly used (unless risky `org-fold-core--optimise-for-huge-buffers' is +;; enabled). Instead, we define unique text property (folding +;; property) for every possible folding spec and add the resulting +;; text properties into `char-property-alias-alist', so that +;; 'invisible text property is automatically defined if any of the +;; folding properties is non-nil. This approach lets us maintain +;; multiple folds for the same text region - poor man's overlays (but +;; much faster). Additionally, folding properties are ensured to be +;; unique for different buffers (especially for indirect +;; buffers). This is done to allow different folding states in +;; indirect buffers. +(defun org-fold-core--property-symbol-get-create (spec &optional buffer return-only) + "Return a unique symbol suitable as folding text property. +Return value is unique for folding SPEC in BUFFER. +If the buffer already have buffer-local setup in `char-property-alias-alist' +and the setup appears to be created for different buffer, +copy the old invisibility state into new buffer-local text properties, +unless RETURN-ONLY is non-nil." + (if (eq org-fold-core-style 'overlays) + (org-fold-core-get-folding-property-symbol spec nil 'global) + (let* ((buf (or buffer (current-buffer)))) + ;; Create unique property symbol for SPEC in BUFFER + (let ((local-prop (or (gethash (cons buf spec) org-fold-core--property-symbol-cache) + (puthash (cons buf spec) + (org-fold-core-get-folding-property-symbol + spec buf + (org-fold-core-get-folding-spec-property spec :global)) + org-fold-core--property-symbol-cache)))) + (prog1 + local-prop + (unless return-only + (with-current-buffer buf + ;; Update folding properties carried over from other + ;; buffer (implying that current buffer is indirect + ;; buffer). Normally, `char-property-alias-alist' in new + ;; indirect buffer is a copy of the same variable from + ;; the base buffer. Then, `char-property-alias-alist' + ;; would contain folding properties, which are not + ;; matching the generated `local-prop'. + (unless (member local-prop (cdr (assq 'invisible char-property-alias-alist))) + ;; Add current buffer to the list of indirect buffers in the base buffer. + (when (buffer-base-buffer) + (with-current-buffer (buffer-base-buffer) + (setq-local org-fold-core--indirect-buffers + (let (bufs) + (org-fold-core-cycle-over-indirect-buffers + (push (current-buffer) bufs)) + (push buf bufs) + (delete-dups bufs))))) + ;; Copy all the old folding properties to preserve the folding state + (with-silent-modifications + (dolist (old-prop (cdr (assq 'invisible char-property-alias-alist))) + (org-with-wide-buffer + (let* ((pos (point-min)) + (spec (org-fold-core-get-folding-spec-from-folding-prop old-prop)) + ;; Generate new buffer-unique folding property + (new-prop (when spec (org-fold-core--property-symbol-get-create spec nil 'return-only)))) + ;; Copy the visibility state for `spec' from `old-prop' to `new-prop' + (unless (eq old-prop new-prop) + (while (< pos (point-max)) + (let ((val (get-text-property pos old-prop)) + (next (next-single-char-property-change pos old-prop))) + (when val + (put-text-property pos next new-prop val)) + (setq pos next))))))) + ;; Update `char-property-alias-alist' with folding + ;; properties unique for the current buffer. + (setq-local char-property-alias-alist + (cons (cons 'invisible + (mapcar (lambda (spec) + (org-fold-core--property-symbol-get-create spec nil 'return-only)) + (org-fold-core-folding-spec-list))) + (remove (assq 'invisible char-property-alias-alist) + char-property-alias-alist))) + ;; Set folding property stickyness according to + ;; their `:font-sticky' and `:rear-sticky' + ;; parameters. + (let (full-prop-list) + (org-fold-core-cycle-over-indirect-buffers + (setq full-prop-list + (append full-prop-list + (delq nil + (mapcar (lambda (spec) + (cond + ((org-fold-core-get-folding-spec-property spec :front-sticky) + (cons (org-fold-core--property-symbol-get-create spec nil 'return-only) + nil)) + ((org-fold-core-get-folding-spec-property spec :rear-sticky) + nil) + (t + (cons (org-fold-core--property-symbol-get-create spec nil 'return-only) + t)))) + (org-fold-core-folding-spec-list)))))) + (org-fold-core-cycle-over-indirect-buffers + (setq-local text-property-default-nonsticky + (delete-dups (append + text-property-default-nonsticky + full-prop-list)))))))))))))) + +(defun org-fold-core-decouple-indirect-buffer-folds () + "Copy and decouple folding state in a newly created indirect buffer. +This function is mostly indented to be used in `clone-indirect-buffer-hook'." + (when (and (buffer-base-buffer) + (eq org-fold-core-style 'text-properties)) + (org-fold-core--property-symbol-get-create (car (org-fold-core-folding-spec-list))))) + +;;; API + +;;;; Modifying folding specs + +(defun org-fold-core-set-folding-spec-property (spec property value &optional force) + "Set PROPERTY of a folding SPEC to VALUE. +Possible properties and values can be found in `org-fold-core--specs' docstring. +Do not check previous value when FORCE is non-nil." + (pcase property + (:ellipsis + (unless (and (not force) (equal value (org-fold-core-get-folding-spec-property spec :ellipsis))) + (remove-from-invisibility-spec (cons spec (org-fold-core-get-folding-spec-property spec :ellipsis))) + (unless (org-fold-core-get-folding-spec-property spec :visible) + (add-to-invisibility-spec (cons spec value))))) + (:visible + (unless (or (memql 'merge-folds org-fold-core--optimise-for-huge-buffers) + (and (not force) (equal value (org-fold-core-get-folding-spec-property spec :visible)))) + (if value + (remove-from-invisibility-spec (cons spec (org-fold-core-get-folding-spec-property spec :ellipsis))) + (add-to-invisibility-spec (cons spec (org-fold-core-get-folding-spec-property spec :ellipsis)))))) + (:alias + ;; Clear symbol cache. + (setq org-fold-core--spec-symbols nil)) + (:isearch-open nil) + (:isearch-ignore nil) + (:front-sticky nil) + (:rear-sticky nil) + (_ nil)) + (setf (cdr (assq property (assq spec org-fold-core--specs))) value)) + +(defun org-fold-core-add-folding-spec (spec &optional properties buffer append) + "Add a new folding SPEC with PROPERTIES in BUFFER. + +SPEC must be a symbol. BUFFER can be a buffer to set SPEC in or nil to +set SPEC in current buffer. + +By default, the added SPEC will have highest priority among the +previously defined specs. When optional APPEND argument is non-nil, +SPEC will have the lowest priority instead. If SPEC was already +defined earlier, it will be redefined according to provided optional +arguments. +` +The folding spec properties will be set to PROPERTIES (see +`org-fold-core--specs' for details)." + (when (eq spec 'all) (error "Cannot use reserved folding spec symbol 'all")) + (with-current-buffer (or buffer (current-buffer)) + ;; Clear the cache. + (setq org-fold-core--spec-list nil + org-fold-core--spec-symbols nil) + (let* ((full-properties (mapcar (lambda (prop) (cons prop (cdr (assq prop properties)))) + '( :visible :ellipsis :isearch-ignore + :global :isearch-open :front-sticky + :rear-sticky :fragile :alias + :font-lock-skip))) + (full-spec (cons spec full-properties))) + (add-to-list 'org-fold-core--specs full-spec append) + (mapc (lambda (prop-cons) (org-fold-core-set-folding-spec-property spec (car prop-cons) (cdr prop-cons) 'force)) full-properties) + ;; Update buffer inivisibility specs. + (org-fold-core--property-symbol-get-create spec)))) + +(defun org-fold-core-remove-folding-spec (spec &optional buffer) + "Remove a folding SPEC in BUFFER. + +SPEC must be a symbol. + +BUFFER can be a buffer to remove SPEC in, nil to remove SPEC in current +buffer, or 'all to remove SPEC in all open `org-mode' buffers and all +future org buffers." + (org-fold-core--check-spec spec) + (when (eq buffer 'all) + (setq-default org-fold-core--specs (delete (cdr (assq spec org-fold-core--specs)) org-fold-core--specs)) + (mapc (lambda (buf) + (org-fold-core-remove-folding-spec spec buf)) + (buffer-list))) + (let ((buffer (or buffer (current-buffer)))) + (with-current-buffer buffer + ;; Clear the cache. + (setq org-fold-core--spec-list nil + org-fold-core--spec-symbols nil) + (org-fold-core-set-folding-spec-property spec :visible t) + (setq org-fold-core--specs (delete (cdr (assq spec org-fold-core--specs)) org-fold-core--specs))))) + +(defun org-fold-core-initialize (&optional specs) + "Setup folding in current buffer using SPECS as value of `org-fold-core--specs'." + ;; Preserve the priorities. + (when specs (setq specs (nreverse specs))) + (unless specs (setq specs org-fold-core--specs)) + (setq org-fold-core--specs nil + org-fold-core--spec-list nil + org-fold-core--spec-symbols nil) + (dolist (spec specs) + (org-fold-core-add-folding-spec (car spec) (cdr spec))) + (add-hook 'after-change-functions 'org-fold-core--fix-folded-region nil 'local) + (add-hook 'clone-indirect-buffer-hook #'org-fold-core-decouple-indirect-buffer-folds nil 'local) + ;; Optimise buffer fontification to not fontify folded text. + (when (eq font-lock-fontify-region-function #'font-lock-default-fontify-region) + (setq-local font-lock-fontify-region-function 'org-fold-core-fontify-region)) + ;; Setup killing text + (setq-local filter-buffer-substring-function #'org-fold-core--buffer-substring-filter) + (if (and (boundp 'isearch-opened-regions) + (eq org-fold-core-style 'text-properties)) + ;; Use new implementation of isearch allowing to search inside text + ;; hidden via text properties. + (org-fold-core--isearch-setup 'text-properties) + (org-fold-core--isearch-setup 'overlays))) + +;;;; Searching and examining folded text + +(defsubst org-fold-core-folded-p (&optional pos spec-or-alias) + "Non-nil if the character after POS is folded. +If POS is nil, use `point' instead. +If SPEC-OR-ALIAS is a folding spec, only check the given folding spec." + (org-fold-core-get-folding-spec spec-or-alias pos)) + +(defun org-fold-core-region-folded-p (beg end &optional spec-or-alias) + "Non-nil if the region between BEG and END is folded. +If SPEC-OR-ALIAS is a folding spec, only check the given folding spec." + (org-with-point-at beg + (catch :visible + (while (< (point) end) + (unless (org-fold-core-get-folding-spec spec-or-alias) (throw :visible nil)) + (goto-char (org-fold-core-next-folding-state-change spec-or-alias nil end))) + t))) + +(defun org-fold-core-get-folding-spec (&optional spec-or-alias pom) + "Get folding state at `point' or POM. +Return nil if there is no folding at point or POM. +If SPEC-OR-ALIAS is nil, return a folding spec with highest priority +among present at `point' or POM. +If SPEC-OR-ALIAS is 'all, return the list of all present folding +specs. +If SPEC-OR-ALIAS is a valid folding spec or a spec alias, return the +corresponding folding spec (if the text is folded using that spec)." + (let ((spec (if (eq spec-or-alias 'all) + 'all + (org-fold-core-get-folding-spec-from-alias spec-or-alias)))) + (when (and spec (not (eq spec 'all))) (org-fold-core--check-spec spec)) + (org-with-point-at pom + (cond + ((eq spec 'all) + (let ((result)) + (dolist (spec (org-fold-core-folding-spec-list)) + (let ((val (get-char-property (point) (org-fold-core--property-symbol-get-create spec nil t)))) + (when val (push val result)))) + (reverse result))) + ((null spec) + (let ((result (get-char-property (point) 'invisible))) + (when (org-fold-core-folding-spec-p result) result))) + (t (get-char-property (point) (org-fold-core--property-symbol-get-create spec nil t))))))) + +(defun org-fold-core-get-folding-specs-in-region (beg end) + "Get all folding specs in region from BEG to END." + (let ((pos beg) + all-specs) + (while (< pos end) + (setq all-specs (append all-specs (org-fold-core-get-folding-spec nil pos))) + (setq pos (org-fold-core-next-folding-state-change nil pos end))) + (unless (listp all-specs) (setq all-specs (list all-specs))) + (delete-dups all-specs))) + +(defun org-fold-core-get-region-at-point (&optional spec-or-alias pom) + "Return region folded using SPEC-OR-ALIAS at POM. +If SPEC is nil, return the largest possible folded region. +The return value is a cons of beginning and the end of the region. +Return nil when no fold is present at point of POM." + (let ((spec (org-fold-core-get-folding-spec-from-alias spec-or-alias))) + (org-with-point-at (or pom (point)) + (if spec + (if (eq org-fold-core-style 'text-properties) + (org-find-text-property-region (point) (org-fold-core--property-symbol-get-create spec nil t)) + (let ((ov (cdr (get-char-property-and-overlay (point) (org-fold-core--property-symbol-get-create spec nil t))))) + (when ov (cons (overlay-start ov) (overlay-end ov))))) + (let ((region (cons (point) (point)))) + (dolist (spec (org-fold-core-get-folding-spec 'all)) + (let ((local-region (org-fold-core-get-region-at-point spec))) + (when (< (car local-region) (car region)) + (setcar region (car local-region))) + (when (> (cdr local-region) (cdr region)) + (setcdr region (cdr local-region))))) + (unless (eq (car region) (cdr region)) region)))))) + +(defun org-fold-core-next-visibility-change (&optional pos limit ignore-hidden-p previous-p) + "Return next point from POS up to LIMIT where text becomes visible/invisible. +By default, text hidden by any means (i.e. not only by folding, but +also via fontification) will be considered. +If IGNORE-HIDDEN-P is non-nil, consider only folded text. +If PREVIOUS-P is non-nil, search backwards." + (let* ((pos (or pos (point))) + (invisible-p (if ignore-hidden-p + #'org-fold-core-folded-p + #'invisible-p)) + (invisible-initially? (funcall invisible-p pos)) + (limit (or limit (if previous-p + (point-min) + (point-max)))) + (cmp (if previous-p #'> #'<)) + (next-change (if previous-p + (if ignore-hidden-p + (lambda (p) (org-fold-core-previous-folding-state-change (org-fold-core-get-folding-spec nil p) p limit)) + (lambda (p) (max limit (1- (previous-single-char-property-change p 'invisible nil limit))))) + (if ignore-hidden-p + (lambda (p) (org-fold-core-next-folding-state-change (org-fold-core-get-folding-spec nil p) p limit)) + (lambda (p) (next-single-char-property-change p 'invisible nil limit))))) + (next pos)) + (while (and (funcall cmp next limit) + (not (org-xor invisible-initially? (funcall invisible-p next)))) + (setq next (funcall next-change next))) + next)) + +(defun org-fold-core-previous-visibility-change (&optional pos limit ignore-hidden-p) + "Call `org-fold-core-next-visibility-change' searching backwards." + (org-fold-core-next-visibility-change pos limit ignore-hidden-p 'previous)) + +(defun org-fold-core-next-folding-state-change (&optional spec-or-alias pos limit previous-p) + "Return point after POS where folding state changes up to LIMIT. +If SPEC-OR-ALIAS is nil, return next point where _any_ single folding +spec changes. +For example, (org-fold-core-next-folding-state-change nil) with point +somewhere in the below structure will return the nearest <...> point. + +* Headline <begin outline fold> +:PROPERTIES:<begin drawer fold> +:ID: test +:END:<end drawer fold> + +Fusce suscipit, wisi nec facilisis facilisis, est dui fermentum leo, +quis tempor ligula erat quis odio. + +** Another headline +:DRAWER:<begin drawer fold> +:END:<end drawer fold> +** Yet another headline +<end of outline fold> + +If SPEC-OR-ALIAS is a folding spec symbol, only consider that folding +spec. + +If SPEC-OR-ALIAS is a list, only consider changes of folding specs +from the list. + +Search backwards when PREVIOUS-P is non-nil." + (when (and spec-or-alias (symbolp spec-or-alias)) + (setq spec-or-alias (list spec-or-alias))) + (when spec-or-alias + (setq spec-or-alias + (mapcar (lambda (spec-or-alias) + (or (org-fold-core-get-folding-spec-from-alias spec-or-alias) + spec-or-alias)) + spec-or-alias)) + (mapc #'org-fold-core--check-spec spec-or-alias)) + (unless spec-or-alias + (setq spec-or-alias (org-fold-core-folding-spec-list))) + (setq pos (or pos (point))) + (apply (if previous-p + #'max + #'min) + (mapcar (if previous-p + (lambda (prop) (max (or limit (point-min)) (previous-single-property-change pos prop nil (or limit (point-min))))) + (lambda (prop) (next-single-property-change pos prop nil (or limit (point-max))))) + (mapcar (lambda (el) (org-fold-core--property-symbol-get-create el nil t)) + spec-or-alias)))) + +(defun org-fold-core-previous-folding-state-change (&optional spec-or-alias pos limit) + "Call `org-fold-core-next-folding-state-change' searching backwards." + (org-fold-core-next-folding-state-change spec-or-alias pos limit 'previous)) + +(defun org-fold-core-search-forward (spec-or-alias &optional limit) + "Search next region folded via folding SPEC-OR-ALIAS up to LIMIT. +Move point right after the end of the region, to LIMIT, or +`point-max'. The `match-data' will contain the region." + (let ((spec (org-fold-core-get-folding-spec-from-alias spec-or-alias))) + (let ((prop-symbol (org-fold-core--property-symbol-get-create spec nil t))) + (goto-char (or (next-single-char-property-change (point) prop-symbol nil limit) limit (point-max))) + (when (and (< (point) (or limit (point-max))) + (not (org-fold-core-get-folding-spec spec))) + (goto-char (next-single-char-property-change (point) prop-symbol nil limit))) + (when (org-fold-core-get-folding-spec spec) + (let ((region (org-fold-core-get-region-at-point spec))) + (when (< (cdr region) (or limit (point-max))) + (goto-char (1+ (cdr region))) + (set-match-data (list (set-marker (make-marker) (car region) (current-buffer)) + (set-marker (make-marker) (cdr region) (current-buffer)))))))))) + +;;;; Changing visibility + +;;;;; Region visibility + +(defvar org-fold-core--fontifying nil + "Flag used to avoid font-lock recursion.") + +;; This is the core function performing actual folding/unfolding. The +;; folding state is stored in text property (folding property) +;; returned by `org-fold-core--property-symbol-get-create'. The value of the +;; folding property is folding spec symbol. +(defun org-fold-core-region (from to flag &optional spec-or-alias) + "Hide or show lines from FROM to TO, according to FLAG. +SPEC-OR-ALIAS is the folding spec or foldable element, as a symbol. +If SPEC-OR-ALIAS is omitted and FLAG is nil, unfold everything in the region." + (let ((spec (org-fold-core-get-folding-spec-from-alias spec-or-alias))) + (when spec (org-fold-core--check-spec spec)) + (with-silent-modifications + (org-with-wide-buffer + (when (eq org-fold-core-style 'overlays) (remove-overlays from to 'invisible spec)) + (if flag + (if (not spec) + (error "Calling `org-fold-core-region' with missing SPEC") + (if (eq org-fold-core-style 'overlays) + ;; Use `front-advance' since text right before to the beginning of + ;; the overlay belongs to the visible line than to the contents. + (let ((o (make-overlay from to nil + (org-fold-core-get-folding-spec-property spec :front-sticky) + (org-fold-core-get-folding-spec-property spec :rear-sticky)))) + (overlay-put o 'evaporate t) + (overlay-put o (org-fold-core--property-symbol-get-create spec) spec) + (overlay-put o 'invisible spec) + (overlay-put o 'isearch-open-invisible #'org-fold-core--isearch-show) + (overlay-put o 'isearch-open-invisible-temporary #'org-fold-core--isearch-show-temporary)) + (put-text-property from to (org-fold-core--property-symbol-get-create spec) spec) + (put-text-property from to 'isearch-open-invisible #'org-fold-core--isearch-show) + (put-text-property from to 'isearch-open-invisible-temporary #'org-fold-core--isearch-show-temporary) + (when (memql 'grab-invisible org-fold-core--optimise-for-huge-buffers) + ;; If the SPEC has highest priority, assign it directly + ;; to 'invisible property as well. This is done to speed + ;; up Emacs redisplay on huge (Mbs) folded regions where + ;; we don't even want Emacs to spend time cycling over + ;; `char-property-alias-alist'. + (when (eq spec (caar org-fold-core--specs)) (put-text-property from to 'invisible spec))))) + (if (not spec) + (mapc (lambda (spec) (org-fold-core-region from to nil spec)) (org-fold-core-folding-spec-list)) + (when (and (memql 'grab-invisible org-fold-core--optimise-for-huge-buffers) + (eq org-fold-core-style 'text-properties)) + (when (eq spec (caar org-fold-core--specs)) + (let ((pos from)) + (while (< pos to) + (if (eq spec (get-text-property pos 'invisible)) + (let ((next (org-fold-core-next-folding-state-change spec pos to))) + (remove-text-properties pos next '(invisible t)) + (setq pos next)) + (setq pos (next-single-char-property-change pos 'invisible nil to))))))) + (when (eq org-fold-core-style 'text-properties) + (remove-text-properties from to (list (org-fold-core--property-symbol-get-create spec) nil))) + ;; Fontify unfolded text. + (unless (or (not font-lock-mode) + org-fold-core--fontifying + (not (org-fold-core-get-folding-spec-property spec :font-lock-skip)) + (not (text-property-not-all from to 'org-fold-core-fontified t))) + (let ((org-fold-core--fontifying t)) + (if jit-lock-mode + (jit-lock-refontify from to) + (save-match-data (font-lock-fontify-region from to))))))))))) + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org-fold-core--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +(defvar-local org-fold-core--isearch-local-regions (make-hash-table :test 'equal) + "Hash table storing temporarily shown folds from isearch matches.") + +(defun org-fold-core--isearch-setup (type) + "Initialize isearch in org buffer. +TYPE can be either `text-properties' or `overlays'." + (pcase type + (`text-properties + (setq-local search-invisible 'open-all) + (add-hook 'isearch-mode-end-hook #'org-fold-core--clear-isearch-state nil 'local) + (add-hook 'isearch-mode-hook #'org-fold-core--clear-isearch-state nil 'local) + (setq-local isearch-filter-predicate #'org-fold-core--isearch-filter-predicate-text-properties)) + (`overlays + (when (eq org-fold-core-style 'text-properties) + (setq-local isearch-filter-predicate #'org-fold-core--isearch-filter-predicate-overlays) + (add-hook 'isearch-mode-end-hook #'org-fold-core--clear-isearch-overlays nil 'local))) + (_ (error "%s: Unknown type of setup for `org-fold-core--isearch-setup'" type)))) + +(defun org-fold-core--isearch-reveal (pos) + "Default function used to reveal hidden text at POS for isearch." + (let ((region (org-fold-core-get-region-at-point pos))) + (org-fold-core-region (car region) (cdr region) nil))) + +(defun org-fold-core--isearch-filter-predicate-text-properties (beg end) + "Make sure that folded text is searchable when user whant so. +This function is intended to be used as `isearch-filter-predicate'." + (and + ;; Check folding specs that cannot be searched + (not (memq nil (mapcar (lambda (spec) (not (org-fold-core-get-folding-spec-property spec :isearch-ignore))) + (org-fold-core-get-folding-specs-in-region beg end)))) + ;; Check 'invisible properties that are not folding specs. + (or (eq search-invisible t) ; User wants to search anyway, allow it. + (let ((pos beg) + unknown-invisible-property) + (while (and (< pos end) + (not unknown-invisible-property)) + (when (and (get-text-property pos 'invisible) + (not (org-fold-core-folding-spec-p (get-text-property pos 'invisible)))) + (setq unknown-invisible-property t)) + (setq pos (next-single-char-property-change pos 'invisible))) + (not unknown-invisible-property))) + (or (and (eq search-invisible t) + ;; FIXME: this opens regions permanenly for now. + ;; I also tried to force search-invisible 'open-all around + ;; `isearch-range-invisible', but that somehow causes + ;; infinite loop in `isearch-lazy-highlight'. + (prog1 t + ;; We still need to reveal the folded location + (org-fold-core--isearch-show-temporary (cons beg end) nil))) + (not (isearch-range-invisible beg end))))) + +(defun org-fold-core--clear-isearch-state () + "Clear `org-fold-core--isearch-local-regions'." + (clrhash org-fold-core--isearch-local-regions)) + +(defun org-fold-core--isearch-show (region) + "Reveal text in REGION found by isearch." + (org-with-point-at (car region) + (while (< (point) (cdr region)) + (funcall org-fold-core-isearch-open-function (car region)) + (goto-char (org-fold-core-next-visibility-change (point) (cdr region) 'ignore-hidden))))) + +(defun org-fold-core--isearch-show-temporary (region hide-p) + "Temporarily reveal text in REGION. +Hide text instead if HIDE-P is non-nil." + (if (not hide-p) + (let ((pos (car region))) + (while (< pos (cdr region)) + (let ((spec-no-open + (catch :found + (dolist (spec (org-fold-core-get-folding-spec 'all pos)) + (unless (org-fold-core-get-folding-spec-property spec :isearch-open) + (throw :found spec)))))) + (if spec-no-open + ;; Skip regions folded with folding specs that cannot be opened. + (setq pos (org-fold-core-next-folding-state-change spec-no-open pos (cdr region))) + (dolist (spec (org-fold-core-get-folding-spec 'all pos)) + (push (cons spec (org-fold-core-get-region-at-point spec pos)) (gethash region org-fold-core--isearch-local-regions))) + (org-fold-core--isearch-show region) + (setq pos (org-fold-core-next-folding-state-change nil pos (cdr region))))))) + (mapc (lambda (val) (org-fold-core-region (cadr val) (cddr val) t (car val))) (gethash region org-fold-core--isearch-local-regions)) + (remhash region org-fold-core--isearch-local-regions))) + +(defvar-local org-fold-core--isearch-special-specs nil + "List of specs that can break visibility state when converted to overlays. +This is a hack, but I do not see a better way around until isearch +gets support of text properties.") +(defun org-fold-core--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the searcheable folded regions will be changed to use overlays +instead of text properties. The created overlays will be stored in +`org-fold-core--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + ;; We need loop below to make sure that we clean all invisible + ;; properties, which may be nested. + (dolist (spec (org-fold-core-get-folding-spec 'all pos)) + (unless (org-fold-core-get-folding-spec-property spec :isearch-ignore) + (let* ((region (org-fold-core-get-region-at-point spec pos))) + (when (memq spec org-fold-core--isearch-special-specs) + (setq pos (min pos (car region))) + (setq end (max end (cdr region)))) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (org-fold-core-region (car region) (cdr region) nil spec) + ;; The overlay is modelled after `outline-flag-region' + ;; [2020-05-09 Sat] overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + (overlay-put o 'org-invisible spec) + ;; Make sure that overlays are applied in the same order + ;; with the folding specs. + ;; Note: `memq` returns cdr with car equal to the first + ;; found matching element. + (overlay-put o 'priority (length (memq spec (org-fold-core-folding-spec-list)))) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (if (org-fold-core-get-folding-spec-property spec :isearch-open) + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (overlay-put o 'isearch-open-invisible #'ignore) + (overlay-put o 'isearch-open-invisible-temporary #'ignore)) + (push o org-fold-core--isearch-overlays)))))) + (setq pos (org-fold-core-next-folding-state-change nil pos end))))) + +(defun org-fold-core--isearch-filter-predicate-overlays (beg end) + "Return non-nil if text between BEG and END is deemed visible by isearch. +This function is intended to be used as `isearch-filter-predicate'." + (org-fold-core--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org-fold-core--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (let ((spec (if isearch-mode-end-hook-quit + ;; Restore all folds. + (overlay-get ov 'org-invisible) + ;; Leave opened folds open. + (overlay-get ov 'invisible)))) + ;; Ignore deleted overlays. + (when (and spec + (overlay-buffer ov)) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (when (<= (overlay-end ov) (point-max)) + (org-fold-core-region (overlay-start ov) (overlay-end ov) t spec))))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org-fold-core--clear-isearch-overlays () + "Convert overlays from `org-fold-core--isearch-overlays' back to text properties." + (when org-fold-core--isearch-overlays + (mapc #'org-fold-core--clear-isearch-overlay org-fold-core--isearch-overlays) + (setq org-fold-core--isearch-overlays nil))) + +;;; Handling changes in folded elements + +(defvar org-fold-core--ignore-modifications nil + "Non-nil: skip processing modifications in `org-fold-core--fix-folded-region'.") +(defvar org-fold-core--ignore-fragility-checks nil + "Non-nil: skip fragility checks in `org-fold-core--fix-folded-region'.") + +(defmacro org-fold-core-ignore-modifications (&rest body) + "Run BODY ignoring buffer modifications in `org-fold-core--fix-folded-region'." + (declare (debug (form body)) (indent 1)) + `(let ((org-fold-core--ignore-modifications t)) + (unwind-protect (progn ,@body) + (setq org-fold-core--last-buffer-chars-modified-tick (buffer-chars-modified-tick))))) + +(defmacro org-fold-core-ignore-fragility-checks (&rest body) + "Run BODY skipping :fragility checks in `org-fold-core--fix-folded-region'." + (declare (debug (form body)) (indent 1)) + `(let ((org-fold-core--ignore-fragility-checks t)) + (progn ,@body))) + +(defvar-local org-fold-core--last-buffer-chars-modified-tick nil + "Variable storing the last return value of `buffer-chars-modified-tick'.") + +(defun org-fold-core--fix-folded-region (from to _) + "Process modifications in folded elements within FROM . TO region. +This function intended to be used as one of `after-change-functions'. + +This function does nothing if text the only modification was changing +text properties (for the sake of reducing overheads). + +If a text was inserted into invisible region, hide the inserted text. +If a text was inserted in front/back of the region, hide it according +to :font-sticky/:rear-sticky folding spec property. + +If the folded region is folded with a spec with non-nil :fragile +property, unfold the region if the :fragile function returns non-nil." + ;; If no insertions or deletions in buffer, skip all the checks. + (unless (or (eq org-fold-core--last-buffer-chars-modified-tick (buffer-chars-modified-tick)) + org-fold-core--ignore-modifications + (memql 'ignore-modification-checks org-fold-core--optimise-for-huge-buffers)) + ;; Store the new buffer modification state. + (setq org-fold-core--last-buffer-chars-modified-tick (buffer-chars-modified-tick)) + (save-match-data + ;; Handle changes in all the indirect buffers and in the base + ;; buffer. Work around Emacs bug#46982. + (when (eq org-fold-core-style 'text-properties) + (org-fold-core-cycle-over-indirect-buffers + ;; Re-hide text inserted in the middle/font/back of a folded + ;; region. + (unless (equal from to) ; Ignore deletions. + (dolist (spec (org-fold-core-folding-spec-list)) + ;; Reveal fully invisible text inserted in the middle + ;; of visible portion of the buffer. This is needed, + ;; for example, when there was a deletion in a folded + ;; heading, the heading was unfolded, end `undo' was + ;; called. The `undo' would insert the folded text. + (when (and (or (eq from (point-min)) + (not (org-fold-core-folded-p (1- from) spec))) + (or (eq to (point-max)) + (not (org-fold-core-folded-p to spec))) + (org-fold-core-region-folded-p from to spec)) + (org-fold-core-region from to nil spec)) + ;; Look around and fold the new text if the nearby folds are + ;; sticky. + (unless (org-fold-core-region-folded-p from to spec) + (let ((spec-to (org-fold-core-get-folding-spec spec (min to (1- (point-max))))) + (spec-from (org-fold-core-get-folding-spec spec (max (point-min) (1- from))))) + ;; Reveal folds around undoed deletion. + (when undo-in-progress + (let ((lregion (org-fold-core-get-region-at-point spec (max (point-min) (1- from)))) + (rregion (org-fold-core-get-region-at-point spec (min to (1- (point-max)))))) + (if (and lregion rregion) + (org-fold-core-region (car lregion) (cdr rregion) nil spec) + (when lregion + (org-fold-core-region (car lregion) (cdr lregion) nil spec)) + (when rregion + (org-fold-core-region (car rregion) (cdr rregion) nil spec))))) + ;; Hide text inserted in the middle of a fold. + (when (and (or spec-from (eq from (point-min))) + (or spec-to (eq to (point-max))) + (or spec-from spec-to) + (eq spec-to spec-from) + (or (org-fold-core-get-folding-spec-property spec :front-sticky) + (org-fold-core-get-folding-spec-property spec :rear-sticky))) + (unless (and (eq from (point-min)) (eq to (point-max))) ; Buffer content replaced. + (org-fold-core-region from to t (or spec-from spec-to)))) + ;; Hide text inserted at the end of a fold. + (when (and spec-from (org-fold-core-get-folding-spec-property spec-from :rear-sticky)) + (org-fold-core-region from to t spec-from)) + ;; Hide text inserted in front of a fold. + (when (and spec-to + (not (eq to (point-max))) ; Text inserted at the end of buffer is not prepended anywhere. + (org-fold-core-get-folding-spec-property spec-to :front-sticky)) + (org-fold-core-region from to t spec-to)))))))) + ;; Process all the folded text between `from' and `to'. Do it + ;; only in current buffer to avoid verifying semantic structure + ;; multiple times in indirect buffers that have exactly same + ;; text anyway. + (unless (or org-fold-core--ignore-fragility-checks + (memql 'ignore-fragility-checks org-fold-core--optimise-for-huge-buffers)) + (dolist (func org-fold-core-extend-changed-region-functions) + (let ((new-region (funcall func from to))) + (setq from (car new-region)) + (setq to (cdr new-region)))) + (dolist (spec (org-fold-core-folding-spec-list)) + ;; No action is needed when :fragile is nil for the spec. + (when (org-fold-core-get-folding-spec-property spec :fragile) + (org-with-wide-buffer + ;; Expand the considered region to include partially present fold. + ;; Note: It is important to do this inside loop over all + ;; specs. Otherwise, the region may be expanded to huge + ;; outline fold, potentially involving majority of the + ;; buffer. That would cause the below code to loop over + ;; almost all the folds in buffer, which would be too slow. + (let ((local-from from) + (local-to to) + (region-from (org-fold-core-get-region-at-point spec (max (point-min) (1- from)))) + (region-to (org-fold-core-get-region-at-point spec (min to (1- (point-max)))))) + (when region-from (setq local-from (car region-from))) + (when region-to (setq local-to (cdr region-to))) + (let ((pos local-from)) + ;; Move to the first hidden region. + (unless (org-fold-core-get-folding-spec spec pos) + (setq pos (org-fold-core-next-folding-state-change spec pos local-to))) + ;; Cycle over all the folds. + (while (< pos local-to) + (save-match-data ; we should not clobber match-data in after-change-functions + (let ((fold-begin (and (org-fold-core-get-folding-spec spec pos) + pos)) + (fold-end (org-fold-core-next-folding-state-change spec pos local-to))) + (when (and fold-begin fold-end) + (when (save-excursion + (funcall (org-fold-core-get-folding-spec-property spec :fragile) + (cons fold-begin fold-end) + spec)) + ;; Reveal completely, not just from the SPEC. + (org-fold-core-region fold-begin fold-end nil))))) + ;; Move to next fold. + (setq pos (org-fold-core-next-folding-state-change spec pos local-to)))))))))))) + +;;; Hanlding killing/yanking of folded text + +;; Backward compatibility with Emacs 24. +(defun org-fold-core--seq-partition (list n) + "Return list of elements of LIST grouped into sub-sequences of length N. +The last list may contain less than N elements. If N is a +negative integer or 0, nil is returned." + (if (fboundp 'seq-partition) + (seq-partition list n) + (unless (< n 1) + (let ((result '())) + (while list + (let (part) + (dotimes (_ n) + (when list (push (car list) part))) + (push part result)) + (dotimes (_ n) + (setq list (cdr list)))) + (nreverse result))))) + +;; By default, all the text properties of the killed text are +;; preserved, including the folding text properties. This can be +;; awkward when we copy a text from an indirect buffer to another +;; indirect buffer (or the base buffer). The copied text might be +;; visible in the source buffer, but might disappear if we yank it in +;; another buffer. This happens in the following situation: +;; ---- base buffer ---- +;; * Headline<begin fold> +;; Some text hidden in the base buffer, but revealed in the indirect +;; buffer.<end fold> +;; * Another headline +;; +;; ---- end of base buffer ---- +;; ---- indirect buffer ---- +;; * Headline +;; Some text hidden in the base buffer, but revealed in the indirect +;; buffer. +;; * Another headline +;; +;; ---- end of indirect buffer ---- +;; If we copy the text under "Headline" from the indirect buffer and +;; insert it under "Another headline" in the base buffer, the inserted +;; text will be hidden since it's folding text properties are copyed. +;; Basically, the copied text would have two sets of folding text +;; properties: (1) Properties for base buffer telling that the text is +;; hidden; (2) Properties for the indirect buffer telling that the +;; text is visible. The first set of the text properties in inactive +;; in the indirect buffer, but will become active once we yank the +;; text back into the base buffer. +;; +;; To avoid the above situation, we simply clear all the properties, +;; unrealated to current buffer when a text is copied. +;; FIXME: Ideally, we may want to carry the folding state of copied +;; text between buffer (probably via user customisation). +(defun org-fold-core--buffer-substring-filter (beg end &optional delete) + "Clear folding state in killed text. +This function is intended to be used as `filter-buffer-substring-function'. +The arguments and return value are as specified for `filter-buffer-substring'." + (let ((return-string (buffer-substring--filter beg end delete)) + ;; The list will be used as an argument to `remove-text-properties'. + props-list) + ;; There is no easy way to examine all the text properties of a + ;; string, so we utilise the fact that printed string + ;; representation lists all its properties. + ;; Loop over the elements of string representation. + (unless (or (string= "" return-string) + (<= end beg) + (eq org-fold-core-style 'overlays)) + ;; Collect all the text properties the string is completely + ;; hidden with. + (dolist (spec (org-fold-core-folding-spec-list)) + (when (and (org-fold-core-region-folded-p beg end spec) + (org-region-invisible-p beg end)) + (push (org-fold-core--property-symbol-get-create spec nil t) props-list))) + (dolist (plist + (if (fboundp 'object-intervals) + (object-intervals return-string) + ;; Backward compatibility with Emacs <28. + ;; FIXME: Is there any better way to do it? + ;; Yes, it is a hack. + ;; The below gives us string representation as a list. + ;; Note that we need to remove unreadable values, like markers (#<...>). + (org-fold-core--seq-partition + (cdr (let ((data (read (replace-regexp-in-string + "^#(" "(" + (replace-regexp-in-string + " #(" " (" + (replace-regexp-in-string + "#<[^>]+>" "dummy" + ;; Get text representation of the string object. + ;; Make sure to print everything (see `prin1' docstring). + ;; `prin1' is used to print "%S" format. + (let (print-level print-length) + (format "%S" return-string)))))))) + (if (listp data) data (list data)))) + 3))) + (let* ((start (car plist)) + (fin (cadr plist)) + (plist (car (cddr plist)))) + ;; Only lists contain text properties. + (when (listp plist) + ;; Collect all the relevant text properties. + (while plist + (let* ((prop (car plist)) + (prop-name (symbol-name prop))) + ;; Reveal hard-hidden text. See + ;; `org-fold-core--optimise-for-huge-buffers'. + (when (and (eq prop 'invisible) + (member (cadr plist) (org-fold-core-folding-spec-list))) + (remove-text-properties start fin '(invisible t) return-string)) + ;; We do not care about values now. + (setq plist (cddr plist)) + (when (string-match-p org-fold-core--spec-property-prefix prop-name) + ;; Leave folding specs from current buffer. See + ;; comments in `org-fold-core--property-symbol-get-create' to + ;; understand why it works. + (unless (member prop (cdr (assq 'invisible char-property-alias-alist))) + (push prop props-list)))))))) + (remove-text-properties 0 (length return-string) props-list return-string)) + return-string)) + +;;; Do not fontify folded text until needed. + +(defun org-fold-core-fontify-region (beg end loudly &optional force) + "Run `font-lock-default-fontify-region' in visible regions." + (let ((pos beg) next + (org-fold-core--fontifying t)) + (while (< pos end) + (setq next (org-fold-core-next-folding-state-change + (if force nil + (let (result) + (dolist (spec (org-fold-core-folding-spec-list)) + (when (and (not (org-fold-core-get-folding-spec-property spec :visible)) + (org-fold-core-get-folding-spec-property spec :font-lock-skip)) + (push spec result))) + result)) + pos + end)) + (while (and (not (catch :found + (dolist (spec (org-fold-core-get-folding-spec 'all next)) + (when (org-fold-core-get-folding-spec-property spec :font-lock-skip) + (throw :found spec))))) + (< next end)) + (setq next (org-fold-core-next-folding-state-change nil next end))) + (save-excursion + (font-lock-default-fontify-region pos next loudly) + (save-match-data + (unless (<= pos (point) next) + (run-hook-with-args 'org-fold-core-first-unfold-functions pos next)))) + (put-text-property pos next 'org-fold-core-fontified t) + (setq pos next)))) + +(defun org-fold-core-update-optimisation (beg end) + "Update huge buffer optimisation between BEG and END. +See `org-fold-core--optimise-for-huge-buffers'." + (when (and (memql 'grab-invisible org-fold-core--optimise-for-huge-buffers) + (eq org-fold-core-style 'text-properties)) + (let ((pos beg)) + (while (< pos end) + (when (and (org-fold-core-folded-p pos (caar org-fold-core--specs)) + (not (eq (caar org-fold-core--specs) (get-text-property pos 'invisible)))) + (put-text-property pos (org-fold-core-next-folding-state-change (caar org-fold-core--specs) pos end) + 'invisible (caar org-fold-core--specs))) + (setq pos (org-fold-core-next-folding-state-change (caar org-fold-core--specs) pos end)))))) + +(defun org-fold-core-remove-optimisation (beg end) + "Remove huge buffer optimisation between BEG and END. +See `org-fold-core--optimise-for-huge-buffers'." + (when (and (memql 'grab-invisible org-fold-core--optimise-for-huge-buffers) + (eq org-fold-core-style 'text-properties)) + (let ((pos beg)) + (while (< pos end) + (if (and (org-fold-core-folded-p pos (caar org-fold-core--specs)) + (eq (caar org-fold-core--specs) (get-text-property pos 'invisible))) + (remove-text-properties pos (org-fold-core-next-folding-state-change (caar org-fold-core--specs) pos end) + '(invisible t))) + (setq pos (org-fold-core-next-folding-state-change (caar org-fold-core--specs) pos end)))))) + +(provide 'org-fold-core) + +;;; org-fold-core.el ends here ^ permalink raw reply related [flat|nested] 192+ messages in thread
* [PATCH 02/35] Separate folding functions from org.el into new library: org-fold 2022-01-29 11:37 ` [PATCH 00/35] Merge org-fold feature branch Ihor Radchenko 2022-01-29 11:37 ` [PATCH 01/35] Add org-fold-core: new folding engine Ihor Radchenko @ 2022-01-29 11:37 ` Ihor Radchenko 2022-01-29 11:37 ` [PATCH 03/35] Separate cycling functions from org.el into new library: org-cycle Ihor Radchenko ` (33 subsequent siblings) 35 siblings, 0 replies; 192+ messages in thread From: Ihor Radchenko @ 2022-01-29 11:37 UTC (permalink / raw) To: Bastien, Kyle Meyer, Nicolas Goaziou, Karl Voit, Christian Heinrich, emacs-orgmode Cc: Ihor Radchenko [-- Attachment #1: Type: text/plain, Size: 150 bytes --] --- lisp/org-fold.el | 1135 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1135 insertions(+) create mode 100644 lisp/org-fold.el [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0002-Separate-folding-functions-from-org.el-into-new-libr.patch --] [-- Type: text/x-patch; name="0002-Separate-folding-functions-from-org.el-into-new-libr.patch", Size: 49902 bytes --] diff --git a/lisp/org-fold.el b/lisp/org-fold.el new file mode 100644 index 000000000..52717fd86 --- /dev/null +++ b/lisp/org-fold.el @@ -0,0 +1,1135 @@ +;;; org-fold.el --- Folding of Org entries -*- lexical-binding: t; -*- +;; +;; Copyright (C) 2020-2020 Free Software Foundation, Inc. +;; +;; Author: Ihor Radchenko <yantar92 at gmail dot com> +;; Keywords: folding, invisible text +;; Homepage: https://orgmode.org +;; +;; This file is part of GNU Emacs. +;; +;; GNU Emacs is free software: you can redistribute it and/or modify +;; it under the terms of the GNU General Public License as published by +;; the Free Software Foundation, either version 3 of the License, or +;; (at your option) any later version. + +;; GNU Emacs is distributed in the hope that it will be useful, +;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;; GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public License +;; along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>. +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; +;;; Commentary: + +;; This file contains code handling temporary invisibility (folding +;; and unfolding) of text in org buffers. + +;; The folding is implemented using generic org-fold-core library. This file +;; contains org-specific implementation of the folding. Also, various +;; useful functions from org-fold-core are aliased under shorted `org-fold' +;; prefix. + +;; The following features are implemented: +;; - Folding/unfolding various Org mode elements and regions of Org buffers: +;; + Region before first heading; +;; + Org headings, their text, children (subtree), siblings, parents, etc; +;; + Org blocks and drawers +;; - Revealing Org structure around invisible point location +;; - Revealing folded Org elements broken by user edits + +;;; Code: + +(require 'org-macs) +(require 'org-fold-core) + +(defvar org-inlinetask-min-level) +(defvar org-link--link-folding-spec) +(defvar org-link--description-folding-spec) +(defvar org-odd-levels-only) +(defvar org-drawer-regexp) +(defvar org-property-end-re) +(defvar org-link-descriptive) +(defvar org-outline-regexp-bol) +(defvar org-custom-properties-hidden-p) +(defvar org-archive-tag) + +;; Needed for overlays only +(defvar org-custom-properties-overlays) + +(declare-function isearch-filter-visible "isearch" (beg end)) +(declare-function org-element-type "org-element" (element)) +(declare-function org-element-at-point "org-element" (&optional pom cached-only)) +(declare-function org-element-property "org-element" (property element)) +(declare-function org-element--current-element "org-element" (limit &optional granularity mode structure)) +(declare-function org-element--cache-active-p "org-element" ()) +(declare-function org-toggle-custom-properties-visibility "org" ()) +(declare-function org-item-re "org-list" ()) +(declare-function org-up-heading-safe "org" ()) +(declare-function org-get-tags "org" (&optional pos local fontify)) +(declare-function org-get-valid-level "org" (level &optional change)) +(declare-function org-before-first-heading-p "org" ()) +(declare-function org-goto-sibling "org" (&optional previous)) +(declare-function org-block-map "org" (function &optional start end)) +(declare-function org-map-region "org" (fun beg end)) +(declare-function org-end-of-subtree "org" (&optional invisible-ok to-heading)) +(declare-function org-back-to-heading-or-point-min "org" (&optional invisible-ok)) +(declare-function org-back-to-heading "org" (&optional invisible-ok)) +(declare-function org-at-heading-p "org" (&optional invisible-not-ok)) +(declare-function org-cycle-hide-drawers "org-cycle" (state)) + +(declare-function outline-show-branches "outline" ()) +(declare-function outline-hide-sublevels "outline" (levels)) +(declare-function outline-get-next-sibling "outline" ()) +(declare-function outline-invisible-p "outline" (&optional pos)) +(declare-function outline-next-heading "outline" ()) + +;;; Customization + +(defgroup org-fold-reveal-location nil + "Options about how to make context of a location visible." + :tag "Org Reveal Location" + :group 'org-structure) + +(defcustom org-fold-show-context-detail '((agenda . local) + (bookmark-jump . lineage) + (isearch . lineage) + (default . ancestors)) + "Alist between context and visibility span when revealing a location. + +\\<org-mode-map>Some actions may move point into invisible +locations. As a consequence, Org always exposes a neighborhood +around point. How much is shown depends on the initial action, +or context. Valid contexts are + + agenda when exposing an entry from the agenda + org-goto when using the command `org-goto' (`\\[org-goto]') + occur-tree when using the command `org-occur' (`\\[org-sparse-tree] /') + tags-tree when constructing a sparse tree based on tags matches + link-search when exposing search matches associated with a link + mark-goto when exposing the jump goal of a mark + bookmark-jump when exposing a bookmark location + isearch when exiting from an incremental search + default default for all contexts not set explicitly + +Allowed visibility spans are + + minimal show current headline; if point is not on headline, + also show entry + + local show current headline, entry and next headline + + ancestors show current headline and its direct ancestors; if + point is not on headline, also show entry + + ancestors-full show current subtree and its direct ancestors + + lineage show current headline, its direct ancestors and all + their children; if point is not on headline, also show + entry and first child + + tree show current headline, its direct ancestors and all + their children; if point is not on headline, also show + entry and all children + + canonical show current headline, its direct ancestors along with + their entries and children; if point is not located on + the headline, also show current entry and all children + +As special cases, a nil or t value means show all contexts in +`minimal' or `canonical' view, respectively. + +Some views can make displayed information very compact, but also +make it harder to edit the location of the match. In such +a case, use the command `org-fold-reveal' (`\\[org-fold-reveal]') to show +more context." + :group 'org-fold-reveal-location + :version "26.1" + :package-version '(Org . "9.0") + :type '(choice + (const :tag "Canonical" t) + (const :tag "Minimal" nil) + (repeat :greedy t :tag "Individual contexts" + (cons + (choice :tag "Context" + (const agenda) + (const org-goto) + (const occur-tree) + (const tags-tree) + (const link-search) + (const mark-goto) + (const bookmark-jump) + (const isearch) + (const default)) + (choice :tag "Detail level" + (const minimal) + (const local) + (const ancestors) + (const ancestors-full) + (const lineage) + (const tree) + (const canonical)))))) + +(defvar org-fold-reveal-start-hook nil + "Hook run before revealing a location.") + +(defcustom org-fold-catch-invisible-edits 'smart + "Check if in invisible region before inserting or deleting a character. +Valid values are: + +nil Do not check, so just do invisible edits. +error Throw an error and do nothing. +show Make point visible, and do the requested edit. +show-and-error Make point visible, then throw an error and abort the edit. +smart Make point visible, and do insertion/deletion if it is + adjacent to visible text and the change feels predictable. + Never delete a previously invisible character or add in the + middle or right after an invisible region. Basically, this + allows insertion and backward-delete right before ellipses. + FIXME: maybe in this case we should not even show?" + :group 'org-edit-structure + :version "24.1" + :type '(choice + (const :tag "Do not check" nil) + (const :tag "Throw error when trying to edit" error) + (const :tag "Unhide, but do not do the edit" show-and-error) + (const :tag "Show invisible part and do the edit" show) + (const :tag "Be smart and do the right thing" smart))) + +;;; Core functionality + +;;; API + +;;;; Modifying folding specs + +(defalias 'org-fold-folding-spec-p #'org-fold-core-folding-spec-p) +(defalias 'org-fold-add-folding-spec #'org-fold-core-add-folding-spec) +(defalias 'org-fold-remove-folding-spec #'org-fold-core-remove-folding-spec) + +(defun org-fold-initialize (ellipsis) + "Setup folding in current Org buffer." + (setq-local org-fold-core-isearch-open-function #'org-fold--isearch-reveal) + (setq-local org-fold-core-extend-changed-region-functions (list #'org-fold--extend-changed-region)) + ;; FIXME: Converting org-link + org-description to overlays when + ;; search matches hidden "[[" part of the link, reverses priority of + ;; link and description and hides the whole link. Working around + ;; this until there will be no need to convert text properties to + ;; overlays for isearch. + (setq-local org-fold-core--isearch-special-specs '(org-link)) + (org-fold-core-initialize `((org-fold-outline + (:ellipsis . ,ellipsis) + (:fragile . ,#'org-fold--reveal-outline-maybe) + (:isearch-open . t) + ;; This is needed to make sure that inserting a + ;; new planning line in folded heading is not + ;; revealed. + (:front-sticky . t) + (:rear-sticky . t) + (:font-lock-skip . t) + (:alias . (headline heading outline inlinetask plain-list))) + (org-fold-block + (:ellipsis . ,ellipsis) + (:fragile . ,#'org-fold--reveal-drawer-or-block-maybe) + (:isearch-open . t) + (:front-sticky . t) + (:alias . ( block center-block comment-block + dynamic-block example-block export-block + quote-block special-block src-block + verse-block))) + (org-fold-drawer + (:ellipsis . ,ellipsis) + (:fragile . ,#'org-fold--reveal-drawer-or-block-maybe) + (:isearch-open . t) + (:front-sticky . t) + (:alias . (drawer property-drawer))) + ,org-link--description-folding-spec + ,org-link--link-folding-spec))) + +;;;; Searching and examining folded text + +(defalias 'org-fold-folded-p #'org-fold-core-folded-p) +(defalias 'org-fold-get-folding-spec #'org-fold-core-get-folding-spec) +(defalias 'org-fold-get-folding-specs-in-region #'org-fold-core-get-folding-specs-in-region) +(defalias 'org-fold-get-region-at-point #'org-fold-core-get-region-at-point) +(defalias 'org-fold-next-visibility-change #'org-fold-core-next-visibility-change) +(defalias 'org-fold-previous-visibility-change #'org-fold-core-previous-visibility-change) +(defalias 'org-fold-next-folding-state-change #'org-fold-core-next-folding-state-change) +(defalias 'org-fold-previous-folding-state-change #'org-fold-core-previous-folding-state-change) +(defalias 'org-fold-search-forward #'org-fold-core-search-forward) + +;;;;; Macros + +(defmacro org-fold-save-outline-visibility--overlays (use-markers &rest body) + "Save and