From: Christian Heinrich <firstname.lastname@example.org> To: email@example.com Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Date: Thu, 07 May 2020 13:04:15 +0200 [thread overview] Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <87y2qi8c8w.fsf@localhost> [-- Attachment #1: Type: text/plain, Size: 8181 bytes --] Hi, thanks for your (initial) patch! I traced another error down today and found your code by chance. I tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using outline-show-all which I had to use anyways to deal with my huge file.) I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind sending a patch file? I hope that this ends up in the master branch at some point. Thanks again! Christian On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote: > > You cannot. You may however mimic it with `cursor-sensor-functions' text > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. > > Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil > internally. Anyway, I came up with some workaround, which seems to work > (see below). Though it would be better if isearch supported hidden text > in addition to overlays. > > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > > experimenting with `cursor-sensor-functions'. > > So far, I came up with the following partial solution searching and > showing hidden text. > > ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we > ;; cannot even use cursor-sensor-functions as a workaround > ;; I used a less ideas approach with advice to isearch-search-string as > ;; a workaround > > (defun org-find-text-property-region (pos prop) > "Find a region containing PROP text property around point POS." > (require 'org-macs) ;; org-with-point-at > (org-with-point-at pos > (let* ((beg (and (get-text-property pos prop) pos)) > (end beg)) > (when beg > (setq beg (or (previous-single-property-change pos prop) > beg)) > (setq end (or (next-single-property-change pos prop) > end)) > (unless (equal beg end) > (cons beg end)))))) > > ;; :FIXME: re-hide properties when point moves away > (define-advice isearch-search-string (:after (&rest _) put-overlay) > "Reveal hidden text at point." > (when-let ((region (org-find-text-property-region (point) 'invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) > 'invisible))) > (remove-text-properties (car region) (cdr region) '(invisible nil)))) > > ;; this seems to be unstable, but I cannot figure out why > (defun org-restore-invisibility-specs (&rest _) > "" > (let ((pos (point-min))) > (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point- > max)) > (when-let ((region (org-find-text-property-region pos 'org-invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org- > invisible)) > (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) > > (add-hook 'post-command-hook #'org-restore-invisibility-specs) > > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > (pcase spec > ('outline > (remove-overlays from to 'invisible spec) > ;; Use `front-advance' since text right before to the beginning of > ;; the overlay belongs to the visible line than to the contents. > (when flag > (let ((o (make-overlay from to nil 'front-advance))) > (overlay-put o 'evaporate t) > (overlay-put o 'invisible spec) > (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > (_ > (with-silent-modifications > (remove-text-properties from to '(invisible nil)) > (when flag > (put-text-property from to 'invisible spec) > ))))) > > ;; This normally deletes invisible text property. We do not want this now. > (defun org-unfontify-region (beg end &optional _maybe_loudly) > "Remove fontification and activation overlays from links." > (font-lock-default-unfontify-region beg end) > (let* ((buffer-undo-list t) > (inhibit-read-only t) (inhibit-point-motion-hooks t) > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > (remove-text-properties beg end > '(mouse-face t keymap t org-linked-text t > ;; Do not remove invisible during fontification > > ;; invisible t > intangible t > org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) > > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. > > I guess that it is a very old story starting from the times when XEmacs > was a thing . I recently heard about binary tree implementation of > overlays (there should be a branch in emacs git repo) , but there was > no update on that branch for a while. So, I do not have much hope on > Emacs implementing efficient overlay access in the near future. (And I > have problems with huge org files already). > >  https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >  https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html > > > Nicolas Goaziou <email@example.com> writes: > > > Hello, > > > > Ihor Radchenko <firstname.lastname@example.org> writes: > > > > > To my surprise, the patch did not break org to unusable state and > > > the performance on the sample org file  improved drastically. You can > > > try by yourself! > > > > It is not a surprise, really. Text properties are much faster than > > overlays, and very close to them features-wise. They are a bit more > > complex to handle, however. > > > > > However, this did introduce some visual glitches with drawer display. > > > Though drawers can still be folded/unfolded with <tab>, they are not > > > folded on org-mode startup for some reason (can be fixed by running > > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) > > > are unfolded for no apparent reason sometimes. A blind guess is that it > > > is something to do with lack of 'isearch-open-invisible, which I am not > > > sure how to set via text properties. > > > > You cannot. You may however mimic it with `cursor-sensor-functions' text > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. > > > > Note there are clear advantages using text properties. For example, when > > you move contents around, text properties are preserved. So there's no > > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > > necessary anymore to re-hide drawers. > > > > > Any thoughts about the use of text properties or about the patch > > > suggestion are welcome. > > > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > > experimenting with `cursor-sensor-functions'. > > > > We could also use text properties for property drawers, and overlays for > > regular ones. This might give us a reasonable speed-up with an > > acceptable feature trade-off. > > > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. > > > > Regards, > > > > -- > > Nicolas Goaziou [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2020-05-07 11:06 UTC|newest] Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-24 6:55 Ihor Radchenko 2020-04-24 8:02 ` Nicolas Goaziou 2020-04-25 0:29 ` stardiviner 2020-04-26 16:04 ` Ihor Radchenko 2020-05-04 16:56 ` Karl Voit 2020-05-07 7:18 ` Karl Voit 2020-05-09 15:43 ` Ihor Radchenko 2020-05-07 11:04 ` Christian Heinrich [this message] 2020-05-09 15:46 ` Ihor Radchenko 2020-05-08 16:38 ` Nicolas Goaziou 2020-05-09 13:58 ` Nicolas Goaziou 2020-05-09 16:22 ` Ihor Radchenko 2020-05-09 17:21 ` Nicolas Goaziou 2020-05-10 5:25 ` Ihor Radchenko 2020-05-10 9:47 ` Nicolas Goaziou 2020-05-10 13:29 ` Ihor Radchenko 2020-05-10 14:46 ` Nicolas Goaziou 2020-05-10 16:21 ` Ihor Radchenko 2020-05-10 16:38 ` Nicolas Goaziou 2020-05-10 17:08 ` Ihor Radchenko 2020-05-10 19:38 ` Nicolas Goaziou 2020-05-09 15:40 ` Ihor Radchenko 2020-05-09 16:30 ` Ihor Radchenko 2020-05-09 17:32 ` Nicolas Goaziou 2020-05-09 18:06 ` Ihor Radchenko 2020-05-10 14:59 ` Nicolas Goaziou 2020-05-10 15:15 ` Kyle Meyer 2020-05-10 16:30 ` Ihor Radchenko 2020-05-10 19:32 ` Nicolas Goaziou 2020-05-12 10:03 ` Nicolas Goaziou 2020-05-17 15:00 ` Ihor Radchenko 2020-05-17 15:40 ` Ihor Radchenko 2020-05-18 14:35 ` Nicolas Goaziou 2020-05-18 16:52 ` Ihor Radchenko 2020-05-19 13:07 ` Nicolas Goaziou 2020-05-23 13:52 ` Ihor Radchenko 2020-05-23 13:53 ` Ihor Radchenko 2020-05-23 15:26 ` Ihor Radchenko 2020-05-26 8:33 ` Nicolas Goaziou 2020-06-02 9:21 ` Ihor Radchenko 2020-06-02 9:23 ` Ihor Radchenko 2020-06-02 12:10 ` Bastien 2020-06-02 13:12 ` Ihor Radchenko 2020-06-02 13:23 ` Bastien 2020-06-02 13:30 ` Ihor Radchenko 2020-06-02 9:25 ` Ihor Radchenko 2020-06-05 7:26 ` Nicolas Goaziou 2020-06-05 8:18 ` Ihor Radchenko 2020-06-05 13:50 ` Nicolas Goaziou 2020-06-08 5:05 ` Ihor Radchenko 2020-06-08 5:06 ` Ihor Radchenko 2020-06-08 5:08 ` Ihor Radchenko 2020-06-10 17:14 ` Nicolas Goaziou 2020-06-21 9:52 ` Ihor Radchenko 2020-06-21 15:01 ` Nicolas Goaziou 2020-08-11 6:45 ` Ihor Radchenko 2020-08-11 23:07 ` Kyle Meyer 2020-08-12 6:29 ` Ihor Radchenko 2020-09-20 5:53 ` Ihor Radchenko 2020-09-20 11:45 ` Kévin Le Gouguec 2020-09-22 9:05 ` Ihor Radchenko 2020-09-22 10:00 ` Ihor Radchenko 2020-09-23 6:16 ` Kévin Le Gouguec 2020-09-23 6:48 ` Ihor Radchenko 2020-09-23 7:09 ` Bastien 2020-09-23 7:30 ` Ihor Radchenko 2020-09-24 18:07 ` Kévin Le Gouguec 2020-09-25 2:16 ` Ihor Radchenko 2020-12-15 17:38 ` [9.4] Fixing logbook visibility during isearch Kévin Le Gouguec 2020-12-16 3:15 ` Ihor Radchenko 2020-12-16 18:05 ` Kévin Le Gouguec 2020-12-17 3:18 ` Ihor Radchenko 2020-12-17 14:50 ` Kévin Le Gouguec 2020-12-18 2:23 ` Ihor Radchenko 2020-12-24 23:37 ` Kévin Le Gouguec 2020-12-25 2:51 ` Ihor Radchenko 2020-12-25 10:59 ` Kévin Le Gouguec 2020-12-25 12:32 ` Ihor Radchenko 2020-12-25 21:35 ` Kévin Le Gouguec 2020-12-26 4:14 ` Ihor Radchenko 2020-12-26 11:44 ` Kévin Le Gouguec 2020-12-26 12:22 ` Ihor Radchenko 2020-12-04 5:58 ` [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko 2021-03-21 9:09 ` Ihor Radchenko 2021-05-03 17:28 ` Bastien 2021-09-21 13:32 ` Timothy 2021-10-26 17:25 ` Matt Price 2021-10-27 6:27 ` Ihor Radchenko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://www.orgmode.org/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).