From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id KKmUMFXRtl7oEQAA0tVLHw (envelope-from ) for ; Sat, 09 May 2020 15:50:45 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id OHFfOGLRtl6bYwAAB5/wlQ (envelope-from ) for ; Sat, 09 May 2020 15:50:58 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9247A940F29 for ; Sat, 9 May 2020 15:50:56 +0000 (UTC) Received: from localhost ([::1]:40928 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jXRke-0001rh-5h for larch@yhetil.org; Sat, 09 May 2020 11:50:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46610) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jXRkB-0001rM-QF for emacs-orgmode@gnu.org; Sat, 09 May 2020 11:50:27 -0400 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]:33254) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jXRkA-0001Sc-GV for emacs-orgmode@gnu.org; Sat, 09 May 2020 11:50:27 -0400 Received: by mail-pg1-x52c.google.com with SMTP id a4so2355286pgc.0 for ; Sat, 09 May 2020 08:50:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:in-reply-to:references:date:message-id:mime-version; bh=ixy7MGWsq13EUi29UnMj20D/48jLkmmXjpTfbwNfimM=; b=M71d7wSy9i+KhPInv9BgVhMhwIXrzYwzby1F+yb7ENUz8ZKyPz+HhVeB+kxEMQfemt MbMp+d+hUtnfgaOgkEQM7z7yv9R8HdNU3+T7Ew9/QgCLyCXhdc9nwnT7BV2p1GdJv32q mEghCXhsLmdDcAhMGE3vsE3VzKOzfDIByXP6OxRZ8umUySHKMBOBbVDqdU8MESYsOvIC 1cc8vwz86656PKepd/7nU+8bSYaidIsmQ7TEIHmeksKBDhfQYVH2TAHX+ZZ5U0jZR4z/ Qd/6b67YyMiylLgNIevhY3+E+hISrCd2uIE9yKOQg7yAHShgYK/aXPH9hdnxTEniqZtr 6+7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version; bh=ixy7MGWsq13EUi29UnMj20D/48jLkmmXjpTfbwNfimM=; b=ZW9s7bceSzISKKx/6gNs5z7U6N4cMjHLKkJ7n9CSz+8jYTVI2+q6NwKzlDbIRc48p7 iSsjSyKcWy9MEXorlLG8k1IP6ToVSPDnKhZfdxSocUWO18Rm3U+OV5pqrgChrG+5z9HC W0+bJYsoo1u2hMi2aVbE4lgZHoXAgr9doko2yyqH0bIaBenai6ASxZBhjiccPJpwe/bs j/ovSNv8lH2/zr2+g4ijsn3BY7zp/gPr4E80s3RRZzc8rVRgSFRP21NKXAh+Psh0EGo5 d+cyeihmIiCWWC6e75rqeewYosj6yv1TXhVr9z4WJ6BzTq4Lp/WIAciSNg03MvLoSqny jRMw== X-Gm-Message-State: AGi0PubfNqYZN7SzLiRve9aPNcYaeRFFmslbeLdABEw/EKdlmEo73hD6 ulPiXM/la7FcQyfQYcUEKVMe7UjJMCCU5A== X-Google-Smtp-Source: APiQypJKbD+/l8yYas1SNuC9YQ6zKiMqXWvAgaNjX+D7M5rMuzBmaUJpk+5JXQY6WT1JegrFRHQ0GA== X-Received: by 2002:a63:935d:: with SMTP id w29mr6705985pgm.92.1589039424923; Sat, 09 May 2020 08:50:24 -0700 (PDT) Received: from localhost ([210.3.160.222]) by smtp.gmail.com with ESMTPSA id h9sm4814319pfo.129.2020.05.09.08.50.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 May 2020 08:50:24 -0700 (PDT) From: Ihor Radchenko To: Christian Heinrich , emacs-orgmode@gnu.org Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers In-Reply-To: <78ccc02ac16d47c97766865fdf5206df57023444.camel@gladbachcity.de> References: <87h7x9e5jo.fsf@localhost> <875zdpia5i.fsf@nicolasgoaziou.fr> <87y2qi8c8w.fsf@localhost> <78ccc02ac16d47c97766865fdf5206df57023444.camel@gladbachcity.de> Date: Sat, 09 May 2020 23:46:08 +0800 Message-ID: <87blmxw1q7.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::52c; envelope-from=yantar92@gmail.com; helo=mail-pg1-x52c.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 X-Spam-Score: -1.21 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=M71d7wSy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Scan-Result: default: False [-1.21 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.54020363824083]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.51.188.0/24:c]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_BLOCKED(0.00)[gmail.com:dkim,209.51.188.17:from]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.09), country: US(-0.00), ip: 209.51.188.17(-0.54)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; TAGGED_FROM(0.00)[larch=yhetil.org]; FROM_NEQ_ENVFROM(0.00)[yantar92@gmail.com,emacs-orgmode-bounces@gnu.org]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_COUNT_FIVE(0.00)[6]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[reddit.com:url,gnu.org:url,sutd.edu.sg:email,gladbachcity.de:email,nicolasgoaziou.fr:email]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[emacs-orgmode@gnu.org]; HAS_LIST_UNSUB(-0.01)[]; DNSWL_BLOCKED(0.00)[209.51.188.17:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.51.188.17:from]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: DHRqK0d/DpdI > I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind > sending a patch file? I hope that this ends up in the master branch at some point. I have sent the patch in another email. Will appreciate any feedback. Best, Ihor Christian Heinrich writes: > Hi, > > thanks for your (initial) patch! I traced another error down today and found your code by chance. I > tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this > patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when > outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using > outline-show-all which I had to use anyways to deal with my huge file.) > > I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind > sending a patch file? I hope that this ends up in the master branch at some point. > > Thanks again! > Christian > > On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote: >> > You cannot. You may however mimic it with `cursor-sensor-functions' text >> > property. These assume Cursor Sensor minor mode is active, tho. >> > I haven't tested it, but I assume it would slow down text properties >> > a bit, too, but hopefully not as much as overlays. >> >> Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil >> internally. Anyway, I came up with some workaround, which seems to work >> (see below). Though it would be better if isearch supported hidden text >> in addition to overlays. >> >> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth >> > experimenting with `cursor-sensor-functions'. >> >> So far, I came up with the following partial solution searching and >> showing hidden text. >> >> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we >> ;; cannot even use cursor-sensor-functions as a workaround >> ;; I used a less ideas approach with advice to isearch-search-string as >> ;; a workaround >> >> (defun org-find-text-property-region (pos prop) >> "Find a region containing PROP text property around point POS." >> (require 'org-macs) ;; org-with-point-at >> (org-with-point-at pos >> (let* ((beg (and (get-text-property pos prop) pos)) >> (end beg)) >> (when beg >> (setq beg (or (previous-single-property-change pos prop) >> beg)) >> (setq end (or (next-single-property-change pos prop) >> end)) >> (unless (equal beg end) >> (cons beg end)))))) >> >> ;; :FIXME: re-hide properties when point moves away >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> "Reveal hidden text at point." >> (when-let ((region (org-find-text-property-region (point) 'invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) >> 'invisible))) >> (remove-text-properties (car region) (cdr region) '(invisible nil)))) >> >> ;; this seems to be unstable, but I cannot figure out why >> (defun org-restore-invisibility-specs (&rest _) >> "" >> (let ((pos (point-min))) >> (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point- >> max)) >> (when-let ((region (org-find-text-property-region pos 'org-invisible))) >> (with-silent-modifications >> (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org- >> invisible)) >> (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) >> >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> >> (defun org-flag-region (from to flag spec) >> "Hide or show lines from FROM to TO, according to FLAG. >> SPEC is the invisibility spec, as a symbol." >> (pcase spec >> ('outline >> (remove-overlays from to 'invisible spec) >> ;; Use `front-advance' since text right before to the beginning of >> ;; the overlay belongs to the visible line than to the contents. >> (when flag >> (let ((o (make-overlay from to nil 'front-advance))) >> (overlay-put o 'evaporate t) >> (overlay-put o 'invisible spec) >> (overlay-put o 'isearch-open-invisible #'delete-overlay)))) >> (_ >> (with-silent-modifications >> (remove-text-properties from to '(invisible nil)) >> (when flag >> (put-text-property from to 'invisible spec) >> ))))) >> >> ;; This normally deletes invisible text property. We do not want this now. >> (defun org-unfontify-region (beg end &optional _maybe_loudly) >> "Remove fontification and activation overlays from links." >> (font-lock-default-unfontify-region beg end) >> (let* ((buffer-undo-list t) >> (inhibit-read-only t) (inhibit-point-motion-hooks t) >> (inhibit-modification-hooks t) >> deactivate-mark buffer-file-name buffer-file-truename) >> (decompose-region beg end) >> (remove-text-properties beg end >> '(mouse-face t keymap t org-linked-text t >> ;; Do not remove invisible during fontification >> >> ;; invisible t >> intangible t >> org-emphasis t)) >> (org-remove-font-lock-display-properties beg end))) >> >> > Anyway, the real fix should come from Emacs itself. There are ways to >> > make overlays faster. These ways have already been discussed on the >> > Emacs devel mailing list, but no one implemented them. It is a bit sad >> > that we have to find workarounds for that. >> >> I guess that it is a very old story starting from the times when XEmacs >> was a thing [1]. I recently heard about binary tree implementation of >> overlays (there should be a branch in emacs git repo) [2], but there was >> no update on that branch for a while. So, I do not have much hope on >> Emacs implementing efficient overlay access in the near future. (And I >> have problems with huge org files already). >> >> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html >> >> >> Nicolas Goaziou writes: >> >> > Hello, >> > >> > Ihor Radchenko writes: >> > >> > > To my surprise, the patch did not break org to unusable state and >> > > the performance on the sample org file [3] improved drastically. You can >> > > try by yourself! >> > >> > It is not a surprise, really. Text properties are much faster than >> > overlays, and very close to them features-wise. They are a bit more >> > complex to handle, however. >> > >> > > However, this did introduce some visual glitches with drawer display. >> > > Though drawers can still be folded/unfolded with , they are not >> > > folded on org-mode startup for some reason (can be fixed by running >> > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) >> > > are unfolded for no apparent reason sometimes. A blind guess is that it >> > > is something to do with lack of 'isearch-open-invisible, which I am not >> > > sure how to set via text properties. >> > >> > You cannot. You may however mimic it with `cursor-sensor-functions' text >> > property. These assume Cursor Sensor minor mode is active, tho. >> > I haven't tested it, but I assume it would slow down text properties >> > a bit, too, but hopefully not as much as overlays. >> > >> > Note there are clear advantages using text properties. For example, when >> > you move contents around, text properties are preserved. So there's no >> > more need for the `org-cycle-hide-drawer' dance, i.e., it is not >> > necessary anymore to re-hide drawers. >> > >> > > Any thoughts about the use of text properties or about the patch >> > > suggestion are welcome. >> > >> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth >> > experimenting with `cursor-sensor-functions'. >> > >> > We could also use text properties for property drawers, and overlays for >> > regular ones. This might give us a reasonable speed-up with an >> > acceptable feature trade-off. >> > >> > Anyway, the real fix should come from Emacs itself. There are ways to >> > make overlays faster. These ways have already been discussed on the >> > Emacs devel mailing list, but no one implemented them. It is a bit sad >> > that we have to find workarounds for that. >> > >> > Regards, >> > >> > -- >> > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg