From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id GDi4MwyypV56BgAA0tVLHw (envelope-from ) for ; Sun, 26 Apr 2020 16:08:44 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id aNMmBxSypV5lPwAA1q6Kng (envelope-from ) for ; Sun, 26 Apr 2020 16:08:52 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id F0A9C9430A5 for ; Sun, 26 Apr 2020 16:08:50 +0000 (UTC) Received: from localhost ([::1]:34512 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jSjpq-0002gm-Ot for larch@yhetil.org; Sun, 26 Apr 2020 12:08:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41968) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jSjpO-0002gQ-LT for emacs-orgmode@gnu.org; Sun, 26 Apr 2020 12:08:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jSjpN-0007Ra-4h for emacs-orgmode@gnu.org; Sun, 26 Apr 2020 12:08:22 -0400 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]:39077) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jSjpM-0007F7-Mz for emacs-orgmode@gnu.org; Sun, 26 Apr 2020 12:08:20 -0400 Received: by mail-pj1-x1034.google.com with SMTP id e6so6249513pjt.4 for ; Sun, 26 Apr 2020 09:08:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=JgmMKTJ5epCN+IDx+NYoz3sXVdhEcX21DAr2VlW55ds=; b=ZJg4DNg+KaXgvrsyQgyKII5RK6K9eO4FfPy34cAPGBpucJq7rRtY5E/S47iAdtQ/iZ qShpi/RC59kzlp0x3UjZqg/G3E1EDxhRIijDxGkX+Ij6iMCt1RTKJZyXyk0uPpTiWTY6 BGp8PFEVKukFOLiqEJnNUFiIkDvQ1ql48KVURraaARRNTKO0+qxuylvKHcmoNilCJmsn BYJQ9Hkf8s9Ac2wweAkivkp53Qz5VE27Ft/ZIa/B9q1e84wPuHzvecGK916Qb5o5PF91 8l4jEHQ3jM+ij6zpMSq5G1QSTsWQgsGE8W2y3QdCDDfOyoalREQmIxuS7sw6FYJ4JaAE 3pPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=JgmMKTJ5epCN+IDx+NYoz3sXVdhEcX21DAr2VlW55ds=; b=rF6OpUjvxlgqhhlV2ls4eiQkvSvgmCOORxH4jQTo3YaPfkm/dh1N3naH365WXsAKVC 3FO61AaWuy/OEQhph/ODBDvZGEvDbPoUo9CjGregXd8vXxWSyIG7LjS99nERRJg29Mea TAx9BEhVsjjAXpauvkNbU8iMvR68wHT4nzEk+Q/XJ9FR1qsPx2C/08h/Bisx/ycD1YoS 9b1DRTk0oDQNwURBohxIBSvkkF1x6G09wz4MUdHTFhr2Usx5QKCBvNhRbuQ4I8SktpyD mSy+Pgeyui9w6XYYWnV64whLp5CPC9v3I9rXCGl1A4KO2ZIMSvjTbiyjsgPIDg7jLBvJ oZ1Q== X-Gm-Message-State: AGi0Pub4tPU9nKcVyiDPU52FG+TMukcOrJKbLXXlm2qQD/1xSTw26KNh Sgnvg5I1eQq9/x1DDhchyHy4GnKMYKyFJw== X-Google-Smtp-Source: APiQypLvhWiboqdVwxmGOpjgixxTZ3T5iPExeXvxNeXQT9dRf5N81hfxeM1U+fCNTUZL0YQn5r2sBw== X-Received: by 2002:a17:902:a413:: with SMTP id p19mr19508200plq.1.1587917298496; Sun, 26 Apr 2020 09:08:18 -0700 (PDT) Received: from localhost ([210.3.160.222]) by smtp.gmail.com with ESMTPSA id x185sm10158748pfx.155.2020.04.26.09.08.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Apr 2020 09:08:17 -0700 (PDT) From: Ihor Radchenko To: Nicolas Goaziou Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers In-Reply-To: <875zdpia5i.fsf@nicolasgoaziou.fr> References: <87h7x9e5jo.fsf@localhost> <875zdpia5i.fsf@nicolasgoaziou.fr> Date: Mon, 27 Apr 2020 00:04:15 +0800 Message-ID: <87y2qi8c8w.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::1034; envelope-from=yantar92@gmail.com; helo=mail-pj1-x1034.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1034 X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode@gnu.org Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 X-Spam-Score: -1.21 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=ZJg4DNg+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Scan-Result: default: False [-1.21 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.55769243639898]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.51.188.0/24:c]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_FAIL(0.00)[gmail.com:server fail,209.51.188.17:server fail]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.20), country: US(-0.00), ip: 209.51.188.17(-0.56)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[209.51.188.17:server fail]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; TAGGED_FROM(0.00)[larch=yhetil.org]; FROM_NEQ_ENVFROM(0.00)[yantar92@gmail.com,emacs-orgmode-bounces@gnu.org]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; URIBL_BLOCKED(0.00)[nicolasgoaziou.fr:email,reddit.com:url,gnu.org:url,sutd.edu.sg:email]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[emacs-orgmode@gnu.org]; HAS_LIST_UNSUB(-0.01)[]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_SEVEN(0.00)[7]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: cd86dncoAgzk > You cannot. You may however mimic it with `cursor-sensor-functions' text > property. These assume Cursor Sensor minor mode is active, tho. > I haven't tested it, but I assume it would slow down text properties > a bit, too, but hopefully not as much as overlays. Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil internally. Anyway, I came up with some workaround, which seems to work (see below). Though it would be better if isearch supported hidden text in addition to overlays. > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > experimenting with `cursor-sensor-functions'. So far, I came up with the following partial solution searching and showing hidden text. ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we ;; cannot even use cursor-sensor-functions as a workaround ;; I used a less ideas approach with advice to isearch-search-string as ;; a workaround (defun org-find-text-property-region (pos prop) "Find a region containing PROP text property around point POS." (require 'org-macs) ;; org-with-point-at (org-with-point-at pos (let* ((beg (and (get-text-property pos prop) pos)) (end beg)) (when beg (setq beg (or (previous-single-property-change pos prop) beg)) (setq end (or (next-single-property-change pos prop) end)) (unless (equal beg end) (cons beg end)))))) ;; :FIXME: re-hide properties when point moves away (define-advice isearch-search-string (:after (&rest _) put-overlay) "Reveal hidden text at point." (when-let ((region (org-find-text-property-region (point) 'invisible))) (with-silent-modifications (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible))) (remove-text-properties (car region) (cdr region) '(invisible nil)))) ;; this seems to be unstable, but I cannot figure out why (defun org-restore-invisibility-specs (&rest _) "" (let ((pos (point-min))) (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max)) (when-let ((region (org-find-text-property-region pos 'org-invisible))) (with-silent-modifications (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible)) (remove-text-properties (car region) (cdr region) '(org-invisible nil))))))) (add-hook 'post-command-hook #'org-restore-invisibility-specs) (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." (pcase spec ('outline (remove-overlays from to 'invisible spec) ;; Use `front-advance' since text right before to the beginning of ;; the overlay belongs to the visible line than to the contents. (when flag (let ((o (make-overlay from to nil 'front-advance))) (overlay-put o 'evaporate t) (overlay-put o 'invisible spec) (overlay-put o 'isearch-open-invisible #'delete-overlay)))) (_ (with-silent-modifications (remove-text-properties from to '(invisible nil)) (when flag (put-text-property from to 'invisible spec) ))))) ;; This normally deletes invisible text property. We do not want this now. (defun org-unfontify-region (beg end &optional _maybe_loudly) "Remove fontification and activation overlays from links." (font-lock-default-unfontify-region beg end) (let* ((buffer-undo-list t) (inhibit-read-only t) (inhibit-point-motion-hooks t) (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) (remove-text-properties beg end '(mouse-face t keymap t org-linked-text t ;; Do not remove invisible during fontification ;; invisible t intangible t org-emphasis t)) (org-remove-font-lock-display-properties beg end))) > Anyway, the real fix should come from Emacs itself. There are ways to > make overlays faster. These ways have already been discussed on the > Emacs devel mailing list, but no one implemented them. It is a bit sad > that we have to find workarounds for that. I guess that it is a very old story starting from the times when XEmacs was a thing [1]. I recently heard about binary tree implementation of overlays (there should be a branch in emacs git repo) [2], but there was no update on that branch for a while. So, I do not have much hope on Emacs implementing efficient overlay access in the near future. (And I have problems with huge org files already). [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html Nicolas Goaziou writes: > Hello, > > Ihor Radchenko writes: > >> To my surprise, the patch did not break org to unusable state and >> the performance on the sample org file [3] improved drastically. You can >> try by yourself! > > It is not a surprise, really. Text properties are much faster than > overlays, and very close to them features-wise. They are a bit more > complex to handle, however. > >> However, this did introduce some visual glitches with drawer display. >> Though drawers can still be folded/unfolded with , they are not >> folded on org-mode startup for some reason (can be fixed by running >> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) >> are unfolded for no apparent reason sometimes. A blind guess is that it >> is something to do with lack of 'isearch-open-invisible, which I am not >> sure how to set via text properties. > > You cannot. You may however mimic it with `cursor-sensor-functions' text > property. These assume Cursor Sensor minor mode is active, tho. > I haven't tested it, but I assume it would slow down text properties > a bit, too, but hopefully not as much as overlays. > > Note there are clear advantages using text properties. For example, when > you move contents around, text properties are preserved. So there's no > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > necessary anymore to re-hide drawers. > >> Any thoughts about the use of text properties or about the patch >> suggestion are welcome. > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth > experimenting with `cursor-sensor-functions'. > > We could also use text properties for property drawers, and overlays for > regular ones. This might give us a reasonable speed-up with an > acceptable feature trade-off. > > Anyway, the real fix should come from Emacs itself. There are ways to > make overlays faster. These ways have already been discussed on the > Emacs devel mailing list, but no one implemented them. It is a bit sad > that we have to find workarounds for that. > > Regards, > > -- > Nicolas Goaziou -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg