From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id OOKkGHiOol7FBwAA0tVLHw (envelope-from ) for ; Fri, 24 Apr 2020 07:00:08 +0000 Received: from aspmx2.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id IHO5MH6Ool4+JgAA1q6Kng (envelope-from ) for ; Fri, 24 Apr 2020 07:00:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx2.migadu.com (Postfix) with ESMTPS id 61741680609 for ; Fri, 24 Apr 2020 07:00:10 +0000 (UTC) Received: from localhost ([::1]:52418 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jRsJi-0002ok-KM for larch@yhetil.org; Fri, 24 Apr 2020 03:00:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55794) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jRsJG-0002o6-OM for emacs-orgmode@gnu.org; Fri, 24 Apr 2020 02:59:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jRsJF-0005iP-Dr for emacs-orgmode@gnu.org; Fri, 24 Apr 2020 02:59:38 -0400 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]:43918) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jRsJE-0005fl-VK for emacs-orgmode@gnu.org; Fri, 24 Apr 2020 02:59:37 -0400 Received: by mail-pg1-x52b.google.com with SMTP id x26so4197059pgc.10 for ; Thu, 23 Apr 2020 23:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=W8NcXMJXT05EsaIyFmdNWuk3DSuGK3Ti82i+ynWH+QI=; b=I6vZRp9ukC+y8P0SHUPyxhiG2snzK4j+CAE5jvF/awDJAZQ+vBRiNfXvXcd9MI6Fu9 tfN1H+n4sip4MvLBiETzrTa2zsBOnXj7WcSLTZcRK2o38yHgJtOP63+QLKBrOxf8qqul wJsW1IHr1vRQxbCo+tDcjpZJO4j3pQBVLXYdFUZPnij/siBpf5KAJ5593H8wU4APgprP emiCZmku8Ti0rcH908mGOCjkMCFe0/t63Fv+hmAqzebhpBBeSooTEpAsmvNWKOOb4hEq KjiqSKsDLt57PKl2NIoXM5OdxUrfyMu/v7SAv+sqCK2vRHqj78q6mpxX8rXYCPBVGVDP wuig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=W8NcXMJXT05EsaIyFmdNWuk3DSuGK3Ti82i+ynWH+QI=; b=gAaGRL2NI9SUu3slmAH1V4yI1oFamYDgR4wS9rV3tk5QQ3jcQV2obPY+rLWTWrz+hN KN02qkJYuzuhlISzwRAl1fzWE3fVk0JB3ULQDndEUaQ3C30aga4tSpV8OyJUfucbZCLd vo3jD8MeRLVhHGzNHmdm09oPzQDqWSu92nAKxMQ+6AfMhP3DOH61SRhKltn4Q01p23za d9hkj7kQadJgL178HI/GHvuhz/YnkO8QfSXC9gCCMkrYbArT96Dv6o32FYWN3gGtknxg QcNjSXvzPwbt54Z160AMLvezt/yEAYbRYYJ58OBXOMYY8xi7erbvYXvNCoQYam3UlJjh P/sQ== X-Gm-Message-State: AGi0PuaEv/KtR2q9wjzdZ0nosbxASGi2DE5tLcapND4z2kNzcm4Sn6Sy 539yCG/UpB9IlAt7JQIksHZO0y6C2v4= X-Google-Smtp-Source: APiQypL6uMtHxYLTV57mN6sibSOtjFuE0TWyhYLLI3WWMrxQ0e0HWiXv6xqYqucnjIcJiu1jwy4dNA== X-Received: by 2002:aa7:9093:: with SMTP id i19mr7722485pfa.152.1587711575175; Thu, 23 Apr 2020 23:59:35 -0700 (PDT) Received: from localhost ([210.3.160.222]) by smtp.gmail.com with ESMTPSA id 14sm4252965pjd.36.2020.04.23.23.59.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 23:59:34 -0700 (PDT) From: Ihor Radchenko To: emacs-orgmode@gnu.org Subject: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Date: Fri, 24 Apr 2020 14:55:39 +0800 Message-ID: <87h7x9e5jo.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::52b; envelope-from=yantar92@gmail.com; helo=mail-pg1-x52b.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::52b X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 X-Spam-Score: -1.31 Authentication-Results: aspmx2.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=I6vZRp9u; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx2.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Scan-Result: default: False [-1.31 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.56159538794114]; R_SPF_ALLOW(-0.20)[+ip4:209.51.188.0/24:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.21), country: US(-0.00), ip: 209.51.188.17(-0.56)]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[209.51.188.17:from]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; TAGGED_FROM(0.00)[larch=yhetil.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_NEQ_ENVFROM(0.00)[yantar92@gmail.com,emacs-orgmode-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[reddit.com:url,sutd.edu.sg:email,gnu.org:url]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[emacs-orgmode@gnu.org]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_SEVEN(0.00)[7]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: DhACezKCnWlt Emacs becomes very slow when opening and moving around huge org files with many drawers. I have reported this issue last year in bug-gnu-emacs [1] and there have been other reports on the same problem in the internet [2]. You can easily see this problem using the attached file if you try to move down the lines when all the headings are folded. Moving a single line down may take over 10 seconds in the file. According to the reply to my initial emacs bug report [1], the reasons of performance degradation is huge number of overlays created by org in the PROPERTY and LOGBOOK drawers. Emacs must loop over all those overlays every time it calculates where the next visible line is located. So, one way to improve the performance would be reducing the number of overlays. I have been looking into usage of overlays in the org-mode code recently and tried to redefine org-flag-region to use text properties instead of overlays: #+begin_src emacs-lisp (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." (pcase spec ;; outlines must still use overlays because they rely on ;; 'reveal-toggle-invisible feature from reveal.el ;; That only works for overlays ('outline (remove-overlays from to 'invisible spec) ;; Use `front-advance' since text right before to the beginning of ;; the overlay belongs to the visible line than to the contents. (when flag (let ((o (make-overlay from to nil 'front-advance))) (overlay-put o 'evaporate t) (overlay-put o 'invisible spec) (overlay-put o 'isearch-open-invisible #'delete-overlay)))) (_ (let ((inhibit-modification-hooks t)) (remove-text-properties from to '(invisible nil)) ;; Use `front-advance' since text right before to the beginning of ;; the overlay belongs to the visible line than to the contents. (when flag (put-text-property from to 'rear-non-sticky t) (put-text-property from to 'front-sticky t) (put-text-property from to 'invisible spec) ;; no idea if 'isearch-open-invisible is needed for text ;; properties ;; (overlay-put o 'isearch-open-invisible #'delete-overlay) ))))) #+end_src To my surprise, the patch did not break org to unusable state and the performance on the sample org file [3] improved drastically. You can try by yourself! However, this did introduce some visual glitches with drawer display. Though drawers can still be folded/unfolded with , they are not folded on org-mode startup for some reason (can be fixed by running (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers) are unfolded for no apparent reason sometimes. A blind guess is that it is something to do with lack of 'isearch-open-invisible, which I am not sure how to set via text properties. Any thoughts about the use of text properties or about the patch suggestion are welcome. Best, Ihor [1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/msg01387.html [2] https://www.reddit.com/r/orgmode/comments/e9p84n/scaling_org_better_to_use_more_medsize_files_or/ [3] See the attached org file in my Emacs bug report: https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/txte6kQp35VOm.txt -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg