From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id YOF0IX22s17nUwAA0tVLHw (envelope-from ) for ; Thu, 07 May 2020 07:19:25 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id mMtmJom2s14WWQAA1q6Kng (envelope-from ) for ; Thu, 07 May 2020 07:19:37 +0000 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:470:142::17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 71612940510 for ; Thu, 7 May 2020 07:19:35 +0000 (UTC) Received: from localhost ([::1]:51960 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWaoh-0006LE-3f for larch@yhetil.org; Thu, 07 May 2020 03:19:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33436) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWany-0006Jf-39 for emacs-orgmode@gnu.org; Thu, 07 May 2020 03:18:50 -0400 Received: from ciao.gmane.io ([159.69.161.202]:54672) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWanw-0006lh-82 for emacs-orgmode@gnu.org; Thu, 07 May 2020 03:18:49 -0400 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1jWant-0007BP-B0 for emacs-orgmode@gnu.org; Thu, 07 May 2020 09:18:45 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Karl Voit Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Date: Thu, 7 May 2020 09:18:38 +0200 Organization: www.karl-voit.at Message-ID: <2020-05-07T09-08-12@devnull.Karl-Voit.at> References: <87h7x9e5jo.fsf@localhost> <875zdpia5i.fsf@nicolasgoaziou.fr> <87y2qi8c8w.fsf@localhost> <2020-05-04T18-47-10@devnull.Karl-Voit.at> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit User-Agent: slrn/pre1.0.0-18 (Linux) X-GPG-Key: http://www.Karl-Voit.at/Karl_Voit_GnuPG_public_key.gpg X-Registered-Linux-User: 224337 X-Confession: Pastafarian http://www.venganza.org/ Received-SPF: pass client-ip=159.69.161.202; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/07 03:18:45 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Karl Voit Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 X-Spam-Score: -0.51 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Scan-Result: default: False [-0.51 / 13.00]; HAS_REPLYTO(0.00)[news1142@Karl-Voit.at]; GENERIC_REPUTATION(0.00)[-0.49754612295356]; DWL_DNSWL_BLOCKED(0.00)[2001:470:142::17:from]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:142::/48:c]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; TO_DN_NONE(0.00)[]; MV_CASE(0.50)[]; HAS_ORG_HEADER(0.00)[]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.11), country: US(-0.00), ip: 2001:470:142::17(-0.50)]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:2001:470:142::/48, country:US]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; FROM_NEQ_ENVFROM(0.00)[devnull@Karl-Voit.at,emacs-orgmode-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[gnu.org:url,reddit.com:url,karl-voit.at:email]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[Karl-Voit.at]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: Perf8srbs8YW Hi, * Karl Voit wrote: > Hi Ihor, > > * Ihor Radchenko wrote: >> >> So far, I came up with the following partial solution searching and >> showing hidden text. >> >> (defun org-find-text-property-region (pos prop) >> (define-advice isearch-search-string (:after (&rest _) put-overlay) >> (defun org-restore-invisibility-specs (&rest _) >> (add-hook 'post-command-hook #'org-restore-invisibility-specs) >> (defun org-flag-region (from to flag spec) >> (defun org-unfontify-region (beg end &optional _maybe_loudly) > > After a couple of hours working with these patches, my feedback is > very positive. Besides some visual glitches when creating a new > heading with org-expiry-insinuate activated (which automatically > adds :CREATED: properties), I could not detect any side-effect so > far (will keep testing). > > The visual glitch looks like that: > >:PROPERTIES:X:CREATED: [2020-05-04 Mon 18>54] > X > > ... with "X" being my character that symbolizes collapsed content. > The way it looked without the patch was a simple collapsed property > drawer. Here some hard numbers to demonstrate the impact: my-org-agenda: from 11-16s down to 10 -> not much of a difference helm-org-contacts-refresh-cache: 29-59s down to 2½ -> HUGE Emacs boot time: 50-65s down to 10 -> HUGE Navigating the cursor in large Org files -> HUGE subjective impact >>> Anyway, the real fix should come from Emacs itself. There are ways to >>> make overlays faster. These ways have already been discussed on the >>> Emacs devel mailing list, but no one implemented them. It is a bit sad >>> that we have to find workarounds for that. >> >> I guess that it is a very old story starting from the times when XEmacs >> was a thing [1]. I recently heard about binary tree implementation of >> overlays (there should be a branch in emacs git repo) [2], but there was >> no update on that branch for a while. So, I do not have much hope on >> Emacs implementing efficient overlay access in the near future. (And I >> have problems with huge org files already). > > I can not express how this also reflects my personal situation. > >> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/ >> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html > > [3] https://karl-voit.at/2020/05/03/current-org-files > -- get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode: > get Memacs from https://github.com/novoid/Memacs < Personal Information Management > http://Karl-Voit.at/tags/pim/ Emacs-related > http://Karl-Voit.at/tags/emacs/