From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id aD/eJKkMV2AsFgAA0tVLHw (envelope-from ) for ; Sun, 21 Mar 2021 09:06:49 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id SJCdIKkMV2A5fAAAB5/wlQ (envelope-from ) for ; Sun, 21 Mar 2021 09:06:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8EF3222886 for ; Sun, 21 Mar 2021 10:06:48 +0100 (CET) Received: from localhost ([::1]:58338 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lNu2p-0003gK-HW for larch@yhetil.org; Sun, 21 Mar 2021 05:06:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43168) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lNu26-0003g0-Lf for emacs-orgmode@gnu.org; Sun, 21 Mar 2021 05:06:02 -0400 Received: from mail-ot1-x330.google.com ([2607:f8b0:4864:20::330]:33596) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lNu24-0008Hx-Ly; Sun, 21 Mar 2021 05:06:02 -0400 Received: by mail-ot1-x330.google.com with SMTP id o19-20020a9d22130000b02901bfa5b79e18so12904634ota.0; Sun, 21 Mar 2021 02:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=ZSHQ6aO29yBsaEk/XtfO4o0vtqHH+sT5nLwm7rz1xNQ=; b=hwSdREfSEFRi/nhz6umRCvXfo8Oq6iFeznM4rxobNCE8ImtZQ2HMrlWE88yutOY3H9 1ZMx9G/BZHjQIbE691Q4q90cgNRGaqGiYXcsq4TGZcJZ19HZLxZNU5AH9sceM7h6WbIj jVSfOye9n0ckMsfOchlLwRjXgkMeY7sEG9S/zrJNVnfBoFXVKpGklFwdh37rpquH6CXl 95IN5MfIxm5ZT2UZ2exX7bWh2WoNEib+rCrxHvyTmX98qcJew0MomEYy2myJuJpd6GzV 3YXe9NCQmi1N/kfE4NEBMGxV6burUamv2XKakeEeHfggAxuIzybCs1UQxvCLs8M1lzNL K17Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=ZSHQ6aO29yBsaEk/XtfO4o0vtqHH+sT5nLwm7rz1xNQ=; b=R9pqjLUfQoP0lP/fOuamq4WPxjSrvjr0TjCA3+k9ifIQpgsu6LetsUCNBW2y7PgvFt zO2rUrathlBHlinkogr6MvbJm/XsFwFIGcMKLj9r1GEogQ6efN7xEakU5HX0CRZU3TI3 2AwB6CjAkD9JllnO3f23f0wnzWSmzmQq3BfD1RbfulLI/SA77PoOxNYeeTr7aBN6NrLb RGlnnuA855nPZP1TEfK9CXS28k9rsUOgRo6S7ZHxjBW/rvAGfLLNQE/Zd/NncRlKTyFP lLfb4TrZ2SChiLE3cgOBzw0AVUGSLF8Mk4nFwy7hGExKHR0kDqwfvNUbljsS3DWz8zG2 tHbg== X-Gm-Message-State: AOAM531A7IKhxNn/f3y36hwolCWxVdN6DQ6VRKWtSMjNafBIEGsJv2Im XnoKKwo6b4ri4Bs1IUnlsXQ= X-Google-Smtp-Source: ABdhPJzCofygPMwvxuALewC9Wyg1we7IfiB5h9eLe45/r0wWYP9PGWtCMvcFJm3KBLp0AkGBJOXL1Q== X-Received: by 2002:a9d:748a:: with SMTP id t10mr7752288otk.198.1616317558265; Sun, 21 Mar 2021 02:05:58 -0700 (PDT) Received: from localhost ([192.200.115.226]) by smtp.gmail.com with ESMTPSA id g6sm2556987otp.68.2021.03.21.02.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Mar 2021 02:05:56 -0700 (PDT) From: Ihor Radchenko To: Kyle Meyer , Nicolas Goaziou , Karl Voit , Christian Heinrich , Bastien Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers In-Reply-To: <87v9di3yee.fsf@localhost> References: <87h7x9e5jo.fsf@localhost> <87imh34usq.fsf@localhost> <87pnbby49m.fsf@nicolasgoaziou.fr> <87tv0efvyd.fsf@localhost> <874kse1seu.fsf@localhost> <87r1vhqpja.fsf@nicolasgoaziou.fr> <87tv0d2nk7.fsf@localhost> <87o8qkhy3g.fsf@nicolasgoaziou.fr> <87sgfqu5av.fsf@localhost> <87sgfn6qpc.fsf@nicolasgoaziou.fr> <87367d4ydc.fsf@localhost> <87r1uuotw8.fsf@nicolasgoaziou.fr> <87mu5iq618.fsf@localhost> <87ftb9pqop.fsf@nicolasgoaziou.fr> <875zc2du63.fsf@localhost> <87wo4en8qk.fsf@nicolasgoaziou.fr> <87mu31adeu.fsf@localhost> <871rkcbx40.fsf@kyleam.com> <87mu301in8.fsf@localhost> <87lfh5vvrp.fsf@localhost> <87v9di3yee.fsf@localhost> Date: Sun, 21 Mar 2021 17:09:58 +0800 Message-ID: <87a6qwdg9l.fsf@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::330; envelope-from=yantar92@gmail.com; helo=mail-ot1-x330.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode@gnu.org Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616317609; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ZSHQ6aO29yBsaEk/XtfO4o0vtqHH+sT5nLwm7rz1xNQ=; b=T59zv6OUXkxCLgU0HeXQkChfISgEsAU6QGXR1e+SEvtsXnof637I8KyK/AQJh6iMtTDw6S yqX4//H7EQzZQTkBaLcK2kmdSdcbw397JIOesS2wopsoXQMfF3C+idEsR2oaCg7aC+tQ7A TtTODzRmVhEJmaBrT7lENmNk8id3l+VaAG/pry0fsTW+G4IvN8LmiKySd30/PAr+gATyy7 oatsY4Gvy6rGOAoa31sjrBKiC7T4ZzEubvAR9yCka3s9E05q2pPMDkTUGPBLip9/uGm7KE 2HWumVawymFUV6Bjamun5RzS5ca9PRfjnCy632F1tdCtPlmakBhj6fu4JEeQpQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616317609; a=rsa-sha256; cv=none; b=aBurWQrKIv6Q3tZv3iAKnfPKIQ0qXLBUezn65A1R1GsmHrVUJt+tsTOimm2LVCTujL0SZ/ JUFfnNKYdv6LovgZmucWxNEolHUIiKXn0NIeKyrcjRodvZIh+skh1cRtZLnhMoc1Hzu/Q1 OYkkNlVeaRN14M13LzmMR+HqHWFx/hvcYkbp4i2MgK3e1n2ht2ELLXGd+RgIerU5URa3m6 nto2+W67knH8xWG/kjROOTtp88PrVZcf1HCnotNjSLnkXi8Qcti5mJIkEVVWQ0O0W+LgDB UBUWJTgWYagalfD7XUmPZKi6tjOsi59NYMGSkrAwna3UW1kLyw3vU/a2CFnlpg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=hwSdREfS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -2.52 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=hwSdREfS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 8EF3222886 X-Spam-Score: -2.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: MlbbtJEUkHQw --=-=-= Content-Type: text/plain Hello, This is another update about the status of the patch. I am mostly happy with the current state of the code, got rid of most of the bugs, and did not get any new bug reports in github for a while. I would like to start the process of applying the patch on master. As a first step, I would like to submit the core folding library (org-fold-core) for review. org-fold-core is pretty much independent from org-mode code base and does not affect anything if applied as is. It will be used by org-specific org-fold.el I will finalise and send later. For now, I would like to hear any suggestions about API and implementation of org-fold-core.el. I tried to document all the details in the code. Looking forward for the feedback. Best, Ihor --=-=-= Content-Type: application/emacs-lisp Content-Disposition: inline; filename=org-fold-core.el Content-Transfer-Encoding: quoted-printable ;;; org-fold-core.el --- Folding buffer text -*- lexical-binding: t; -*- ;; ;; Copyright (C) 2020-2020 Free Software Foundation, Inc. ;; ;; Author: Ihor Radchenko ;; Keywords: folding, invisible text ;; Homepage: https://orgmode.org ;; ;; This file is part of GNU Emacs. ;; ;; GNU Emacs is free software: you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; GNU Emacs is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs. If not, see . ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;= ;; ;; ;;; Commentary: ;; This file contains library to control temporary invisibility ;; (folding and unfolding) of text in buffers. ;; The file implements the following functionality: ;; ;; - Folding/unfolding regions of text ;; - Searching and examining boundaries of folded text ;; - Interactive searching in folded text (via isearch) ;; - Handling edits in folded text ;; - Killing/yanking (copying/pasting) of the folded text ;; - Fontification of the folded text ;; To setup folding in an arbitrary buffer, one must call ;; `org-fold-core-initialize', optionally providing the list of folding spe= cs to be ;; used in the buffer. The specs can be added, removed, or ;; re-configured later. Read below for more details. ;;; Folding/unfolding regions of text ;; User can temporarily hide/reveal (fold/unfold) arbitrary regions or ;; text. The folds can be nested. ;; Internally, nested folds are marked with different folding specs ;; Overlapping folds marked with the same folding spec are ;; automatically merged, while folds with different folding specs can ;; coexist and be folded/unfolded independently. ;; When multiple folding specs are applied to the same region of text, ;; text visibility is decided according to the folding spec with ;; topmost priority. ;; By default, we define two types of folding specs: ;; - 'org-fold-visible :: the folded text is not hidden ;; - 'org-fold-hidden :: the folded text is completely hidden ;; ;; The 'org-fold-visible spec has highest priority allowing parts of ;; text folded with 'org-fold-hidden to be shown unconditionally. ;; Consider the following Org mode link: ;; [[file:/path/to/file/file.ext][description]] ;; Only the word "description" is normally visible in this link. ;;=20 ;; The way this partial visibility is achieved is combining the two ;; folding specs. The whole link is folded using 'org-fold-hidden ;; folding spec, but the visible part is additionally folded using ;; 'org-fold-visible: ;; ;; [[file:/path/to/file/file.ext][description]] ;;=20 ;; Because 'org-fold-visible has higher priority than ;; 'org-fold-hidden, it suppresses the 'org-fold-hidden effect and ;; thus reveals the description part of the link. ;; Similar to 'org-fold-visible, display of any arbitrary folding spec ;; can be configured using folding spec properties. In particular, ;; `:visible' folding spec proprety controls whether the folded text ;; is visible or not. If the `:visible' folding spec property is nil, ;; folded text is hidden or displayed as a constant string (ellipsis) ;; according to the value of `:ellipsis' folding spec property. See ;; docstring of `org-fold-core--specs' for the description of all the avail= able ;; folding spec properties. ;; Folding spec properties of any valid folding spec can be changed ;; any time using `org-fold-core-set-folding-spec-property'. ;; If necessary, one can add or remove folding specs using ;; `org-fold-core-add-folding-spec' and `org-fold-core-remove-folding-spec'. ;; If a buffer initialised with `org-fold-core-initialize' is cloned into i= ndirect ;; buffers, it's folding state is copied to that indirect buffer. ;; The folding states are independent. ;; When working with indirect buffers that are handled by this ;; library, one has to keep in mind that folding state is preserved on ;; copy when using non-interactive functions. Moreover, the folding ;; states of all the indirect buffers will be copied together. ;; ;; Example of the implications: ;; Consider a base buffer and indirect buffer with the following state: ;; ----- base buffer -------- ;; * Heading ;; Some text folded in the base buffer, but unfolded in the indirect buffer= ;; * Other heading ;; Heading unfolded in both the buffers. ;; --------------------------- ;; ------ indirect buffer ---- ;; * Heading ;; Some text folded in the base buffer, but unfolded in the indirect buffer ;; * Other heading ;; Heading unfolded in both the buffers. ;; ---------------------------- ;; If some Elisp code copies the whole "Heading" from the indirect ;; buffer with `buffer-substring' or match data and inserts it into ;; the base buffer, the inserted heading will be folded since the ;; internal setting for the folding state is shared between the base ;; and indirect buffers. It's just that the indirect buffer ignores ;; the base buffer folding settings. However, as soon as the text is ;; copied back to the base buffer, the folding state will become ;; respected again. ;; If the described situation is undesired, Elisp code can use ;; `filter-buffer-substring' instead of `buffer-substring'. All the ;; folding states that do not belong to the currently active buffer ;; will be cleared in the copied text then. See ;; `org-fold-core--buffer-substring-filter' for more details. ;; Because of details of implementation of the folding, it is also not ;; recommended to set text visibility in buffer directly by setting ;; `invisible' text property to anything other than t. While this ;; should usually work just fine, normal folding can be broken if one ;; sets `invisible' text property to a value not listed in ;; `buffer-invisibility-spec'. ;;; Searching and examining boundaries of folded text ;; It is possible to examine folding specs (there may be several) of ;; text at point or search for regions with the same folding spec. ;; See functions defined under ";;;; Searching and examining folded ;; text" below for details. ;; All the folding specs can be specified by symbol representing their ;; name. However, this is not always convenient, especially if the ;; same spec can be used for fold different syntaxical structures. ;; Any folding spec can be additionally referenced by a symbol listed ;; in the spec's `:alias' folding spec property. For example, Org ;; mode's `org-fold-outline' folding spec can be referened as any ;; symbol from the following list: '(headline heading outline ;; inlinetask plain-list) The list is the value of the spec's `:alias' ;; property. ;; Most of the functions defined below that require a folding spec ;; symbol as their argument, can also accept any symbol from the ;; `:alias' spec property to reference that folding spec. ;; If one wants to search invisible text without using the provided ;; functions, it is important to keep in mind that 'invisible text ;; property may have multiple possible values (not just nil and ;; t). Hence, (next-single-char-property-change pos 'invisible) is not ;; guarantied to return the boundary of invisible/visible text. ;;; Interactive searching inside folded text (via isearch) ;; The library provides a way to control if the folded text can be ;; searchable using isearch. If the text is searchable, it is also ;; possible to control to unfold it temporarily during interactive ;; isearch session. ;; The isearch behaviour is controlled on per-folding-spec basis by ;; setting `isearch-open' and `isearch-ignore' folding spec ;; properties. The the docstring of `org-fold-core--specs' for more detail= s. ;;; Handling edits inside folded text ;; The visibility of the text inserted in front, rear, or in the ;; middle of a folded region is managed according to `:front-sticky' ;; and `:rear-sticky' folding properties of the corresponding folding ;; spec. The rules are the same with stickyness of text properties in ;; Elisp. ;; If a text being inserted into the buffer is already folded and ;; invisible (before applying the stickyness rules), then it is ;; revealed. This behaviour can be changed by wrapping the insertion ;; code into `org-fold-core-ignore-modifications' macro. The macro will di= sable ;; all the processing related to buffer modifications. ;; The library also provides a way to unfold the text after some ;; destructive changes breaking syntaxical structure of the buffer. ;; For example, Org mode automatically reveals folded drawers when the ;; drawer becomes syntaxically incorrect: ;; ------- before modification ------- ;; :DRAWER: ;; Some folded text inside drawer ;; :END: ;; ----------------------------------- ;; If the ":END:" is edited, drawer syntax is not correct anymore and ;; the folded text is automatically unfolded. ;; ------- after modification -------- ;; :DRAWER: ;; Some folded text inside drawer ;; :EN: ;; ----------------------------------- ;; The described automatic unfolding is controlled by `:fragile' ;; folding spec property. It's value can be a function checking if ;; changes inside (or around) the fold should drigger the unfold. By ;; default, only changes that directly involve folded regions will ;; trigger the check. In addition, `org-fold-core-extend-changed-region-fu= nctions' ;; can be set to extend the checks to all folded regions intersecting ;; with the region returned by the functions listed in the variable. ;; The fragility checks can be bypassed if the code doing ;; modifications is wrapped into `org-fold-core-ignore-fragility-checks' ma= cro. ;;; Fontification of the folded text ;; When working with huge buffers, `font-lock' may take a lot of time ;; to fontify all the buffer text during startup. This library ;; provides a way to delay fontification of initially folded text to ;; the time when the text is unfolded. The fontification is ;; controlled on per-folding-spec basis according to `:font-lock-skip' ;; fonlding spec property. ;; This library replaces `font-lock-fontify-region-function' to implement t= he ;; delayed fontification. However, it only does so when ;; `font-lock-fontify-region-function' is not modified at the initialisation ;; time. If one needs to use both delayed fontification and custom ;; `font-lock-fontify-region-function', it is recommended to consult the ;; source code of `org-fold-core-fontify-region'. ;;; Performance considerations ;; This library is using text properties to hide text. Text ;; properties are much faster than overlays, that could be used for ;; the same purpose. Overlays are implemented with O(n) complexity in ;; Emacs (2021-03-11). It means that any attempt to move through ;; hidden text in a file with many invisible overlays will require ;; time scaling with the number of folded regions (the problem ;; Overlays note of the manual warns about). For curious, historical ;; reasons why overlays are not efficient can be found in ;; https://www.jwz.org/doc/lemacs.html. ;; Despite using text properties, the performance is still limited by ;; Emacs display engine. For example, >7Mb of text hidden within ;; visible part of a buffer may cause noticeable lags (which is still ;; orders of magnitude better in comparison with overlays). If the ;; performance issues become critical while using this library, it is ;; recommended to minimise the number of folding specs used in the ;; same buffer at a time. ;; Alternatively, the library provides `org-fold-core--optimise-for-huge-bu= ffers' ;; to provide additional speedup. This can be used as file-local ;; variable in huge buffers. This optimisation may break external ;; packages though. When using this optimisation, folded regions ;; copied to other buffers (including buffers that do not use this ;; library) will remain invisible. org-fold-core provides functions ;; to work around this issue: `org-fold-core-remove-optimisation' and ;; `org-fold-core-update-optimisation', but it is unlikely that a random ex= ternal ;; package will use them. ;; Another possible bottleneck is the fragility check after changed ;; related to the folded text. The functions used in `:fragile' ;; folding properties must be optimised. Also, ;; `org-fold-core-ignore-fragility-checks' or even `org-fold-core-ignore-mo= difications' may be ;; used when appropriate in the performance-critical code. ;; Performance of isearch is currently limited by Emacs isearch ;; implementation. For now, Emacs isearch only supports searching ;; through text hidden using overlays. This library handles isearch ;; by converting folds with matching text to overlays, which may ;; affect performance in case of large number of mathes. In the ;; future, Emacs will hopefully accept the relevant patch allowing ;; isearch to work with text hidden via text properties, but the ;; performance hit has to be accepted meanwhile. ;;; Code: (require 'org-macs) (declare-function isearch-filter-visible "isearch" (beg end)) ;;; Customization (defvar-local org-fold-core-isearch-open-function #'org-fold-core--isearch-= reveal "Function used to reveal hidden text found by isearch. The function is called with a single argument - point where text is to be revealed.") (defvar-local org-fold-core--optimise-for-huge-buffers nil "Non-nil turns on extra speedup on huge buffers (Mbs of folded text). This setting is risky and may cause folded text to become invisible where it should not. It is recommended to enable it on per-buffer basis as file-local variable.") (put 'org-fold-core--optimise-for-huge-buffers 'safe-local-variable 'boolea= np) ;;; Core functionality ;;;; Folding specs (defvar-local org-fold-core--specs '((org-fold-visible (:visible . t) (:alias . (visible))) (org-fold-hidden (:ellipsis . "...") (:isearch-open . t) (:alias . (hidden)))) "Folding specs defined in current buffer. Each spec is a list (SPEC-SYMBOL SPEC-PROPERTIES). SPEC-SYMBOL is the symbol respresenting the folding spec. SPEC-PROPERTIES is an alist defining folding spec properties. If a text region is folded using multiple specs, only the folding spec listed earlier is used. The following properties are known: - :ellipsis :: must be nil or string to show when text is folded using this spec. - :isearch-ignore :: non-nil means that folded text is not searchable using isearch. - :isearch-open :: non-nil means that isearch can reveal text hidden using this spec. This property does nothing when 'isearch-ignore property is non-nil. - :front-sticky :: non-nil means that text prepended to the folded text is automatically folded. - :rear-sticky :: non-nil means that text appended to the folded text is folded. - :visible :: non-nil means that folding spec visibility is not managed. Instead, visibility settings in `buffer-invisibility-spec' will be used as is. Note that changing this property from nil to t may clear the setting in `buffer-invisibility-spec'. - :alias :: a list of aliases for the SPEC-SYMBOL. - :font-lock-skip :: Suppress font-locking in folded text. - :fragile :: Must be a function accepting two arguments. Non-nil means that changes in region may cause the region to be revealed. The region is revealed after changes if the function returns non-nil. The function called after changes are made with two arguments: cons (beg . end) representing the folded region and spec symbol.") (defvar-local org-fold-core-extend-changed-region-functions nil "Special hook run just before handling changes in buffer. This is used to account changes outside folded regions that still affect the folded region visibility. For example, removing all stars at the beginning of a folded Org mode heading should trigger the folded text to be revealed. Each function is called with two arguments: beginning and the end of the changed region.") ;;; Utility functions (defsubst org-fold-core-folding-spec-list (&optional buffer) "Return list of all the folding spec symbols in BUFFER." (with-current-buffer (or buffer (current-buffer)) (mapcar #'car org-fold-core--specs))) (defun org-fold-core-get-folding-spec-from-alias (spec-or-alias) "Return the folding spec symbol for SPEC-OR-ALIAS. Return nil when there is no matching folding spec." (and spec-or-alias (or (and (memq spec-or-alias (org-fold-core-folding-spec-list)) spec= -or-alias) (seq-some (lambda (spec) (and (memq spec-or-alias (alist-get :al= ias (alist-get spec org-fold-core--specs))) spec)) (org-fold-core-folding-s= pec-list))))) (defun org-fold-core-folding-spec-p (spec-or-alias) "Check if SPEC-OR-ALIAS is a registered folding spec." (org-fold-core-get-folding-spec-from-alias spec-or-alias)) (defun org-fold-core--check-spec (spec-or-alias) "Throw an error if SPEC-OR-ALIAS is not present in `org-fold-core--spec-p= riority-list'." (unless (org-fold-core-folding-spec-p spec-or-alias) (error "%s is not a valid folding spec" spec-or-alias))) (defun org-fold-core-get-folding-spec-property (spec-or-alias property) "Get PROPERTY of a folding SPEC-OR-ALIAS. Possible properties can be found in `org-fold-core--specs' docstring." (org-fold-core--check-spec spec-or-alias) (alist-get property (alist-get (org-fold-core-get-folding-spec-from-alias= spec-or-alias) org-fold-core--specs))) (defconst org-fold-core--spec-property-prefix "org-fold--spec-" "Prefix used to create property symbol.") (defsubst org-fold-core-get-folding-property-symbol (spec &optional buffer) "Get folding text property using to store SPEC in current buffer or BUFFE= R." (intern (format (concat org-fold-core--spec-property-prefix "%s-%S") (symbol-name spec) ;; (sxhash buf) appears to be not constant over time. ;; Using buffer-name is safe, since the only place where ;; buffer-local text property actually matters is an indi= rect ;; buffer, where the name cannot be same anyway. (sxhash (buffer-name (or buffer (current-buffer))))))) (defsubst org-fold-core-get-folding-spec-from-folding-prop (folding-prop) "Return folding spec symbol used for folding property with name FOLDING-P= ROP." (catch :exit (dolist (spec (org-fold-core-folding-spec-list)) ;; We know that folding properties have ;; folding spec in their name. (when (string-match-p (symbol-name spec) (symbol-name folding-prop)) (throw :exit spec))))) (defvar org-fold-core--property-symbol-cache (make-hash-table :test 'equal) "Saved values of folding properties for (buffer . spec) conses.") (defvar-local org-fold-core--indirect-buffers nil "List of indirect buffers created from current buffer. This variable is needed to work around Emacs bug#46982, while Emacs does not provide a way `after-change-functions' in any other buffer than the buffer where the change was actually made.") ;; This is the core function used to fold text in buffers. We use ;; text properties to hide folded text, however 'invisible property is ;; not directly used (unless risky `org-fold-core--optimise-for-huge-buffer= s' is ;; enabled). Instead, we define unique text property (folding ;; property) for every possible folding spec and add the resulting ;; text properties into `char-property-alias-alist', so that ;; 'invisible text property is automatically defined if any of the ;; folding properties is non-nil. This approach lets us maintain ;; multiple folds for the same text region - poor man's overlays (but ;; much faster). Additionally, folding properties are ensured to be ;; unique for different buffers (especially for indirect ;; buffers). This is done to allow different folding states in ;; indirect buffers. (defun org-fold-core--property-symbol-get-create (spec &optional buffer ret= urn-only) "Return a unique symbol suitable as folding text property. Return value is unique for folding SPEC in BUFFER. If the buffer already have buffer-local setup in `char-property-alias-alist' and the setup appears to be created for different buffer, copy the old invisibility state into new buffer-local text properties, unless RETURN-ONLY is non-nil." (org-fold-core--check-spec spec) (let* ((buf (or buffer (current-buffer)))) ;; Create unique property symbol for SPEC in BUFFER (let ((local-prop (or (gethash (cons buf spec) org-fold-core--property-= symbol-cache) (puthash (cons buf spec) (org-fold-core-get-folding-property-symb= ol spec buf) org-fold-core--property-symbol-cache)))) (prog1 local-prop (unless return-only (with-current-buffer buf ;; Update folding properties carried over from other ;; buffer (implying that current buffer is indirect ;; buffer). Normally, `char-property-alias-alist' in new ;; indirect buffer is a copy of the same variable from ;; the base buffer. Then, `char-property-alias-alist' ;; would contain folding properties, which are not ;; matching the generated `local-prop'. (unless (member local-prop (cdr (assq 'invisible char-property-alias-a= list))) ;; Add current buffer to the list of indirect buffers in the = base buffer. (when (buffer-base-buffer) (with-current-buffer (buffer-base-buffer) (setq-local org-fold-core--indirect-buffers (seq-filter #'buffer-live-p (delete-dups (app= end org-fold-core--indirect-buffers (list buf))))))) ;; Copy all the old folding properties to preserve the foldin= g state (with-silent-modifications (dolist (old-prop (cdr (assq 'invisible char-property-alias= -alist))) (org-with-wide-buffer (let* ((pos (point-min)) (spec (org-fold-core-get-folding-spec-from-folding-prop = old-prop)) ;; Generate new buffer-unique folding property (new-prop (org-fold-core--property-symbol-get-create spe= c nil 'return-only))) ;; Copy the visibility state for `spec' from `old-prop= ' to `new-prop' (while (< pos (point-max)) (let ((val (get-text-property pos old-prop)) (next (next-single-char-property-change pos ol= d-prop))) (when val (put-text-property pos next new-prop val)) (setq pos next)))))) ;; Update `char-property-alias-alist' with folding ;; properties unique for the current buffer. (setq-local char-property-alias-alist (cons (cons 'invisible (mapcar (lambda (spec) (org-fold-core--property-symbol-get-create spec nil '= return-only)) (org-fold-core-folding-spec-list))) (remove (assq 'invisible char-property-alias-alist) char-property-alias-alist))) ;; Do not let Emacs manage folding spec stickyness. ;; The default Emacs behavior make stickyness work ;; only for interfactive edits, which is when we do ;; not want folding state to be sticky anyway. (setq-local text-property-default-nonsticky (delete-dups (append text-property-default-nons= ticky (mapcar (lambda (spec) (cons (org-fold-= core--property-symbol-get-create spec nil 'return-only) t)) (org-fold-core-fol= ding-spec-list))))))))))))) (defun org-fold-core-decouple-indirect-buffer-folds () "Copy and decouple folding state in a newly created indirect buffer. This function is mostly indented to be used in `clone-indirect-buffer-hook'= ." (when (buffer-base-buffer) (org-fold-core--property-symbol-get-create (car (org-fold-core-folding-= spec-list))))) ;;; API ;;;; Modifying folding specs (defun org-fold-core-set-folding-spec-property (spec property value &option= al force) "Set PROPERTY of a folding SPEC to VALUE. Possible properties and values can be found in `org-fold-core--specs' docst= ring. Do not check previous value when FORCE is non-nil." (pcase property (:ellipsis (unless (and (not force) (equal value (org-fold-core-get-folding-spec-= property spec :ellipsis))) (remove-from-invisibility-spec (cons spec (org-fold-core-get-folding= -spec-property spec :ellipsis))) (unless (org-fold-core-get-folding-spec-property spec :visible) (add-to-invisibility-spec (cons spec value))))) (:visible (unless (and (not force) (equal value (org-fold-core-get-folding-spec-= property spec :visible))) (if value (remove-from-invisibility-spec (cons spec (org-fold-core-get-folding-sp= ec-property spec :ellipsis))) (add-to-invisibility-spec (cons spec (org-fold-core-get-folding-sp= ec-property spec :ellipsis)))))) (:alias nil) (:isearch-open nil) (:isearch-ignore nil) (:front-sticky nil) (:rear-sticky nil) (_ nil)) (setf (alist-get property (alist-get spec org-fold-core--specs)) value)) (defun org-fold-core-add-folding-spec (spec &optional properties buffer app= end) "Add a new folding SPEC with PROPERTIES in BUFFER. SPEC must be a symbol. BUFFER can be a buffer to set SPEC in or nil to set SPEC in current buffer. By default, the added SPEC will have highest priority among the previously defined specs. When optional APPEND argument is non-nil, SPEC will have the lowest priority instead. If SPEC was already defined earlier, it will be redefined according to provided optional arguments. ` The folding spec properties will be set to PROPERTIES (see `org-fold-core--specs' for details)." (when (eq spec 'all) (error "Cannot use reserved folding spec symbol 'all= ")) (with-current-buffer (or buffer (current-buffer)) (let* ((full-properties (mapcar (lambda (prop) (cons prop (alist-get pr= op properties))) '( :visible :ellipsis :isearch-ignore :isearch-open :front-sticky :rear-st= icky :fragile :alias :font-lock-skip))) (full-spec (cons spec full-properties))) (add-to-list 'org-fold-core--specs full-spec append) (mapc (lambda (prop-cons) (org-fold-core-set-folding-spec-property sp= ec (car prop-cons) (cdr prop-cons) 'force)) full-properties)))) (defun org-fold-core-remove-folding-spec (spec &optional buffer) "Remove a folding SPEC in BUFFER. SPEC must be a symbol. BUFFER can be a buffer to remove SPEC in, nil to remove SPEC in current buf= fer, or 'all to remove SPEC in all open `org-mode' buffers and all future org bu= ffers." (org-fold-core--check-spec spec) (when (eq buffer 'all) (setq-default org-fold-core--specs (delete (alist-get spec org-fold-cor= e--specs) org-fold-core--specs)) (mapc (lambda (buf) (org-fold-core-remove-folding-spec spec buf)) (buffer-list))) (let ((buffer (or buffer (current-buffer)))) (with-current-buffer buffer (org-fold-core-set-folding-spec-property spec :visible t) (setq org-fold-core--specs (delete (alist-get spec org-fold-core--spe= cs) org-fold-core--specs))))) (defun org-fold-core-initialize (&optional specs) "Setup folding in current buffer using SPECS as value of `org-fold-core--= specs'." ;; Preserve the priorities. (hack-local-variables) (when specs (setq specs (nreverse specs))) (unless specs (setq specs org-fold-core--specs)) (setq org-fold-core--specs nil) (dolist (spec specs) (org-fold-core-add-folding-spec (car spec) (cdr spec))) (add-hook 'after-change-functions 'org-fold-core--fix-folded-region nil '= local) (add-hook 'clone-indirect-buffer-hook #'org-fold-core-decouple-indirect-b= uffer-folds nil 'local) ;; Optimise buffer fontification to not fontify folded text. (when (eq font-lock-fontify-region-function #'font-lock-default-fontify-r= egion) (setq-local font-lock-fontify-region-function 'org-fold-core-fontify-re= gion)) ;; Setup killing text (setq-local filter-buffer-substring-function #'org-fold-core--buffer-subs= tring-filter) (require 'isearch) (if (boundp 'isearch-opened-regions) ;; Use new implementation of isearch allowing to search inside text ;; hidden via text properties. (org-fold-core--isearch-setup 'text-properties) (org-fold-core--isearch-setup 'overlays))) ;;;; Searching and examining folded text (defun org-fold-core-folded-p (&optional pos spec-or-alias) "Non-nil if the character after POS is folded. If POS is nil, use `point' instead. If SPEC-OR-ALIAS is a folding spec, only check the given folding spec." (org-fold-core-get-folding-spec spec-or-alias pos)) (defun org-fold-core-region-folded-p (beg end &optional spec-or-alias) "Non-nil if the region between BEG and END is folded. If SPEC-OR-ALIAS is a folding spec, only check the given folding spec." (org-with-point-at beg (catch :visible (while (< (point) end) (unless (org-fold-core-get-folding-spec spec-or-alias) (throw :visi= ble nil)) (goto-char (org-fold-core-next-folding-state-change spec-or-alias n= il end))) t))) (defun org-fold-core-get-folding-spec (&optional spec-or-alias pom) "Get folding state at `point' or POM. Return nil if there is no folding at point or POM. If SPEC-OR-ALIAS is nil, return a folding spec with highest priority among present at `point' or POM. If SPEC-OR-ALIAS is 'all, return the list of all present folding specs. If SPEC-OR-ALIAS is a valid folding spec or a spec alias, return the corresponding folding spec (if the text is folded using that spec)." (let ((spec (if (eq spec-or-alias 'all) 'all (org-fold-core-get-folding-spec-from-alias spec-or-alias)))) (when (and spec (not (eq spec 'all))) (org-fold-core--check-spec spec)) (org-with-point-at pom (cond ((eq spec 'all) (let ((result)) (dolist (spec (org-fold-core-folding-spec-list)) (let ((val (get-char-property (point) (org-fold-core--property-symbol-= get-create spec nil t)))) (when val (push val result)))) (reverse result))) ((null spec) (let ((result (get-text-property (point) 'invisible))) (when (org-fold-core-folding-spec-p result) result))) (t (get-char-property (point) (org-fold-core--property-symbol-get-cr= eate spec nil t))))))) (defun org-fold-core-get-folding-specs-in-region (beg end) "Get all folding specs in region from BEG to END." (let ((pos beg) all-specs) (while (< pos end) (setq all-specs (append all-specs (org-fold-core-get-folding-spec nil= pos))) (setq pos (org-fold-core-next-folding-state-change nil pos end))) (unless (listp all-specs) (setq all-specs (list all-specs))) (delete-dups all-specs))) (defun org-fold-core-get-region-at-point (&optional spec-or-alias pom) "Return region folded using SPEC-OR-ALIAS at POM. If SPEC is nil, return the largest possible folded region. The return value is a cons of beginning and the end of the region. Return nil when no fold is present at point of POM." (let ((spec (org-fold-core-get-folding-spec-from-alias spec-or-alias))) (org-with-point-at (or pom (point)) (if spec (org-find-text-property-region (point) (org-fold-core--property-symbol-g= et-create spec nil t)) (let ((region (cons (point) (point)))) (dolist (spec (org-fold-core-get-folding-spec 'all)) (let ((local-region (org-fold-core-get-region-at-point spec))) (when (< (car local-region) (car region)) (setcar region (car local-region))) (when (> (cdr local-region) (cdr region)) (setcdr region (cdr local-region))))) (unless (eq (car region) (cdr region)) region)))))) (defun org-fold-core-next-visibility-change (&optional pos limit ignore-hid= den-p previous-p) "Return next point from POS up to LIMIT where text becomes visible/invisi= ble. By default, text hidden by any means (i.e. not only by folding, but also via fontification) will be considered. If IGNORE-HIDDEN-P is non-nil, consider only folded text. If PREVIOUS-P is non-nil, search backwards." (let* ((pos (or pos (point))) (invisible-p (if ignore-hidden-p #'org-fold-core-folded-p #'invisible-p)) (invisible-initially? (funcall invisible-p pos)) (limit (or limit (if previous-p (point-min) (point-max)))) (cmp (if previous-p #'> #'<)) (next-change (if previous-p (if ignore-hidden-p (lambda (p) (org-fold-core-previous-folding-s= tate-change (org-fold-core-get-folding-spec nil p) p limit)) (lambda (p) (max limit (1- (previous-single-char-property-change p '= invisible nil limit))))) (if ignore-hidden-p (lambda (p) (org-fold-core-next-folding-state-c= hange (org-fold-core-get-folding-spec nil p) p limit)) (lambda (p) (next-single-char-property-change p 'invisible nil limit))= ))) (next pos)) (while (and (funcall cmp next limit) (not (xor invisible-initially? (funcall invisible-p next)))) (setq next (funcall next-change next))) next)) (defun org-fold-core-previous-visibility-change (&optional pos limit ignore= -hidden-p) "Call `org-fold-core-next-visibility-change' searching backwards." (org-fold-core-next-visibility-change pos limit ignore-hidden-p 'previous= )) (defun org-fold-core-next-folding-state-change (&optional spec-or-alias pos= limit previous-p) "Return next point where folding state changes relative to POS up to LIMI= T. If SPEC-OR-ALIAS is nil, return next point where _any_ single folding spec changes. For example, (org-fold-core-next-folding-state-change nil) with point somewhere in the below structure will return the nearest <...> point. * Headline :PROPERTIES: :ID: test :END: Fusce suscipit, wisi nec facilisis facilisis, est dui fermentum leo, quis t= empor ligula erat quis odio. ** Another headline :DRAWER: :END: ** Yet another headline If SPEC-OR-ALIAS is a folding spec symbol, only consider that folding spec. If SPEC-OR-ALIAS is a list, only consider changes of folding specs from the list. Search backwards when PREVIOUS-P is non-nil." (when (and spec-or-alias (symbolp spec-or-alias)) (setq spec-or-alias (list spec-or-alias))) (when spec-or-alias (setq spec-or-alias (mapcar (lambda (spec-or-alias) (or (org-fold-core-get-folding-spec-from-alias spec-or-alias) spec-or-alias)) spec-or-alias)) (mapc #'org-fold-core--check-spec spec-or-alias)) (unless spec-or-alias (setq spec-or-alias (org-fold-core-folding-spec-list))) (let* ((pos (or pos (point)))) (apply (if previous-p #'max #'min) (mapcar (if previous-p (lambda (prop) (max (or limit (point-min)) (previous-single-char-p= roperty-change pos prop nil (or limit (point-min))))) (lambda (prop) (next-single-char-property-change pos prop nil (or li= mit (point-max))))) (mapcar (lambda (el) (org-fold-core--property-symbol-get= -create el nil t)) spec-or-alias))))) (defun org-fold-core-previous-folding-state-change (&optional spec-or-alias= pos limit) "Call `org-fold-core-next-folding-state-change' searching backwards." (org-fold-core-next-folding-state-change spec-or-alias pos limit 'previou= s)) ;;;; Changing visibility ;;;;; Region visibility (defvar org-fold-core--fontifying nil "Flag used to restrict `org-fold-core-region' -> `font-lock-fontify-regio= n' -> `org-fold-core-region' recursion.") ;; This is the core function performing actual folding/unfolding. The ;; folding state is stored in text property (folding property) ;; returned by `org-fold-core--property-symbol-get-create'. The value of t= he ;; folding property is folding spec symbol. (defun org-fold-core-region (from to flag &optional spec-or-alias) "Hide or show lines from FROM to TO, according to FLAG. SPEC-OR-ALIAS is the folding spec or foldable element, as a symbol. If SPEC-OR-ALIAS is omitted and FLAG is nil, unfold everything in the regio= n." (let ((spec (org-fold-core-get-folding-spec-from-alias spec-or-alias))) (when spec (org-fold-core--check-spec spec)) (with-silent-modifications (org-with-wide-buffer (if flag (if (not spec) (error "Calling `org-fold-core-region' with missing SPEC") (put-text-property from to (org-fold-core--property-symbol-get-create= spec) spec) (put-text-property from to 'isearch-open-invisible #'org-fold-core--i= search-show) (put-text-property from to 'isearch-open-invisible-temporary #'org-fo= ld-core--isearch-show-temporary) (when org-fold-core--optimise-for-huge-buffers ;; If the SPEC has highest priority, assign it directly ;; to 'invisible property as well. This is done to speed ;; up Emacs redisplay on huge (Mbs) folded regions where ;; we don't even want Emacs to spend time cycling over ;; `char-property-alias-alist'. (when (eq spec (caar org-fold-core--specs)) (put-text-proper= ty from to 'invisible spec)))) (if (not spec) (mapc (lambda (spec) (org-fold-core-region from to nil spec)) = (org-fold-core-folding-spec-list)) (when org-fold-core--optimise-for-huge-buffers (when (eq spec (caar org-fold-core--specs)) (let ((pos from)) (while (< pos to) (if (eq spec (get-text-property pos 'invisible)) (let ((next (org-fold-core-next-folding-state-change= spec pos to))) (remove-text-properties pos next '(invisible t)) (setq pos next)) (setq pos (next-single-char-property-change pos 'invis= ible nil to))))))) (remove-text-properties from to (list (org-fold-core--property-symbol-g= et-create spec) nil)) ;; Fontify unfolded text. (unless (or org-fold-core--fontifying (not (org-fold-core-get-folding-spec-property spec := font-lock-skip))) (let ((org-fold-core--fontifying t)) (save-match-data (font-lo= ck-fontify-region from to)))))))))) ;;; Make isearch search in some text hidden via text propertoes (defvar org-fold-core--isearch-overlays nil "List of overlays temporarily created during isearch. This is used to allow searching in regions hidden via text properties. As for [2020-05-09 Sat], Isearch only has special handling of hidden overla= ys. Any text hidden via text properties is not revealed even if `search-invisib= le' is set to 't.") (defvar-local org-fold-core--isearch-local-regions (make-hash-table :test '= equal) "Hash table storing temporarily shown folds from isearch matches.") (defun org-fold-core--isearch-setup (type) "Initialize isearch in org buffer. TYPE can be either `text-properties' or `overlays'." (pcase type (`text-properties (setq-local search-invisible 'open-all) (add-hook 'isearch-mode-end-hook #'org-fold-core--clear-isearch-state = nil 'local) (add-hook 'isearch-mode-hook #'org-fold-core--clear-isearch-state nil = 'local) (setq-local isearch-filter-predicate #'org-fold-core--isearch-filter-p= redicate-text-properties)) (`overlays (setq-local isearch-filter-predicate #'org-fold-core--isearch-filter-p= redicate-overlays) (add-hook 'isearch-mode-end-hook #'org-fold-core--clear-isearch-overla= ys nil 'local)) (_ (error "%s: Unknown type of setup for `org-fold-core--isearch-setup'= " type)))) (defun org-fold-core--isearch-reveal (pos) "Default function used to reveal hidden text at POS for isearch." (let ((region (org-fold-core-get-region-at-point pos))) (org-fold-core-region (car region) (cdr region) nil))) (defun org-fold-core--isearch-filter-predicate-text-properties (beg end) "Make sure that folded text is searchable when user whant so. This function is intended to be used as `isearch-filter-predicate'." (and ;; Check folding specs that cannot be searched (seq-every-p (lambda (spec) (not (org-fold-core-get-folding-spec-propert= y spec :isearch-ignore))) (org-fold-core-get-folding-specs-in-region beg end)) ;; Check 'invisible properties that are not folding specs. (or (eq search-invisible t) ; User wants to search anyway, allow it. (let ((pos beg) unknown-invisible-property) (while (and (< pos end) (not unknown-invisible-property)) (when (and (get-text-property pos 'invisible) (not (org-fold-core-folding-spec-p (get-text-property= pos 'invisible)))) (setq unknown-invisible-property t)) (setq pos (next-single-char-property-change pos 'invisible))) (not unknown-invisible-property))) (or (and (eq search-invisible t) ;; FIXME: this opens regions permanenly for now. ;; I also tried to force search-invisible 'open-all around ;; `isearch-range-invisible', but that somehow causes ;; infinite loop in `isearch-lazy-highlight'. (prog1 t ;; We still need to reveal the folded location (org-fold-core--isearch-show-temporary (cons beg end) nil))) (not (isearch-range-invisible beg end))))) (defun org-fold-core--clear-isearch-state () "Clear `org-fold-core--isearch-local-regions'." (clrhash org-fold-core--isearch-local-regions)) (defun org-fold-core--isearch-show (region) "Reveal text in REGION found by isearch." (org-with-point-at (car region) (while (< (point) (cdr region)) (funcall org-fold-core-isearch-open-function (car region)) (goto-char (org-fold-core-next-visibility-change (point) (cdr region)= 'ignore-hidden))))) (defun org-fold-core--isearch-show-temporary (region hide-p) "Temporarily reveal text in REGION. Hide text instead if HIDE-P is non-nil." (if (not hide-p) (let ((pos (car region))) (while (< pos (cdr region)) (let ((spec-no-open (seq-find (lambda (spec) (not (org-fold-core-= get-folding-spec-property spec :isearch-open))) (org-fold-core-get-folding-= spec 'all pos)))) (if spec-no-open ;; Skip regions folded with folding specs that cannot be op= ened. (setq pos (org-fold-core-next-folding-state-change spec-no-= open pos (cdr region))) (dolist (spec (org-fold-core-get-folding-spec 'all pos)) (push (cons spec (org-fold-core-get-region-at-point spec pos)) (ge= thash region org-fold-core--isearch-local-regions))) (org-fold-core--isearch-show region) (setq pos (org-fold-core-next-folding-state-change nil pos (cdr regi= on))))))) (mapc (lambda (val) (org-fold-core-region (cadr val) (cddr val) t (car = val))) (gethash region org-fold-core--isearch-local-regions)) (remhash region org-fold-core--isearch-local-regions))) (defvar-local org-fold-core--isearch-special-specs nil "List of specs that can break visibility state when converted to overlays. This is a hack, but I do not see a better way around until isearch gets support of text properties.") (defun org-fold-core--create-isearch-overlays (beg end) "Replace text property invisibility spec by overlays between BEG and END. All the searcheable folded regions will be changed to use overlays instead of text properties. The created overlays will be stored in `org-fold-core--isearch-overlays'." (let ((pos beg)) (while (< pos end) ;; We need loop below to make sure that we clean all invisible ;; properties, which may be nested. (while (org-fold-core-get-folding-spec nil pos) (let* ((spec (org-fold-core-get-folding-spec nil pos)) (region (org-fold-core-get-region-at-point spec pos))) (when (memq spec org-fold-core--isearch-special-specs) (setq pos (min pos (car region))) (setq end (max end (cdr region)))) ;; Changing text properties is considered buffer modification. ;; We do not want it here. (with-silent-modifications (org-fold-core-region (car region) (cdr region) nil spec) ;; The overlay is modelled after `outline-flag-region' ;; [2020-05-09 Sat] overlay for 'outline blocks. (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) (overlay-put o 'evaporate t) (overlay-put o 'invisible spec) (overlay-put o 'org-invisible spec) ;; Make sure that overlays are applied in the same order ;; with the folding specs. ;; Note: `memq` returns cdr with car equal to the first ;; found matching element. (overlay-put o 'priority (length (memq spec (org-fold-core-fo= lding-spec-list)))) ;; `delete-overlay' here means that spec information will be lost ;; for the region. The region will remain visible. (if (org-fold-core-get-folding-spec-property spec :isearch-op= en) (overlay-put o 'isearch-open-invisible #'delete-overlay) (overlay-put o 'isearch-open-invisible #'ignore) (overlay-put o 'isearch-open-invisible-temporary #'ignore)) (push o org-fold-core--isearch-overlays))))) (setq pos (org-fold-core-next-folding-state-change nil pos end))))) (defun org-fold-core--isearch-filter-predicate-overlays (beg end) "Return non-nil if text between BEG and END is deemed visible by isearch. This function is intended to be used as `isearch-filter-predicate'." (org-fold-core--create-isearch-overlays beg end) ;; trick isearch by crea= ting overlays in place of invisible text (isearch-filter-visible beg end)) (defun org-fold-core--clear-isearch-overlay (ov) "Convert OV region back into using text properties." (let ((spec (if isearch-mode-end-hook-quit ;; Restore all folds. (overlay-get ov 'org-invisible) ;; Leave opened folds open. (overlay-get ov 'invisible)))) ;; Ignore deleted overlays. (when (and spec (overlay-buffer ov)) ;; Changing text properties is considered buffer modification. ;; We do not want it here. (with-silent-modifications (when (<=3D (overlay-end ov) (point-max)) (org-fold-core-region (overlay-start ov) (overlay-end ov) t spec))))) (when (member ov isearch-opened-overlays) (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) (delete-overlay ov)) (defun org-fold-core--clear-isearch-overlays () "Convert overlays from `org-fold-core--isearch-overlays' back into using = text properties." (when org-fold-core--isearch-overlays (mapc #'org-fold-core--clear-isearch-overlay org-fold-core--isearch-ove= rlays) (setq org-fold-core--isearch-overlays nil))) ;;; Handling changes in folded elements (defvar org-fold-core--ignore-modifications nil "Non-nil: skip processing modifications in `org-fold-core--fix-folded-reg= ion'.") (defvar org-fold-core--ignore-fragility-checks nil "Non-nil: skip fragility checks in `org-fold-core--fix-folded-region'.") (defmacro org-fold-core-ignore-modifications (&rest body) "Run BODY ignoring buffer modifications in `org-fold-core--fix-folded-reg= ion'." (declare (debug (form body)) (indent 1)) `(let ((org-fold-core--ignore-modifications t)) (unwind-protect (progn ,@body) (setq org-fold-core--last-buffer-chars-modified-tick (buffer-chars-m= odified-tick))))) (defmacro org-fold-core-ignore-fragility-checks (&rest body) "Run BODY skipping :fragility checks in `org-fold-core--fix-folded-region= '." (declare (debug (form body)) (indent 1)) `(let ((org-fold-core--ignore-fragility-checks t)) (progn ,@body))) (defvar-local org-fold-core--last-buffer-chars-modified-tick nil "Variable storing the last return value of `buffer-chars-modified-tick'.") (defun org-fold-core--fix-folded-region (from to _) "Process modifications in folded elements within FROM . TO region. This function intended to be used as one of `after-change-functions'. This function does nothing if text the only modification was changing text properties (for the sake of reducing overheads). If a text was inserted into invisible region, hide the inserted text. If a text was inserted in front/back of the region, hide it according to :font-sticky/:rear-sticky folding spec property. If the folded region is folded with a spec with non-nil :fragile property, unfold the region if the :fragile function returns non-nil." ;; If no insertions or deletions in buffer, skip all the checks. (unless (or (eq org-fold-core--last-buffer-chars-modified-tick (buffer-ch= ars-modified-tick)) org-fold-core--ignore-modifications) ;; Store the new buffer modification state. (setq org-fold-core--last-buffer-chars-modified-tick (buffer-chars-modi= fied-tick)) (save-match-data ;; Handle changes in all the indirect buffers and in the base ;; buffer. Work around Emacs bug#46982. (let ((buffers (seq-filter #'buffer-live-p (cons (or (buffer-base-buffer) (current-bu= ffer)) (with-current-buffer (or (buffer-bas= e-buffer) (current-buffer)) org-fold-core--indirect-buffers))))) (dolist (buf buffers) (with-current-buffer buf ;; Re-hide text inserted in the middle/font/back of a folded ;; region. (unless (equal from to) ; Ignore deletions. (dolist (spec (org-fold-core-folding-spec-list)) ;; Reveal fully invisible text. This is needed, for ;; example, when there was a deletion in a folded heading, ;; the heading was unfolded, end `undo' was called. The ;; `undo' would insert the folded text. (when (and (org-fold-core-region-folded-p from to spec) (org-region-invisible-p from to)) (org-fold-core-region from to nil spec)) ;; Look around and fold the new text if the nearby folds are ;; sticky. (let ((spec-to (org-fold-core-get-folding-spec spec (min to (1- (p= oint-max))))) (spec-from (org-fold-core-get-folding-spec spec (max (point-min) (1= - from))))) ;; Hide text inserted in the middle of a fold. (when (and (or spec-from (eq from (point-min))) (or spec-to (eq to (point-max))) (or spec-from spec-to) (eq spec-to spec-from) (or (org-fold-core-get-folding-spec-property s= pec :front-sticky) (org-fold-core-get-folding-spec-property s= pec :rear-sticky))) (org-fold-core-region from to t (or spec-from spec-to))) ;; Hide text inserted at the end of a fold. (when (and spec-from (org-fold-core-get-folding-spec-prop= erty spec-from :rear-sticky)) (org-fold-core-region from to t spec-from)) ;; Hide text inserted in front of a fold. (when (and spec-to (org-fold-core-get-folding-spec-proper= ty spec-to :front-sticky)) (org-fold-core-region from to t spec-to)))))))) ;; Process all the folded text between `from' and `to'. Do it ;; only in current buffer to avoid verifying semantic structure ;; multiple times in indirect buffers that have exactly same ;; text anyway. (unless org-fold-core--ignore-fragility-checks (dolist (func org-fold-core-extend-changed-region-functions) (let ((new-region (funcall func from to))) (setq from (car new-region)) (setq to (cdr new-region)))) (dolist (spec (org-fold-core-folding-spec-list)) ;; No action is needed when :fragile is nil for the spec. (when (org-fold-core-get-folding-spec-property spec :fragile) (org-with-wide-buffer ;; Expand the considered region to include partially present f= old. ;; Note: It is important to do this inside loop over all ;; specs. Otherwise, the region may be expanded to huge ;; outline fold, potentially involving majority of the ;; buffer. That would cause the below code to loop over ;; almost all the folds in buffer, which would be too slow. (let ((local-from from) (local-to to) (region-from (org-fold-core-get-region-at-point spec (ma= x (point-min) (1- from)))) (region-to (org-fold-core-get-region-at-point spec (min = to (1- (point-max)))))) (when region-from (setq local-from (car region-from))) (when region-to (setq local-to (cdr region-to))) (let ((pos local-from)) ;; Move to the first hidden region. (unless (org-fold-core-get-folding-spec spec pos) (setq pos (org-fold-core-next-folding-state-change spec = pos local-to))) ;; Cycle over all the folds. (while (< pos local-to) (save-match-data ; we should not clobber match-data in a= fter-change-functions (let ((fold-begin (and (org-fold-core-get-folding-spec= spec pos) pos)) (fold-end (org-fold-core-next-folding-state-chan= ge spec pos local-to))) (when (and fold-begin fold-end) (when (save-excursion (funcall (org-fold-core-get-folding-spec-p= roperty spec :fragile) (cons fold-begin fold-end) spec)) ;; Reveal completely, not just from the SPEC. (org-fold-core-region fold-begin fold-end nil)))= )) ;; Move to next fold. (setq pos (org-fold-core-next-folding-state-change spec = pos local-to)))))))))))) ;;; Hanlding killing/yanking of folded text ;; By default, all the text properties of the killed text are ;; preserved, including the folding text properties. This can be ;; awkward when we copy a text from an indirect buffer to another ;; indirect buffer (or the base buffer). The copied text might be ;; visible in the source buffer, but might disappear if we yank it in ;; another buffer. This happens in the following situation: ;; ---- base buffer ---- ;; * Headline ;; Some text hidden in the base buffer, but revealed in the indirect ;; buffer. ;; * Another headline ;; ;; ---- end of base buffer ---- ;; ---- indirect buffer ---- ;; * Headline ;; Some text hidden in the base buffer, but revealed in the indirect ;; buffer. ;; * Another headline ;; ;; ---- end of indirect buffer ---- ;; If we copy the text under "Headline" from the indirect buffer and ;; insert it under "Another headline" in the base buffer, the inserted ;; text will be hidden since it's folding text properties are copyed. ;; Basically, the copied text would have two sets of folding text ;; properties: (1) Properties for base buffer telling that the text is ;; hidden; (2) Properties for the indirect buffer telling that the ;; text is visible. The first set of the text properties in inactive ;; in the indirect buffer, but will become active once we yank the ;; text back into the base buffer. ;; ;; To avoid the above situation, we simply clear all the properties, ;; unrealated to current buffer when a text is copied. ;; FIXME: Ideally, we may want to carry the folding state of copied ;; text between buffer (probably via user customisation). (defun org-fold-core--buffer-substring-filter (beg end &optional delete) "Clear folding state in killed text. This function is intended to be used as `filter-buffer-substring-function'. The arguments and return value are as specified for `filter-buffer-substrin= g'." (let ((return-string (buffer-substring--filter beg end delete)) ;; The list will be used as an argument to `remove-text-properties'. props-list) ;; There is no easy way to examine all the text properties of a ;; string, so we utilise the fact that printed string ;; representation lists all its properties. ;; Loop over the elements of string representation. (unless (string-empty-p return-string) ;; Collect all the text properties the string is completely ;; hidden with. (dolist (spec (org-fold-core-folding-spec-list)) (when (and (org-fold-core-region-folded-p beg end spec) (org-region-invisible-p beg end)) (push (org-fold-core--property-symbol-get-create spec nil t) prop= s-list))) (dolist (plist (object-intervals return-string)) (let* ((start (car plist)) (fin (cadr plist)) (plist (caddr plist))) ;; Only lists contain text properties. (when (listp plist) ;; Collect all the relevant text properties. (while plist (let* ((prop (car plist)) (prop-name (symbol-name prop))) ;; Reveal hard-hidden text. See ;; `org-fold-core--optimise-for-huge-buffers'. (when (and (eq prop 'invisible) (member (cadr plist) (org-fold-core-folding-spec= -list))) (remove-text-properties start fin '(invisible t) return-s= tring)) ;; We do not care about values now. (setq plist (cddr plist)) (when (string-match-p org-fold-core--spec-property-prefix p= rop-name) ;; Leave folding specs from current buffer. See ;; comments in `org-fold-core--property-symbol-get-create= ' to ;; understand why it works. (unless (member prop (alist-get 'invisible char-property-= alias-alist)) (push prop props-list)))))))) (remove-text-properties 0 (length return-string) props-list return-st= ring)) return-string)) ;;; Do not fontify folded text until needed. (defun org-fold-core-fontify-region (beg end loudly) "Run `font-lock-default-fontify-region' unless we are trying to fontify i= nvisible text." (let ((pos beg) next) (while (< pos end) (setq next (org-fold-core-next-visibility-change pos end t)) (unless (and (org-invisible-p pos) (seq-find (lambda (spec) (org-fold-core-get-folding-spec= -property spec :font-lock-skip)) (org-fold-core-get-folding-spec 'all pos))) (font-lock-default-fontify-region pos next loudly)) (setq pos next)))) (defun org-fold-core-update-optimisation (beg end) "Update huge buffer optimisation between BEG and END. See `org-fold-core--optimise-for-huge-buffers'." (when org-fold-core--optimise-for-huge-buffers (let ((pos beg)) (while (< pos end) (when (and (org-fold-core-folded-p pos (caar org-fold-core--specs)) (not (eq (caar org-fold-core--specs) (get-text-property = pos 'invisible)))) (put-text-property pos (org-fold-core-next-folding-state-change (= caar org-fold-core--specs) pos end) 'invisible (caar org-fold-core--specs))) (setq pos (org-fold-core-next-folding-state-change (caar org-fold-c= ore--specs) pos end)))))) (defun org-fold-core-remove-optimisation (beg end) "Remove huge buffer optimisation between BEG and END. See `org-fold-core--optimise-for-huge-buffers'." (when org-fold-core--optimise-for-huge-buffers (let ((pos beg)) (while (< pos end) (when (and (org-fold-core-folded-p pos (caar org-fold-core--specs)) (eq (caar org-fold-core--specs) (get-text-property pos '= invisible))) (remove-text-properties pos (org-fold-core-next-folding-state-cha= nge (caar org-fold-core--specs) pos end) '(invisible t))) (setq pos (org-fold-core-next-folding-state-change (caar org-fold-c= ore--specs) pos end)))))) (provide 'org-fold-core) ;;; org-fold-core.el ends here --=-=-=--