From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 0BujByQsyV4oZgAA0tVLHw (envelope-from ) for ; Sat, 23 May 2020 13:59:00 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 4BuDAyQsyV4bMAAAB5/wlQ (envelope-from ) for ; Sat, 23 May 2020 13:59:00 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6F0039404C7 for ; Sat, 23 May 2020 13:58:59 +0000 (UTC) Received: from localhost ([::1]:52540 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jcUfy-00067k-Da for larch@yhetil.org; Sat, 23 May 2020 09:58:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jcUfU-00062X-4K for emacs-orgmode@gnu.org; Sat, 23 May 2020 09:58:28 -0400 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]:35071) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jcUfQ-0002nd-6u for emacs-orgmode@gnu.org; Sat, 23 May 2020 09:58:27 -0400 Received: by mail-pf1-x435.google.com with SMTP id n18so6526671pfa.2 for ; Sat, 23 May 2020 06:58:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=Bg1blx33F2j51twZA81SqHMaz2C7Q8hWAluLKEiqD/8=; b=ItrjIQihLO35jn+WRmqMbczkh54BRcyODuAgFhek0sGUNkb/8Ew6c3UnwMai9l2vm4 z/j1QYxn5w1GmOWPvIvrbW2Qu37H1SuNVH/6SMKpLrQHk8vTLn6a8IBItLHPMQpMxVqU 5OoQKCOYcYfgnIDC8uTqP5JsPfB4EVWAHqqyDIFxZHigVFhzCUChG+7IYDdzWwKRvYhi APYoXsHMqw3vKNCQSmv966dEm4Wto3QhwwwYqVmPSB063W/kvRLRDI2FcAyvtHrazS3U 6zE2tP1khpcR+vT/pcOrsyhwnvZQvVuKPtbXKd5E1Xa+IV+YDnxH5jfJpwQvZgSbYdPa UHow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=Bg1blx33F2j51twZA81SqHMaz2C7Q8hWAluLKEiqD/8=; b=iXoIPCPzWuPwT9Nfw1jvKNpIthgRjcBBp7/1ZDOZQQqAmAUj5R88nzPSuyHTkf/0fx d/e/ZrBTRWhlCdwzbklH5S3u8gic4HrdL7cCy1++YUZFWYSWtl3HVVa8pssEpkr0Q/AZ ClByhlautAZZ0XFJ4C/Kjjl4ZMzBAr26+aI6jr2B6hqwqOf3bASNIfDotBQ90WKr6B4q vPnrTgmoeZZ5pUF0eS+cuJqxLuYma0anEnzFkPdj2G/P/t4bbxQJd+XOoXOwCLQxPOFO 6YOzeInej+JhUI2OEZQc56y+sy1384PttAH5j/MuFGANiiH8KnMT3RIQVWS5WLLnVDJM 1m2g== X-Gm-Message-State: AOAM532N0RpJ2LJ8qqs12SqaYQmdcYQr3lsDqEiqmJXtAPUszrgAL132 h60bpAG6ufVuFxvwCmsU1PiXc3yk/WVKZg== X-Google-Smtp-Source: ABdhPJzZySwMu+fDBISLQYEinfuDw/3IZ8Rc02iSltvq8sFVfvRCqy6RZjXVSAvZjN38wYX4CckJFg== X-Received: by 2002:a62:5ac2:: with SMTP id o185mr8723069pfb.148.1590242302807; Sat, 23 May 2020 06:58:22 -0700 (PDT) Received: from localhost ([104.151.6.52]) by smtp.gmail.com with ESMTPSA id s94sm9674024pjb.20.2020.05.23.06.58.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 May 2020 06:58:22 -0700 (PDT) From: Ihor Radchenko To: Nicolas Goaziou Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers In-Reply-To: <87sgfqu5av.fsf@localhost> References: <87h7x9e5jo.fsf@localhost> <875zdpia5i.fsf@nicolasgoaziou.fr> <87y2qi8c8w.fsf@localhost> <87r1vu5qmc.fsf@nicolasgoaziou.fr> <87imh5w1zt.fsf@localhost> <87blmxjckl.fsf@localhost> <87y2q13tgs.fsf@nicolasgoaziou.fr> <878si1j83x.fsf@localhost> <87d07bzvhd.fsf@nicolasgoaziou.fr> <87imh34usq.fsf@localhost> <87pnbby49m.fsf@nicolasgoaziou.fr> <87tv0efvyd.fsf@localhost> <874kse1seu.fsf@localhost> <87r1vhqpja.fsf@nicolasgoaziou.fr> <87tv0d2nk7.fsf@localhost> <87o8qkhy3g.fsf@nicolasgoaziou.fr> <87sgfqu5av.fsf@localhost> Date: Sat, 23 May 2020 21:53:42 +0800 Message-ID: <87pnauu595.fsf@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::435; envelope-from=yantar92@gmail.com; helo=mail-pf1-x435.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode@gnu.org Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmail.com header.s=20161025 header.b=ItrjIQih; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Spam-Score: 0.59 X-TUID: 0Pe0hTT+P7zi --=-=-= Content-Type: text/plain The patch is attached --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=featuredrawertextprop-20200523.patch diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el index 9f8677871..ab470ea9b 100644 --- a/contrib/lisp/org-notify.el +++ b/contrib/lisp/org-notify.el @@ -246,7 +246,7 @@ seconds. The default value for SECS is 20." (switch-to-buffer (find-file-noselect file)) (org-with-wide-buffer (goto-char begin) - (outline-show-entry)) + (org-show-entry)) (goto-char begin) (search-forward "DEADLINE: <") (search-forward ":") diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el index bfc4d6c3e..2312b235c 100644 --- a/contrib/lisp/org-velocity.el +++ b/contrib/lisp/org-velocity.el @@ -325,7 +325,7 @@ use it." (save-excursion (when narrow (org-narrow-to-subtree)) - (outline-show-all))) + (org-show-all))) (defun org-velocity-edit-entry/inline (heading) "Edit entry at HEADING in the original buffer." diff --git a/doc/org-manual.org b/doc/org-manual.org index 96b160175..2ebe94538 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and Switch back to the startup visibility of the buffer (see [[*Initial visibility]]). -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) :: +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) :: #+cindex: show all, command #+kindex: C-u C-u C-u TAB - #+findex: outline-show-all + #+findex: org-show-all Show all, including drawers. - {{{kbd(C-c C-r)}}} (~org-reveal~) :: @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and headings. With a double prefix argument, also show the entire subtree of the parent. -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) :: +- {{{kbd(C-c C-k)}}} (~org-show-branches~) :: #+cindex: show branches, command #+kindex: C-c C-k - #+findex: outline-show-branches + #+findex: org-show-branches Expose all the headings of the subtree, but not their bodies. -- {{{kbd(C-c TAB)}}} (~outline-show-children~) :: +- {{{kbd(C-c TAB)}}} (~org-show-children~) :: #+cindex: show children, command #+kindex: C-c TAB - #+findex: outline-show-children + #+findex: org-show-children Expose all direct children of the subtree. With a numeric prefix argument {{{var(N)}}}, expose all children down to level {{{var(N)}}}. @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way: command (see [[*Visibility Cycling]]). You can force cycling archived subtrees with {{{kbd(C-TAB)}}}, or by setting the option ~org-cycle-open-archived-trees~. Also normal outline commands, like - ~outline-show-all~, open archived subtrees. + ~org-show-all~, open archived subtrees. - #+vindex: org-sparse-tree-open-archived-trees diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index ab13f926c..ad9244940 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'." (t " %-12:c%?-12t% s"))) (start 0) varform vars var e c f opt) - (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)" + (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)" s start) (setq var (or (cdr (assoc (match-string 4 s) '(("c" . category) ("t" . time) ("l" . level) ("s" . extra) @@ -9138,20 +9138,20 @@ if it was hidden in the outline." ((and (called-interactively-p 'any) (= more 1)) (message "Remote: show with default settings")) ((= more 2) - (outline-show-entry) + (org-show-entry) (org-show-children) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'children)) (message "Remote: CHILDREN")) ((= more 3) - (outline-show-subtree) + (org-show-subtree) (save-excursion (org-back-to-heading) (run-hook-with-args 'org-cycle-hook 'subtree)) (message "Remote: SUBTREE")) ((> more 3) - (outline-show-subtree) + (org-show-subtree) (message "Remote: SUBTREE AND ALL DRAWERS"))) (select-window win))) diff --git a/lisp/org-archive.el b/lisp/org-archive.el index d3e12d17b..d864dad8a 100644 --- a/lisp/org-archive.el +++ b/lisp/org-archive.el @@ -330,7 +330,7 @@ direct children of this heading." (insert (if datetree-date "" "\n") heading "\n") (end-of-line 0)) ;; Make the subtree visible - (outline-show-subtree) + (org-show-subtree) (if org-archive-reversed-order (progn (org-back-to-heading t) diff --git a/lisp/org-colview.el b/lisp/org-colview.el index e50a4d7c8..e656df555 100644 --- a/lisp/org-colview.el +++ b/lisp/org-colview.el @@ -699,7 +699,7 @@ FUN is a function called with no argument." (move-beginning-of-line 2) (org-at-heading-p t))))) (unwind-protect (funcall fun) - (when hide-body (outline-hide-entry))))) + (when hide-body (org-hide-entry))))) (defun org-columns-previous-allowed-value () "Switch to the previous allowed value for this column." diff --git a/lisp/org-compat.el b/lisp/org-compat.el index 635a38dcd..8fe271896 100644 --- a/lisp/org-compat.el +++ b/lisp/org-compat.el @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer." ;;; Emacs < 25.1 compatibility (when (< emacs-major-version 25) - (defalias 'outline-hide-entry 'hide-entry) - (defalias 'outline-hide-sublevels 'hide-sublevels) - (defalias 'outline-hide-subtree 'hide-subtree) (defalias 'outline-show-branches 'show-branches) (defalias 'outline-show-children 'show-children) - (defalias 'outline-show-entry 'show-entry) (defalias 'outline-show-subtree 'show-subtree) (defalias 'xref-find-definitions 'find-tag) (defalias 'format-message 'format) diff --git a/lisp/org-element.el b/lisp/org-element.el index ac41b7650..2d5c8d771 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -4320,7 +4320,7 @@ element or object. Meaningful values are `first-section', TYPE is the type of the current element or object. If PARENT? is non-nil, assume the next element or object will be -located inside the current one. " +located inside the current one." (if parent? (pcase type (`headline 'section) diff --git a/lisp/org-keys.el b/lisp/org-keys.el index c006e9c12..deb5d7b90 100644 --- a/lisp/org-keys.el +++ b/lisp/org-keys.el @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names." #'org-next-visible-heading) (define-key org-mode-map [remap outline-previous-visible-heading] #'org-previous-visible-heading) -(define-key org-mode-map [remap show-children] #'org-show-children) +(define-key org-mode-map [remap outline-show-children] #'org-show-children) ;;;; Make `C-c C-x' a prefix key (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap)) diff --git a/lisp/org-macs.el b/lisp/org-macs.el index a02f713ca..fa0a658f0 100644 --- a/lisp/org-macs.el +++ b/lisp/org-macs.el @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead." -;;; Overlays +;;; Overlays and text properties (defun org-overlay-display (ovl text &optional face evap) "Make overlay OVL display TEXT with face FACE." @@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays." (delete (delete-overlay ov)) (t (push ov found)))))) +(defun org-remove-text-properties (start end properties &optional object) + "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions. +Do not remove invisible text properties specified by 'outline, +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this +is needed to keep outlines, drawers, and blocks hidden unless they are +toggled by user. +Note: The below may be too specific and create troubles if more +invisibility specs are added to org in future" + (when (plist-member properties 'invisible) + (let ((pos start) + next spec) + (while (< pos end) + (setq next (next-single-property-change pos 'invisible nil end) + spec (get-text-property pos 'invisible)) + (unless (memq spec (list 'org-hide-block + 'org-hide-drawer + 'outline)) + (remove-text-properties pos next '(invisible nil) object)) + (setq pos next)))) + (when-let ((properties-stripped (org-plist-delete properties 'invisible))) + (remove-text-properties start end properties-stripped object))) + +(defun org--find-text-property-region (pos prop) + "Find a region containing PROP text property around point POS." + (let* ((beg (and (get-text-property pos prop) pos)) + (end beg)) + (when beg + ;; when beg is the first point in the region, `previous-single-property-change' + ;; will return nil. + (setq beg (or (previous-single-property-change pos prop) + beg)) + ;; when end is the last point in the region, `next-single-property-change' + ;; will return nil. + (setq end (or (next-single-property-change pos prop) + end)) + (unless (= beg end) ; this should not happen + (cons beg end))))) + (defun org-flag-region (from to flag spec) "Hide or show lines from FROM to TO, according to FLAG. SPEC is the invisibility spec, as a symbol." - (remove-overlays from to 'invisible spec) - ;; Use `front-advance' since text right before to the beginning of - ;; the overlay belongs to the visible line than to the contents. - (when flag - (let ((o (make-overlay from to nil 'front-advance))) - (overlay-put o 'evaporate t) - (overlay-put o 'invisible spec) - (overlay-put o 'isearch-open-invisible #'delete-overlay)))) - + (pcase spec + ('outline + (remove-overlays from to 'invisible spec) + ;; Use `front-advance' since text right before to the beginning of + ;; the overlay belongs to the visible line than to the contents. + (when flag + (let ((o (make-overlay from to nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + (overlay-put o 'isearch-open-invisible #'delete-overlay)))) + (_ + ;; Use text properties instead of overlays for speed. + ;; Overlays are too slow (Emacs Bug#35453). + (with-silent-modifications + ;; keep a backup stack of old text properties + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((old-spec (get-text-property (point) 'invisible)) + (end (next-single-property-change (point) 'invisible nil to))) + (when old-spec + (alter-text-property (point) end 'org-property-stack-invisible + (lambda (stack) + (if (or (eq old-spec (car stack)) + (eq spec old-spec) + (eq old-spec 'outline)) + stack + (cons old-spec stack))))) + (goto-char end)))) + + ;; cleanup everything + (remove-text-properties from to '(invisible nil)) + + ;; Recover properties from the backup stack + (unless flag + (save-excursion + (goto-char from) + (while (< (point) to) + (let ((stack (get-text-property (point) 'org-property-stack-invisible)) + (end (next-single-property-change (point) 'org-property-stack-invisible nil to))) + (if (not stack) + (remove-text-properties (point) end '(org-property-stack-invisible nil)) + (put-text-property (point) end 'invisible (car stack)) + (alter-text-property (point) end 'org-property-stack-invisible + (lambda (stack) + (cdr stack)))) + (goto-char end))))) + + (when flag + (put-text-property from to 'rear-non-sticky nil) + (put-text-property from to 'front-sticky t) + (put-text-property from to 'invisible spec)))))) ;;; Regexp matching diff --git a/lisp/org-src.el b/lisp/org-src.el index c9eef744e..e89a1c580 100644 --- a/lisp/org-src.el +++ b/lisp/org-src.el @@ -523,8 +523,8 @@ Leave point in edit buffer." (org-src-switch-to-buffer buffer 'edit) ;; Insert contents. (insert contents) - (remove-text-properties (point-min) (point-max) - '(display nil invisible nil intangible nil)) + (org-remove-text-properties (point-min) (point-max) + '(display nil invisible nil intangible nil)) (unless preserve-ind (org-do-remove-indentation)) (set-buffer-modified-p nil) (setq buffer-file-name nil) diff --git a/lisp/org-table.el b/lisp/org-table.el index 6462b99c4..75801161b 100644 --- a/lisp/org-table.el +++ b/lisp/org-table.el @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'." (arg (let ((b (save-excursion (skip-chars-backward "^|") (point))) (e (save-excursion (skip-chars-forward "^|\r\n") (point)))) - (remove-text-properties b e '(invisible t intangible t)) + (org-remove-text-properties b e '(invisible t intangible t)) (if (and (boundp 'font-lock-mode) font-lock-mode) (font-lock-fontify-block)))) (t @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'." (setq word-wrap t) (goto-char (setq p (point-max))) (insert (org-trim field)) - (remove-text-properties p (point-max) '(invisible t intangible t)) + (org-remove-text-properties p (point-max) '(invisible t intangible t)) (goto-char p) (setq-local org-finish-function 'org-table-finish-edit-field) (setq-local org-window-configuration cw) diff --git a/lisp/org.el b/lisp/org.el index e577dc661..360974135 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function cdlatex-math-symbol "ext:cdlatex") (declare-function Info-goto-node "info" (nodename &optional fork strict-case)) (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag)) +(declare-function isearch-filter-visible "isearch" (beg end)) (declare-function org-add-archive-files "org-archive" (files)) (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom)) (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour)) @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.") (defvar ffap-url-regexp) (defvar org-element-paragraph-separate) +(defvar org-element-all-objects) +(defvar org-element-all-elements) +(defvar org-element-greater-elements) (defvar org-indent-indentation-per-level) (defvar org-radio-target-regexp) (defvar org-target-link-regexp) @@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.") ;;;; Define the Org mode +;;; Handling buffer modifications + (defun org-before-change-function (_beg _end) "Every change indicates that a table might need an update." (setq org-table-may-need-update t)) + +(defvar-local org--modified-elements nil + "List of elements, marked as recently modified. +There is no guarantee that the elements in this list are fully parsed. +Only the element type, :begin and :end properties of the elements are +guaranteed to be available. The :begin and :end element properties +contain markers instead of positions.") + +(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command) + "List of commands triggerring element modifications unconditionally.") + +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$") + (property-drawer . ,org-property-start-re) + (drawer . ,org-drawer-regexp) + (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$") + (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$")) + "Alist of regexps matching beginning of elements. +Group 1 in the regexps (if any) contains the element type.") + +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$") + (property-drawer . ,org-property-end-re) + (drawer . ,org-property-end-re) + (quote-block . "^[ \t]*#\\+end_quote[ \t]*$") + (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$")) + "Alist of regexps matching end of elements. +Group 1 in the regexps (if any) contains the element type or END.") + +(defvar org-track-element-modifications + `((property-drawer . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (drawer . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (center-block . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (quote-block . (:after-change-function + org--drawer-or-block-unfold-maybe)) + (special-block . (:after-change-function + org--drawer-or-block-unfold-maybe))) + "Alist of elements to be tracked for modifications. +The modification is only triggered according to :sensitive-re-list and +:sensitive-command-list (see below). +Each element of the alist is a cons of an element symbol and plist +defining how and when the modifications are handled. +In case of recursive elements/duplicates, the first element from the +list is considered. +The plist can have the following properties: +- :element-beginning-re :: regex matching beginning of the element + (default) :: (alist-get element org--element-beginning-re-alist) +- :element-end-re :: regex matching end of the element + (default) :: (alist-get element org--element-end-re-alist) +- :after-change-function :: function called after the modification +The function must accept a single argument - element from +`org--modified-elements'.") + +(defun org--get-element-region-at-point (types) + "Return TYPES element at point or nil. +If TYPES is a list, return first element at point from the list. The +returned value is partially parsed element only containing :begin and +:end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (begin-limit (save-excursion (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok)) + (point))) + (end-limit (or (save-excursion (outline-next-heading)) + (point-max))) + (point (point)) + begin end closest-begin) + (when (and begin-re end-re) + (save-excursion + (end-of-line) + (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point))) + (when (re-search-forward end-re end-limit 'noerror) (setq end (point))) + (setq closest-begin begin) + ;; slurp unmatched begin-re + (when (and begin end) + (goto-char begin) + (while (and (re-search-backward begin-re begin-limit 'noerror) + (= end (save-excursion (re-search-forward end-re end-limit 'noerror)))) + (setq begin (point))) + (when (and (>= point begin) (<= point end)) + (throw 'exit + (list type + (list + :begin begin + :end end))))))))))) + +(defun org--get-next-element-region-at-point (types &optional limit previous) + "Return TYPES element after point or nil. +If TYPES is a list, return first element after point from the list. +If PREVIOUS is non-nil, return first TYPES element before point. +Limit search by LIMIT or previous/next heading. +The returned value is partially parsed element only containing :begin +and :end properties. Only elements listed in +org--element-beginning-re-alist and org--element-end-re-alist can be +parsed here." + (catch 'exit + (dolist (type (if (listp types) types (list types))) + (let* ((begin-re (alist-get type org--element-beginning-re-alist)) + (end-re (alist-get type org--element-end-re-alist)) + (limit (or limit (if previous + (save-excursion + (org-with-limited-levels + (org-back-to-heading-or-point-min 'invisible-ok) + (point))) + (or (save-excursion (outline-next-heading)) + (point-max))))) + begin end) + (when (and begin-re end-re) + (save-excursion + (if previous + (when (re-search-backward begin-re limit 'noerror) + (when-let ((el (org--get-element-region-at-point type))) + (setq begin (org-element-property :begin el)) + (setq end (org-element-property :end el)))) + (when (re-search-forward begin-re limit 'noerror) + (when-let ((el (org--get-element-region-at-point type))) + (setq begin (org-element-property :begin el)) + (setq end (org-element-property :end el)))))) + (when (and begin end) + (throw 'exit + (list type + (list + :begin begin + :end end))))))))) + +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours) + "Find all elements from ELEMENTS in region BEG . END. +All the listed elements must be resolvable by +`org--get-element-region-at-point'. +Include elements if they are partially inside region when +INCLUDE-PARTIAL is non-nil. +Include preceding/subsequent neighbouring elements when no partial +element is found at the beginning/end of the region and +INCLUDE-NEIGHBOURS is non-nil." + (when include-partial + (org-with-point-at beg + (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements)))) + (if new-beg + (setq beg new-beg) + (when (and include-neighbours + (setq new-beg (org-element-property :begin + (org--get-next-element-region-at-point elements + (point-min) + 'previous)))) + (setq beg new-beg)))) + (when (memq 'headline elements) + (when-let ((new-beg (save-excursion + (org-with-limited-levels (outline-previous-heading))))) + (setq beg new-beg)))) + (org-with-point-at end + (let ((new-end (org-element-property :end (org--get-element-region-at-point elements)))) + (if new-end + (setq end new-end) + (when (and include-neighbours + (setq new-end (org-element-property :end + (org--get-next-element-region-at-point elements (point-max))))) + (setq end new-end)))) + (when (memq 'headline elements) + (when-let ((new-end (org-with-limited-levels (outline-next-heading)))) + (setq end (1- new-end)))))) + (save-excursion + (save-restriction + (narrow-to-region beg end) + (goto-char (point-min)) + (let (result el) + (while (setq el (org--get-next-element-region-at-point elements end)) + (push el result) + (goto-char (org-element-property :end el))) + result)))) + +(defun org--drawer-or-block-unfold-maybe (el) + "Update visibility of modified folded drawer/block EL. +If text was added to hidden drawer/block, make sure that the text is +also hidden, unless the change was done by a command listed in +`org-track-element-modification-default-sensitive-commands'. If the +modification destroyed the drawer/block, reveal the hidden text in +former drawer/block. If the modification shrinked/expanded the +drawer/block beyond the hidden text, reveal the affected +drawers/blocks as well. +Examples: +---------------------------------------------- +---------------------------------------------- +Case #1 (the element content is hidden): +---------------------------------------------- +:PROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:ROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +Text is revealed, because we have drawer in place of property-drawer +---------------------------------------------- +---------------------------------------------- +Case #2 (the element content is hidden): +---------------------------------------------- +:ROPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:OPERTIES: +:ID: 279e797c-f4a7-47bb-80f6-e72ac6f3ec55 +:END: +---------------------------------------------- +The text remains hidden since it is still a drawer. +---------------------------------------------- +---------------------------------------------- +Case #3: (the element content is hidden): +---------------------------------------------- +:FOO: +bar +tmp +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:FOO: +bar +:END: +tmp +:END: +---------------------------------------------- +The text is revealed because the drawer contents shrank. +---------------------------------------------- +---------------------------------------------- +Case #4: (the element content is hidden in both the drawers): +---------------------------------------------- +:FOO: +bar +tmp +:END: +:BAR: +jjd +:END: +---------------------------------------------- +is changed to +---------------------------------------------- +:FOO: +bar +tmp +:BAR: +jjd +:END: +---------------------------------------------- +The text is revealed in both the drawers because the drawers are merged +into a new drawer. +---------------------------------------------- +---------------------------------------------- +Case #5: (the element content is hidden) +---------------------------------------------- +:test: +Vivamus id enim. +:end: +---------------------------------------------- +is changed to +---------------------------------------------- +:drawer: +:test: +Vivamus id enim. +:end: +---------------------------------------------- +The text is revealed in the drawer because the drawer expended. +---------------------------------------------- +---------------------------------------------- +Case #6: (the element content is hidden): +---------------------------------------------- +:test: +Vivamus id enim. +:end: +---------------------------------------------- +is changed to +---------------------------------------------- +:test: +Vivamus id enim. +:end: +Nam a sapien. +:end: +---------------------------------------------- +The text remains hidden because drawer contents is always before the +first :end:." + (save-match-data + (save-excursion + (save-restriction + (goto-char (org-element-property :begin el)) + (let* ((newel (org--get-element-region-at-point + (mapcar (lambda (el) + (when (string-match-p (regexp-opt '("block" "drawer")) + (symbol-name (car el))) + (car el))) + org-track-element-modifications))) + (spec (if (string-match-p "block" (symbol-name (org-element-type el))) + 'org-hide-block + (if (string-match-p "drawer" (symbol-name (org-element-type el))) + 'org-hide-drawer + t))) + (toggle-func (if (eq spec 'org-hide-drawer) + #'org-hide-drawer-toggle + (if (eq spec 'org-hide-block) + #'org-hide-block-toggle + #'ignore)))) ; this should not happen + (if (and (equal (org-element-type el) (org-element-type newel)) + (equal (marker-position (org-element-property :begin el)) + (org-element-property :begin newel)) + (equal (marker-position (org-element-property :end el)) + (org-element-property :end newel))) + (when (text-property-any (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + 'invisible spec) + (goto-char (org-element-property :begin newel)) + (if (memq this-command org-track-element-modification-default-sensitive-commands) + ;; reveal if change was made by typing + (funcall toggle-func 'off) + ;; re-hide the inserted text + ;; FIXME: opening the drawer before hiding should not be needed here + (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis + (funcall toggle-func 'hide))) + ;; The element was destroyed. Reveal everything. + (org-flag-region (marker-position (org-element-property :begin el)) + (marker-position (org-element-property :end el)) + nil spec) + (when newel + (org-flag-region (org-element-property :begin newel) + (org-element-property :end newel) + nil spec)))))))) + +(defun org--before-element-change-function (beg end) + "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END." + (save-match-data + (save-excursion + (save-restriction + (widen) + (dolist (el (org--find-elements-in-region beg + end + (mapcar #'car org-track-element-modifications) + 'include-partial + 'include-neighbours)) + (let* ((beg-marker (copy-marker (org-element-property :begin el) 't)) + (end-marker (copy-marker (org-element-property :end el) 't))) + (when (and (marker-position beg-marker) (marker-position end-marker)) + (org-element-put-property el :begin beg-marker) + (org-element-put-property el :end end-marker) + (add-to-list 'org--modified-elements el)))))))) + +;; FIXME: this function may be called many times during routine modifications +;; The normal way to avoid this is `combine-after-change-calls' - not +;; the case in most org primitives. +(defun org--after-element-change-function (&rest _) + "Handle changed elements from `org--modified-elements'." + (dolist (el org--modified-elements) + (save-match-data + (save-excursion + (save-restriction + (widen) + (when-let* ((type (org-element-type el)) + (change-func (plist-get (alist-get type org-track-element-modifications) + :after-change-function))) + (with-demoted-errors + (funcall (symbol-function change-func) el))))))) + (setq org--modified-elements nil)) + (defvar org-mode-map) (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param. (defvar org-agenda-keep-modes nil) ; Dynamically-scoped param. @@ -4818,6 +5194,9 @@ The following commands are available: ;; Activate before-change-function (setq-local org-table-may-need-update t) (add-hook 'before-change-functions 'org-before-change-function nil 'local) + (add-hook 'before-change-functions 'org--before-element-change-function nil 'local) + ;; Activate after-change-function + (add-hook 'after-change-functions 'org--after-element-change-function nil 'local) ;; Check for running clock before killing a buffer (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local) ;; Initialize macros templates. @@ -4869,6 +5248,10 @@ The following commands are available: (setq-local outline-isearch-open-invisible-function (lambda (&rest _) (org-show-context 'isearch))) + ;; Make isearch search in blocks hidden via text properties + (setq-local isearch-filter-predicate #'org--isearch-filter-predicate) + (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local) + ;; Setup the pcomplete hooks (setq-local pcomplete-command-completion-function #'org-pcomplete-initial) (setq-local pcomplete-command-name-function #'org-command-at-point) @@ -5050,8 +5433,8 @@ stacked delimiters is N. Escaping delimiters is not possible." (when verbatim? (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 2) (match-end 2) - '(display t invisible t intangible t))) + (org-remove-text-properties (match-beginning 2) (match-end 2) + '(display t invisible t intangible t))) (add-text-properties (match-beginning 2) (match-end 2) '(font-lock-multiline t org-emphasis t)) (when (and org-hide-emphasis-markers @@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links." (if (not (eq 'bracket style)) (add-text-properties start end properties) ;; Handle invisible parts in bracket links. - (remove-text-properties start end '(invisible nil)) + (org-remove-text-properties start end '(invisible nil)) (let ((hidden (append `(invisible ,(or (org-link-get-parameter type :display) @@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links." (defun org-activate-code (limit) (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) t)) (defcustom org-src-fontify-natively t @@ -5258,8 +5641,8 @@ by a #." (setq block-end (match-beginning 0)) ; includes the final newline. (when quoting (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline) - (remove-text-properties beg end-of-endline - '(display t invisible t intangible t))) + (org-remove-text-properties beg end-of-endline + '(display t invisible t intangible t))) (add-text-properties beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) (org-remove-flyspell-overlays-in beg bol-after-beginline) @@ -5313,8 +5696,8 @@ by a #." '(font-lock-fontified t face org-document-info)))) ((string-prefix-p "+caption" dc1) (org-remove-flyspell-overlays-in (match-end 2) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) ;; Handle short captions. (save-excursion (beginning-of-line) @@ -5336,8 +5719,8 @@ by a #." '(font-lock-fontified t face font-lock-comment-face))) (t ;; just any other in-buffer setting, but not indented (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) - (remove-text-properties (match-beginning 0) (match-end 0) - '(display t invisible t intangible t)) + (org-remove-text-properties (match-beginning 0) (match-end 0) + '(display t invisible t intangible t)) (add-text-properties beg (match-end 0) '(font-lock-fontified t face org-meta-line)) t)))))) @@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group." (inhibit-modification-hooks t) deactivate-mark buffer-file-name buffer-file-truename) (decompose-region beg end) - (remove-text-properties beg end - '(mouse-face t keymap t org-linked-text t - invisible t intangible t - org-emphasis t)) + (org-remove-text-properties beg end + '(mouse-face t keymap t org-linked-text t + invisible t + intangible t + org-emphasis t)) (org-remove-font-lock-display-properties beg end))) (defconst org-script-display '(((raise -0.3) (height 0.7)) @@ -5970,6 +6354,29 @@ open and agenda-wise Org files." ;;;; Headlines visibility +(defun org-hide-entry () + "Hide the body directly following this heading." + (interactive) + (save-excursion + (outline-back-to-heading) + (outline-end-of-heading) + (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline))) + +(defun org-hide-subtree () + "Hide everything after this heading at deeper levels." + (interactive) + (org-flag-subtree t)) + +(defun org-hide-sublevels (levels) + "Hide everything but the top LEVELS levels of headers, in whole buffer. +This also unhides the top heading-less body, if any. + +Interactively, the prefix argument supplies the value of LEVELS. +When invoked without a prefix argument, LEVELS defaults to the level +of the current heading, or to 1 if the current line is not a heading." + (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region)) + (org-hide-sublevels levels))) + (defun org-show-entry () "Show the body directly following this heading. Show the heading too, if it is currently invisible." @@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible." 'outline) (org-cycle-hide-property-drawers 'children)))) +(defun org-show-heading () + "Show the current heading and move to its end." + (org-flag-region (- (point) + (if (bobp) 0 + (if (and outline-blank-line + (eq (char-before (1- (point))) ?\n)) + 2 1))) + (progn (outline-end-of-heading) (point)) + nil)) + (defun org-show-children (&optional level) "Show all direct subheadings of this heading. Prefix arg LEVEL is how many levels below the current level @@ -6031,6 +6448,11 @@ heading to appear." (org-flag-region (point) (save-excursion (org-end-of-subtree t t)) nil 'outline)) +(defun org-show-branches () + "Show all subheadings of this heading, but not their bodies." + (interactive) + (org-show-children 1000)) + ;;;; Blocks and drawers visibility (defun org--hide-wrapper-toggle (element category force no-error) @@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful." (unless (let ((eol (line-end-position))) (and (> eol start) (/= eol end))) (let* ((spec (cond ((eq category 'block) 'org-hide-block) - ((eq type 'property-drawer) 'outline) - (t 'org-hide-drawer))) + ((eq category 'drawer) 'org-hide-drawer) + (t 'outline))) (flag (cond ((eq force 'off) nil) (force t) @@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of (when (org-at-property-drawer-p) (let* ((case-fold-search t) (end (re-search-forward org-property-end-re))) - ;; Property drawers use `outline' invisibility spec - ;; so they can be swallowed once we hide the - ;; outline. - (org-flag-region start end t 'outline))))))))))) + (org-flag-region start end t 'org-hide-drawer))))))))))) ;;;; Visibility cycling @@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level." (org-narrow-to-subtree) (org-content)))) ((or "all" "showall") - (outline-show-subtree)) + (org-show-subtree)) (_ nil))) (org-end-of-subtree))))))) @@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'." (while (re-search-forward re nil t) (when (and (not (org-invisible-p)) (org-invisible-p (line-end-position))) - (outline-hide-entry)))) + (org-hide-entry)))) (org-cycle-hide-property-drawers 'all) (org-cycle-show-empty-lines 'overview))))) @@ -6683,8 +7102,13 @@ information." ;; expose it. (dolist (o (overlays-at (point))) (when (memq (overlay-get o 'invisible) - '(org-hide-block org-hide-drawer outline)) + '(outline)) (delete-overlay o))) + (when (memq (get-text-property (point) 'invisible) + '(org-hide-block org-hide-drawer)) + (let ((spec (get-text-property (point) 'invisible)) + (region (org--find-text-property-region (point) 'invisible))) + (org-flag-region (car region) (cdr region) nil spec))) (unless (org-before-first-heading-p) (org-with-limited-levels (cl-case detail @@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard." (skip-chars-forward " \t\n\r") (setq beg (point)) (when (and (org-invisible-p) visp) - (save-excursion (outline-show-heading))) + (save-excursion (org-show-heading))) ;; Shift if necessary. (unless (= shift 0) (save-restriction @@ -8103,7 +8527,7 @@ function is being called interactively." (point)) what "children") (goto-char start) - (outline-show-subtree) + (org-show-subtree) (outline-next-heading)) (t ;; we will sort the top-level entries in this file @@ -13150,7 +13574,7 @@ drawer is immediately hidden." (inhibit-read-only t)) (unless (bobp) (insert "\n")) (insert ":PROPERTIES:\n:END:") - (org-flag-region (line-end-position 0) (point) t 'outline) + (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer) (when (or (eobp) (= begin (point-min))) (insert "\n")) (org-indent-region begin (point)))))) @@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer." (defun org-show-branches-buffer () "Show all branches in the buffer." (org-flag-above-first-heading) - (outline-hide-sublevels 1) + (org-hide-sublevels 1) (unless (eobp) - (outline-show-branches) + (org-show-branches) (while (outline-get-next-sibling) - (outline-show-branches))) + (org-show-branches))) (goto-char (point-min))) (defun org-kill-note-or-show-branches () @@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer." (t (let ((beg (progn (org-back-to-heading) (point))) (end (save-excursion (org-end-of-subtree t t) (point)))) - (outline-hide-subtree) - (outline-show-branches) + (org-hide-subtree) + (org-show-branches) (org-hide-archived-subtrees beg end))))) (defun org-delete-indentation (&optional arg) @@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'. ARG is the level to hide." (if (org-before-first-heading-p) (progn (org-flag-above-first-heading) - (outline-hide-sublevels (or arg 1)) + (org-hide-sublevels (or arg 1)) (goto-char (point-min))) - (outline-hide-subtree) + (org-hide-subtree) (org-show-children arg)))) (defun org-ctrl-c-star () @@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'." (t default-org-info-node)))))) + +;;; Make isearch search in some text hidden via text propertoes + +(defvar org--isearch-overlays nil + "List of overlays temporarily created during isearch. +This is used to allow searching in regions hidden via text properties. +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays. +Any text hidden via text properties is not revealed even if `search-invisible' +is set to 't.") + +;; Not sure if it needs to be a user option +;; One might want to reveal hidden text in, for example, hidden parts of the links. +;; Currently, hidden text in links is never revealed by isearch. +(defvar org-isearch-specs '(org-hide-block + org-hide-drawer) + "List of text invisibility specs to be searched by isearch. +By default ([2020-05-09 Sat]), isearch does not search in hidden text, +which was made invisible using text properties. Isearch will be forced +to search in hidden text with any of the listed 'invisible property value.") + +(defun org--create-isearch-overlays (beg end) + "Replace text property invisibility spec by overlays between BEG and END. +All the regions with invisibility text property spec from +`org-isearch-specs' will be changed to use overlays instead +of text properties. The created overlays will be stored in +`org--isearch-overlays'." + (let ((pos beg)) + (while (< pos end) + (when-let* ((spec (get-text-property pos 'invisible)) + (spec (memq spec org-isearch-specs)) + (region (org--find-text-property-region pos 'invisible))) + (setq spec (get-text-property pos 'invisible)) + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat] + ;; overlay for 'outline blocks. + (let ((o (make-overlay (car region) (cdr region) nil 'front-advance))) + (overlay-put o 'evaporate t) + (overlay-put o 'invisible spec) + ;; `delete-overlay' here means that spec information will be lost + ;; for the region. The region will remain visible. + (overlay-put o 'isearch-open-invisible #'delete-overlay) + (push o org--isearch-overlays)) + (org-flag-region (car region) (cdr region) nil spec))) + (setq pos (next-single-property-change pos 'invisible nil end))))) + +(defun org--isearch-filter-predicate (beg end) + "Return non-nil if text between BEG and END is deemed visible by Isearch. +This function is intended to be used as `isearch-filter-predicate'. +Unlike `isearch-filter-visible', make text with 'invisible text property +value listed in `org-isearch-specs' visible to Isearch." + (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text + (isearch-filter-visible beg end)) + +(defun org--clear-isearch-overlay (ov) + "Convert OV region back into using text properties." + (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays + ;; Changing text properties is considered buffer modification. + ;; We do not want it here. + (with-silent-modifications + (org-flag-region (overlay-start ov) (overlay-end ov) t spec))) + (when (member ov isearch-opened-overlays) + (setq isearch-opened-overlays (delete ov isearch-opened-overlays))) + (delete-overlay ov)) + +(defun org--clear-isearch-overlays () + "Convert overlays from `org--isearch-overlays' back into using text properties." + (when org--isearch-overlays + (mapc #'org--clear-isearch-overlay org--isearch-overlays) + (setq org--isearch-overlays nil))) + + + ;;; Finish up (add-hook 'org-mode-hook ;remove overlays when changing major mode --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ihor Radchenko writes: > Hello, > > [The patch itself will be provided in the following email] > > I have five updates from the previous version of the patch: > > 1. I implemented a simplified version of element parsing to detect > changes in folded drawers or blocks. No computationally expensive calls > of org-element-at-point or org-element-parse-buffer are needed now. > > 2. The patch is now compatible with master (commit 2e96dc639). I > reverted the earlier change in folding drawers and blocks. Now, they are > back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would > achieve nothing when we use text properties. > > 3. 'invisible text property can now be nested. This is important, for > example, when text inside drawers contains fontified links (which also > use 'invisible text property to hide parts of the link). Now, the old > 'invisible spec is recovered after unfolding. > > 4. Some outline-* function calls in org referred to outline-flag-region > implementation, which is not in sync with org-flag-region in this patch. > I have implemented their org-* versions and replaced the calls > throughout .el files. Actually, some org-* versions were already > implemented in org, but not used for some reason (or not mentioned in > the manual). I have updated the relevant sections of manual. These > changes might be relevant to org independently of this feature branch. > > 5. I have managed to get a working version of outline folding via text > properties. However, that approach has a big downside - folding state > cannot be different in indirect buffer when we use text properties. I > have seen packages relying on this feature of org and I do not see any > obvious way to achieve different folding state in indirect buffer while > using text properties for outline folding. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the new implementation for tracking changes: > >> Of course we can. It is only necessary to focus on changes that would >> break the structure of the element. This does not entail a full parsing. > > I have limited parsing to matching beginning and end of a drawer/block. > The basic functions are org--get-element-region-at-point, > org--get-next-element-region-at-point, and org--find-elements-in-region. > They are simplified versions of org-element-* parsers and do not require > parsing everything from the beginning of the section. > > For now, I keep everything in org.el, but those simplified parsers > probably belong to org-element.el. > >> If we can stick with `after-change-functions' (or local equivalent), >> that's better. It is more predictable than `before-change-functions' and >> alike. > > For now, I still used before/after-change-functions combination. > I see the following problems with using only after-change-functions:=20 > > 1. They are not guaranteed to be called after every single change: > > From (elisp) Change Hooks: > "... some complex primitives call =E2=80=98before-change-functions=E2=80= =99 once before > making changes, and then call =E2=80=98after-change-functions=E2=80=99 ze= ro or more > times" > > The consequence of it is a possibility that region passed to the > after-change-functions is quite big (including all the singular changes, > even if they are distant). This region may contain changed drawers as > well and unchanged drawers and needs to be parsed to determine which > drawers need to be re-folded. > >> And, more importantly, they are not meant to be used together, i.e., you >> cannot assume that a single call to `before-change-functions' always >> happens before calling `after-change-functions'. This can be tricky if >> you want to use the former to pass information to the latter. > > The fact that before-change-functions can be called multiple times > before after-change-functions, is trivially solved by using buffer-local > changes register (see org--modified-elements). The register is populated > by before-change-functions and cleared by after-change-functions. > >> Well, `before-change-fuctions' and `after-change-functions' are not >> clean at all: you modify an unrelated part of the buffer, but still call >> those to check if a drawer needs to be unfolded somewhere. > > 2. As you pointed, instead of global before-change-functions, we can use > modification-hooks text property on sensitive parts of the > drawers/blocks. This would work, but I am concerned about one annoying > special case: > > ------------------------------------------------------------------------- > :BLAH: > > > > :DRAWER: > Donec at pede. > :END: > ------------------------------------------------------------------------- > In this example, the user would not be able to unfold the folder DRAWER > because it will technically become a part of a new giant BLAH drawer. > This may be especially annoying if is more than one screen > long and there is no easy way to identify why unfolding does not work > (with point at :DRAWER:). > > Because of this scenario, limiting before-change-functions to folded > drawers is not sufficient. Any change in text may need to trigger > unfolding. > > In the patch, I always register possible modifications in the > blocks/drawers intersecting with the modified region + a drawer/block > right next to the region. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on the nested 'invisible text property implementation. > > The idea is to keep 'invisible property stack push and popping from it > as we add/remove 'invisible text property. All the work is done in > org-flag-region. > > This was originally intended for folding outlines via text properties. > Since using text properties for folding outlines is not a good idea, > nested text properties have much less use. As I mentioned, they do > preserve link fontification, but I am not sure if it worth it for the > overhead to org-flag-region. Keeping this here mostly in the case if > someone has any ideas how it can be useful. > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > More details on replaced outline-* -> org-* function calls. > > I have implemented org-* versions of the following functions: > > - outline-hide-entry > - outline-hide-subtree > - outline-hide-sublevels > - outline-show-heading > - outline-show-branches > > The org-* versions trivially use org-flag-region instead of > outline-flag-region. > > Replaced outline-* calls where org- versions were already available: > > - outline-show-entry > - outline-show-all > - outline-show-subtree > > I reflected the new (including already available) functions in the > manual and removed some defalias from org-compat.el where they are not > needed.=20 > > ----------------------------------------------------------------------- > ----------------------------------------------------------------------- > > Further work: > > 1. after-change-functions use org-hide-drawer/block-toggle to > fold/unfold after modification. However, I just found that they call > org-element-at-point, which slows down modifications in folded > drawers/blocks. For example, realigning a long table inside folded > drawer takes >1sec, while it is instant in the unfolded drawer. > > 2. org-toggle-custom-properties is terribly slow on large org documents, > similarly to folded drawers on master. It should be relatively easy to > use text properties there instead of overlays. > > 3. Multiple calls to before/after-change-functions is still a problem. I > am looking into following ways to reduce this number: > - reduce the number of elements registered as potentially modified > + do not add duplicates to org--modified-elements > + do not add unfolded elements to org--modified-elements > + register after-change-function as post-command hook and remove it > from global after-change-functions. This way, it will be called > twice per command only. > - determine common region containing org--modified-elements. if change > is happening within that region, there is no need to parse > drawers/blocks there again. > > P.S. > >>> It was mostly an annoyance, because they returned different results on >>> the same element. Specifically, they returned different :post-blank and >>> :end properties, which does not sound right. >> >> OK. If you have a reproducible recipe, I can look into it and see what >> can be done. > > Recipe to have different (org-element-at-point) and > (org-element-parse-buffer 'element) > ------------------------------------------------------------------------- > > :PROPERTIES: > :CREATED: [2020-05-23 Sat 02:32] > :END: > > > > ------------------------------------------------------------------------- > > > Best, > Ihor > > Nicolas Goaziou writes: > >> Hello, >> >> Ihor Radchenko writes: >> >>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an >>>> element is a O(N) operation by the number of elements before it in >>>> a section. In particular, it is not bounded, and not mitigated by >>>> a cache. For large documents, it is going to be unbearably slow, too. >>> >>> Ouch. I thought it is faster. >>> What do you mean by "not mitigated by a cache"? >> >> Parsing starts from the closest headline, every time. So, if Org parses >> the Nth element in the entry two times, it really parses 2N elements. >> >> With a cache, assuming the buffer wasn't modified, Org would parse >> N elements only. With a smarter cache, with fine grained cache >> invalidation, it could also reduce the number of subsequent parsed >> elements. >> >>> The reason I would like to utilise org-element parser to make tracking >>> modifications more robust. Using details of the syntax would make the >>> code fragile if any modifications are made to syntax in future. >> >> I don't think the code would be more fragile. Also, the syntax we're >> talking about is not going to be modified anytime soon. Moreover, if >> folding breaks, it is usually visible, so the bug will not be unnoticed. >> >> This code is going to be as low-level as it can be. >> >>> Debugging bugs in modification functions is not easy, according to my >>> experience. >> >> No, it's not.=20 >> >> But this is not really related to whether you use Element or not. >> >>> One possible way to avoid performance issues during modification is >>> running parser in advance. For example, folding an element may >>> as well add information about the element to its text properties. >>> This will not degrade performance of folding since we are already >>> parsing the element during folding (at least, in >>> org-hide-drawer-toggle). >> >> We can use this information stored at fold time. But I'm not even sure >> we need it. >> >>> The problem with parsing an element during folding is that we cannot >>> easily detect changes like below without re-parsing. >> >> Of course we can. It is only necessary to focus on changes that would >> break the structure of the element. This does not entail a full parsing. >> >>> :PROPERTIES: >>> :CREATED: [2020-05-18 Mon] >>> :END: <- added line >>> :ID: test >>> :END: >>> >>> or even >>> >>> :PROPERTIES: >>> :CREATED: [2020-05-18 Mon] >>> :ID: test >>> :END: <- delete this line >>> >>> :DRAWER: >>> test >>> :END: >> >> Please have a look at the "sensitive parts" I wrote about. This takes >> care of this kind of breakage. >> >>> The re-parsing can be done via regexp, as you suggested, but I don't >>> like this idea, because it will end up re-implementing >>> org-element-*-parser. >> >> You may have misunderstood my suggestion. See below. >> >>> Would it be acceptable to run org-element-*-parser >>> in after-change-functions? >> >> I'd rather not do that. This is unnecessary consing, and matching, etc. >> >>> If I understand correctly, it is not as easy. >>> Consider the following example: >>> >>> :PROPERTIES: >>> :CREATED: [2020-05-18 Mon] >>> >>> :ID: example >>> :END: >>> >>> <... a lot of text, maybe containing other drawers ...> >>> >>> Nullam rutrum. >>> Pellentesque dapibus suscipit ligula. >>> >>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus. >>> >>> If the region gets deleted, the modification hooks from chars inside >>> drawer will be called as (hook-function >>> ). So, there is still a need to find the drawer somehow to >>> mark it as about to be modified (modification hooks are ran before >>> actual modification). >> >> If we can stick with `after-change-functions' (or local equivalent), >> that's better. It is more predictable than `before-change-functions' and >> alike. >> >> If it is a deletion, here is the kind of checks we could do, depending >> on when they are performed. >> >> Before actual changes : >> >> 1. The deletion is happening within a folded drawer (unnecessary step >> in local functions). >> 2. The change deleted the sensitive line ":END:". >> 3. Conclusion : unfold. >> >> Or, after actual changes : >> >> 1. The deletion involves a drawer. >> 2. Text properties indicate that the beginning of the propertized part >> of the buffer start with org-drawer-regexp, but doesn't end with >> `org-property-end-re'. A "sensitive part" disappeared! >> 3. Conclusion : unfold >> >> This is far away from parsing. IMO, a few checks cover all cases. Let me >> know if you have questions about it. >> >> Also, note that the kind of change you describe will happen perhaps >> 0.01% of the time. Most change are about one character, or a single >> line, long. >> >>> The only difference between using modification hooks and >>> before-change-functions is that modification hooks will trigger less >>> frequently.=20 >> >> Exactly. Much less frequently. But extra care is required, as you noted >> already. >> >>> Considering the performance of org-element-at-point, it is >>> probably worth doing. Initially, I wanted to avoid it because setting a >>> single before-change-functions hook sounded cleaner than setting >>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks. >> >> Well, `before-change-fuctions' and `after-change-functions' are not >> clean at all: you modify an unrelated part of the buffer, but still call >> those to check if a drawer needs to be unfolded somewhere. >> >> And, more importantly, they are not meant to be used together, i.e., you >> cannot assume that a single call to `before-change-functions' always >> happens before calling `after-change-functions'. This can be tricky if >> you want to use the former to pass information to the latter. >> >> But I understand that they are easier to use than their local >> counterparts. If you stick with (before|after)-change-functions, the >> function being called needs to drop the ball very quickly if the >> modification is not about folding changes. Also, I very much suggest to >> stick to only `after-change-functions', if feasible (I think it is), per >> above. >> >>> Moreover, these text properties would be copied by default if one uses= =20 >>> buffer-substring. Then, the hooks will also trigger later in the yanked >>> text, which may cause all kinds of bugs. >> >> Indeed, that would be something to handle specifically. I.e., >> destructive modifications (i.e., those that unfold) could clear such >> properties. >> >> It is more work. I don't know if it is worth the trouble if we can get >> out quickly of `after-change-functions' for unrelated changes. >> >>> It was mostly an annoyance, because they returned different results on >>> the same element. Specifically, they returned different :post-blank and >>> :end properties, which does not sound right. >> >> OK. If you have a reproducible recipe, I can look into it and see what >> can be done. >> >> Regards, >> >> --=20 >> Nicolas Goaziou > > --=20 > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong= University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg --=20 Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong U= niversity, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg --=-=-=--