Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Ihor Radchenko <yantar92@gmail.com>]

Github link to the patch:
https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef 


Ihor Radchenko <yantar92@gmail.com> writes:

> The patch is attached
>
> diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
> index 9f8677871..ab470ea9b 100644
> --- a/contrib/lisp/org-notify.el
> +++ b/contrib/lisp/org-notify.el
> @@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
>            (switch-to-buffer (find-file-noselect file))
>            (org-with-wide-buffer
>             (goto-char begin)
> -           (outline-show-entry))
> +           (org-show-entry))
>            (goto-char begin)
>            (search-forward "DEADLINE: <")
>            (search-forward ":")
> diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
> index bfc4d6c3e..2312b235c 100644
> --- a/contrib/lisp/org-velocity.el
> +++ b/contrib/lisp/org-velocity.el
> @@ -325,7 +325,7 @@ use it."
>    (save-excursion
>      (when narrow
>        (org-narrow-to-subtree))
> -    (outline-show-all)))
> +    (org-show-all)))
>  
>  (defun org-velocity-edit-entry/inline (heading)
>    "Edit entry at HEADING in the original buffer."
> diff --git a/doc/org-manual.org b/doc/org-manual.org
> index 96b160175..2ebe94538 100644
> --- a/doc/org-manual.org
> +++ b/doc/org-manual.org
> @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
>    Switch back to the startup visibility of the buffer (see [[*Initial
>    visibility]]).
>  
> -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
> +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
>  
>    #+cindex: show all, command
>    #+kindex: C-u C-u C-u TAB
> -  #+findex: outline-show-all
> +  #+findex: org-show-all
>    Show all, including drawers.
>  
>  - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
> @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
>    headings.  With a double prefix argument, also show the entire
>    subtree of the parent.
>  
> -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
> +- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
>  
>    #+cindex: show branches, command
>    #+kindex: C-c C-k
> -  #+findex: outline-show-branches
> +  #+findex: org-show-branches
>    Expose all the headings of the subtree, but not their bodies.
>  
> -- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
> +- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
>  
>    #+cindex: show children, command
>    #+kindex: C-c TAB
> -  #+findex: outline-show-children
> +  #+findex: org-show-children
>    Expose all direct children of the subtree.  With a numeric prefix
>    argument {{{var(N)}}}, expose all children down to level
>    {{{var(N)}}}.
> @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
>    command (see [[*Visibility Cycling]]).  You can force cycling archived
>    subtrees with {{{kbd(C-TAB)}}}, or by setting the option
>    ~org-cycle-open-archived-trees~.  Also normal outline commands, like
> -  ~outline-show-all~, open archived subtrees.
> +  ~org-show-all~, open archived subtrees.
>  
>  -
>    #+vindex: org-sparse-tree-open-archived-trees
> diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
> index ab13f926c..ad9244940 100644
> --- a/lisp/org-agenda.el
> +++ b/lisp/org-agenda.el
> @@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'."
>  	    (t "  %-12:c%?-12t% s")))
>  	(start 0)
>  	varform vars var e c f opt)
> -    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
> +    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
>  			 s start)
>        (setq var (or (cdr (assoc (match-string 4 s)
>  				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
> @@ -9138,20 +9138,20 @@ if it was hidden in the outline."
>       ((and (called-interactively-p 'any) (= more 1))
>        (message "Remote: show with default settings"))
>       ((= more 2)
> -      (outline-show-entry)
> +      (org-show-entry)
>        (org-show-children)
>        (save-excursion
>  	(org-back-to-heading)
>  	(run-hook-with-args 'org-cycle-hook 'children))
>        (message "Remote: CHILDREN"))
>       ((= more 3)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (save-excursion
>  	(org-back-to-heading)
>  	(run-hook-with-args 'org-cycle-hook 'subtree))
>        (message "Remote: SUBTREE"))
>       ((> more 3)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (message "Remote: SUBTREE AND ALL DRAWERS")))
>      (select-window win)))
>  
> diff --git a/lisp/org-archive.el b/lisp/org-archive.el
> index d3e12d17b..d864dad8a 100644
> --- a/lisp/org-archive.el
> +++ b/lisp/org-archive.el
> @@ -330,7 +330,7 @@ direct children of this heading."
>  		      (insert (if datetree-date "" "\n") heading "\n")
>  		      (end-of-line 0))
>  		    ;; Make the subtree visible
> -		    (outline-show-subtree)
> +		    (org-show-subtree)
>  		    (if org-archive-reversed-order
>  			(progn
>  			  (org-back-to-heading t)
> diff --git a/lisp/org-colview.el b/lisp/org-colview.el
> index e50a4d7c8..e656df555 100644
> --- a/lisp/org-colview.el
> +++ b/lisp/org-colview.el
> @@ -699,7 +699,7 @@ FUN is a function called with no argument."
>  			  (move-beginning-of-line 2)
>  			  (org-at-heading-p t)))))
>      (unwind-protect (funcall fun)
> -      (when hide-body (outline-hide-entry)))))
> +      (when hide-body (org-hide-entry)))))
>  
>  (defun org-columns-previous-allowed-value ()
>    "Switch to the previous allowed value for this column."
> diff --git a/lisp/org-compat.el b/lisp/org-compat.el
> index 635a38dcd..8fe271896 100644
> --- a/lisp/org-compat.el
> +++ b/lisp/org-compat.el
> @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer."
>  ;;; Emacs < 25.1 compatibility
>  
>  (when (< emacs-major-version 25)
> -  (defalias 'outline-hide-entry 'hide-entry)
> -  (defalias 'outline-hide-sublevels 'hide-sublevels)
> -  (defalias 'outline-hide-subtree 'hide-subtree)
>    (defalias 'outline-show-branches 'show-branches)
>    (defalias 'outline-show-children 'show-children)
> -  (defalias 'outline-show-entry 'show-entry)
>    (defalias 'outline-show-subtree 'show-subtree)
>    (defalias 'xref-find-definitions 'find-tag)
>    (defalias 'format-message 'format)
> diff --git a/lisp/org-element.el b/lisp/org-element.el
> index ac41b7650..2d5c8d771 100644
> --- a/lisp/org-element.el
> +++ b/lisp/org-element.el
> @@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
>  TYPE is the type of the current element or object.
>  
>  If PARENT? is non-nil, assume the next element or object will be
> -located inside the current one.  "
> +located inside the current one."
>    (if parent?
>        (pcase type
>  	(`headline 'section)
> diff --git a/lisp/org-keys.el b/lisp/org-keys.el
> index c006e9c12..deb5d7b90 100644
> --- a/lisp/org-keys.el
> +++ b/lisp/org-keys.el
> @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
>    #'org-next-visible-heading)
>  (define-key org-mode-map [remap outline-previous-visible-heading]
>    #'org-previous-visible-heading)
> -(define-key org-mode-map [remap show-children] #'org-show-children)
> +(define-key org-mode-map [remap outline-show-children] #'org-show-children)
>  
>  ;;;; Make `C-c C-x' a prefix key
>  (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
> diff --git a/lisp/org-macs.el b/lisp/org-macs.el
> index a02f713ca..fa0a658f0 100644
> --- a/lisp/org-macs.el
> +++ b/lisp/org-macs.el
> @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
>  
>  
>  \f
> -;;; Overlays
> +;;; Overlays and text properties
>  
>  (defun org-overlay-display (ovl text &optional face evap)
>    "Make overlay OVL display TEXT with face FACE."
> @@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays."
>  	    (delete (delete-overlay ov))
>  	    (t (push ov found))))))
>  
> +(defun org-remove-text-properties (start end properties &optional object)
> +  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
> +Do not remove invisible text properties specified by 'outline,
> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
> +is needed to keep outlines, drawers, and blocks hidden unless they are
> +toggled by user.
> +Note: The below may be too specific and create troubles if more
> +invisibility specs are added to org in future"
> +  (when (plist-member properties 'invisible)
> +    (let ((pos start)
> +	  next spec)
> +      (while (< pos end)
> +	(setq next (next-single-property-change pos 'invisible nil end)
> +              spec (get-text-property pos 'invisible))
> +	(unless (memq spec (list 'org-hide-block
> +				 'org-hide-drawer
> +				 'outline))
> +          (remove-text-properties pos next '(invisible nil) object))
> +	(setq pos next))))
> +  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
> +    (remove-text-properties start end properties-stripped object)))
> +
> +(defun org--find-text-property-region (pos prop)
> +  "Find a region containing PROP text property around point POS."
> +  (let* ((beg (and (get-text-property pos prop) pos))
> +	 (end beg))
> +    (when beg
> +      ;; when beg is the first point in the region, `previous-single-property-change'
> +      ;; will return nil.
> +      (setq beg (or (previous-single-property-change pos prop)
> +		    beg))
> +      ;; when end is the last point in the region, `next-single-property-change'
> +      ;; will return nil.
> +      (setq end (or (next-single-property-change pos prop)
> +		    end))
> +      (unless (= beg end) ; this should not happen
> +        (cons beg end)))))
> +
>  (defun org-flag-region (from to flag spec)
>    "Hide or show lines from FROM to TO, according to FLAG.
>  SPEC is the invisibility spec, as a symbol."
> -  (remove-overlays from to 'invisible spec)
> -  ;; Use `front-advance' since text right before to the beginning of
> -  ;; the overlay belongs to the visible line than to the contents.
> -  (when flag
> -    (let ((o (make-overlay from to nil 'front-advance)))
> -      (overlay-put o 'evaporate t)
> -      (overlay-put o 'invisible spec)
> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> -
> +  (pcase spec
> +    ('outline
> +     (remove-overlays from to 'invisible spec)
> +     ;; Use `front-advance' since text right before to the beginning of
> +     ;; the overlay belongs to the visible line than to the contents.
> +     (when flag
> +       (let ((o (make-overlay from to nil 'front-advance)))
> +	 (overlay-put o 'evaporate t)
> +	 (overlay-put o 'invisible spec)
> +	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> +    (_
> +     ;; Use text properties instead of overlays for speed.
> +     ;; Overlays are too slow (Emacs Bug#35453).
> +     (with-silent-modifications
> +       ;; keep a backup stack of old text properties
> +       (save-excursion
> +	 (goto-char from)
> +	 (while (< (point) to)
> +	   (let ((old-spec (get-text-property (point) 'invisible))
> +		 (end (next-single-property-change (point) 'invisible nil to)))
> +	     (when old-spec
> +	       (alter-text-property (point) end 'org-property-stack-invisible
> +				    (lambda (stack)
> +				      (if (or (eq old-spec (car stack))
> +					      (eq spec old-spec)
> +					      (eq old-spec 'outline))
> +					  stack
> +					(cons old-spec stack)))))
> +	     (goto-char end))))
> +
> +       ;; cleanup everything
> +       (remove-text-properties from to '(invisible nil))
> +
> +       ;; Recover properties from the backup stack
> +       (unless flag
> +	 (save-excursion
> +	   (goto-char from)
> +	   (while (< (point) to)
> +             (let ((stack (get-text-property (point) 'org-property-stack-invisible))
> +		   (end (next-single-property-change (point) 'org-property-stack-invisible nil to)))
> +               (if (not stack)
> +		   (remove-text-properties (point) end '(org-property-stack-invisible nil))
> +		 (put-text-property (point) end 'invisible (car stack))
> +		 (alter-text-property (point) end 'org-property-stack-invisible
> +				      (lambda (stack)
> +					(cdr stack))))
> +               (goto-char end)))))
> +       
> +       (when flag
> +	 (put-text-property from to 'rear-non-sticky nil)
> +	 (put-text-property from to 'front-sticky t)
> +	 (put-text-property from to 'invisible spec))))))
>  
>  \f
>  ;;; Regexp matching
> diff --git a/lisp/org-src.el b/lisp/org-src.el
> index c9eef744e..e89a1c580 100644
> --- a/lisp/org-src.el
> +++ b/lisp/org-src.el
> @@ -523,8 +523,8 @@ Leave point in edit buffer."
>  	(org-src-switch-to-buffer buffer 'edit)
>  	;; Insert contents.
>  	(insert contents)
> -	(remove-text-properties (point-min) (point-max)
> -				'(display nil invisible nil intangible nil))
> +	(org-remove-text-properties (point-min) (point-max)
> +				    '(display nil invisible nil intangible nil))
>  	(unless preserve-ind (org-do-remove-indentation))
>  	(set-buffer-modified-p nil)
>  	(setq buffer-file-name nil)
> diff --git a/lisp/org-table.el b/lisp/org-table.el
> index 6462b99c4..75801161b 100644
> --- a/lisp/org-table.el
> +++ b/lisp/org-table.el
> @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
>     (arg
>      (let ((b (save-excursion (skip-chars-backward "^|") (point)))
>  	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
> -      (remove-text-properties b e '(invisible t intangible t))
> +      (org-remove-text-properties b e '(invisible t intangible t))
>        (if (and (boundp 'font-lock-mode) font-lock-mode)
>  	  (font-lock-fontify-block))))
>     (t
> @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
>        (setq word-wrap t)
>        (goto-char (setq p (point-max)))
>        (insert (org-trim field))
> -      (remove-text-properties p (point-max) '(invisible t intangible t))
> +      (org-remove-text-properties p (point-max) '(invisible t intangible t))
>        (goto-char p)
>        (setq-local org-finish-function 'org-table-finish-edit-field)
>        (setq-local org-window-configuration cw)
> diff --git a/lisp/org.el b/lisp/org.el
> index e577dc661..360974135 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  (declare-function cdlatex-math-symbol "ext:cdlatex")
>  (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
>  (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
> +(declare-function isearch-filter-visible "isearch" (beg end))
>  (declare-function org-add-archive-files "org-archive" (files))
>  (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
>  (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
> @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  
>  (defvar ffap-url-regexp)
>  (defvar org-element-paragraph-separate)
> +(defvar org-element-all-objects)
> +(defvar org-element-all-elements)
> +(defvar org-element-greater-elements)
>  (defvar org-indent-indentation-per-level)
>  (defvar org-radio-target-regexp)
>  (defvar org-target-link-regexp)
> @@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.")
>  
>  ;;;; Define the Org mode
>  
> +;;; Handling buffer modifications
> +
>  (defun org-before-change-function (_beg _end)
>    "Every change indicates that a table might need an update."
>    (setq org-table-may-need-update t))
> +
> +(defvar-local org--modified-elements nil
> +  "List of elements, marked as recently modified.
> +There is no guarantee that the elements in this list are fully parsed.
> +Only the element type, :begin and :end properties of the elements are
> +guaranteed to be available. The :begin and :end element properties
> +contain markers instead of positions.")
> +
> +(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command)
> +  "List of commands triggerring element modifications unconditionally.")
> +
> +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$")
> +                                       (property-drawer . ,org-property-start-re)
> +				       (drawer . ,org-drawer-regexp)
> +                                       (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$")
> +                                       (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$"))
> +  "Alist of regexps matching beginning of elements.
> +Group 1 in the regexps (if any) contains the element type.")
> +
> +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$")
> +				 (property-drawer . ,org-property-end-re)
> +				 (drawer . ,org-property-end-re)
> +				 (quote-block . "^[ \t]*#\\+end_quote[ \t]*$")
> +				 (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$"))
> +  "Alist of regexps matching end of elements.
> +Group 1 in the regexps (if any) contains the element type or END.")
> +
> +(defvar org-track-element-modifications
> +  `((property-drawer  . (:after-change-function
> +                         org--drawer-or-block-unfold-maybe))
> +    (drawer  . (:after-change-function
> +                org--drawer-or-block-unfold-maybe))
> +    (center-block . (:after-change-function
> +                     org--drawer-or-block-unfold-maybe))
> +    (quote-block  . (:after-change-function
> +                     org--drawer-or-block-unfold-maybe))
> +    (special-block  . (:after-change-function
> +                       org--drawer-or-block-unfold-maybe)))
> +  "Alist of elements to be tracked for modifications.
> +The modification is only triggered according to :sensitive-re-list and
> +:sensitive-command-list (see below).
> +Each element of the alist is a cons of an element symbol and plist
> +defining how and when the modifications are handled.
> +In case of recursive elements/duplicates, the first element from the
> +list is considered.
> +The plist can have the following properties:
> +- :element-beginning-re   :: regex matching beginning of the element
> +  (default)               :: (alist-get element org--element-beginning-re-alist)
> +- :element-end-re         :: regex matching end of the element
> +  (default)               :: (alist-get element org--element-end-re-alist)
> +- :after-change-function  :: function called after the modification
> +The function must accept a single argument - element from
> +`org--modified-elements'.")
> +
> +(defun org--get-element-region-at-point (types)
> +  "Return TYPES element at point or nil.
> +If TYPES is a list, return first element at point from the list.  The
> +returned value is partially parsed element only containing :begin and
> +:end properties.  Only elements listed in
> +org--element-beginning-re-alist and org--element-end-re-alist can be
> +parsed here."
> +  (catch 'exit
> +    (dolist (type (if (listp types) types (list types)))
> +      (let ((begin-re (alist-get type org--element-beginning-re-alist))
> +	    (end-re (alist-get type org--element-end-re-alist))
> +            (begin-limit (save-excursion (org-with-limited-levels
> +					  (org-back-to-heading-or-point-min 'invisible-ok))
> +					 (point)))
> +            (end-limit (or (save-excursion (outline-next-heading))
> +			   (point-max)))
> +            (point (point))
> +	    begin end closest-begin)
> +	(when (and begin-re end-re)
> +	  (save-excursion
> +	    (end-of-line)
> +	    (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point)))
> +	    (when (re-search-forward end-re end-limit 'noerror) (setq end (point)))
> +            (setq closest-begin begin)
> +            ;; slurp unmatched begin-re
> +	    (when (and begin end)
> +              (goto-char begin)
> +              (while (and (re-search-backward begin-re begin-limit 'noerror)
> +			  (= end (save-excursion (re-search-forward end-re end-limit 'noerror))))
> +		(setq begin (point)))
> +              (when (and (>= point begin) (<= point end))
> +		(throw 'exit
> +		       (list type
> +			     (list
> +			      :begin begin
> +			      :end end)))))))))))
> +
> +(defun org--get-next-element-region-at-point (types &optional limit previous)
> +  "Return TYPES element after point or nil.
> +If TYPES is a list, return first element after point from the list.
> +If PREVIOUS is non-nil, return first TYPES element before point.
> +Limit search by LIMIT or previous/next heading.
> +The returned value is partially parsed element only containing :begin
> +and :end properties.  Only elements listed in
> +org--element-beginning-re-alist and org--element-end-re-alist can be
> +parsed here."
> +  (catch 'exit
> +    (dolist (type (if (listp types) types (list types)))
> +      (let* ((begin-re (alist-get type org--element-beginning-re-alist))
> +	     (end-re (alist-get type org--element-end-re-alist))
> +             (limit (or limit (if previous
> +				  (save-excursion
> +				    (org-with-limited-levels
> +				     (org-back-to-heading-or-point-min 'invisible-ok)
> +				     (point)))
> +				(or (save-excursion (outline-next-heading))
> +				    (point-max)))))
> +	     begin end)
> +	(when (and begin-re end-re)
> +	  (save-excursion
> +            (if previous
> +                (when (re-search-backward begin-re limit 'noerror)
> +		  (when-let ((el (org--get-element-region-at-point type)))
> +		    (setq begin (org-element-property :begin el))
> +		    (setq end (org-element-property :end el))))
> +	      (when (re-search-forward begin-re limit 'noerror)
> +                (when-let ((el (org--get-element-region-at-point type)))
> +		  (setq begin (org-element-property :begin el))
> +		  (setq end (org-element-property :end el))))))
> +	  (when (and begin end)
> +            (throw 'exit
> +		   (list type
> +			 (list
> +			  :begin begin
> +			  :end end)))))))))
> +
> +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours)
> +  "Find all elements from ELEMENTS in region BEG . END.
> +All the listed elements must be resolvable by
> +`org--get-element-region-at-point'.
> +Include elements if they are partially inside region when
> +INCLUDE-PARTIAL is non-nil.
> +Include preceding/subsequent neighbouring elements when no partial
> +element is found at the beginning/end of the region and
> +INCLUDE-NEIGHBOURS is non-nil."
> +  (when include-partial
> +    (org-with-point-at beg
> +      (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements))))
> +	(if new-beg
> +	    (setq beg new-beg)
> +          (when (and include-neighbours
> +		     (setq new-beg (org-element-property :begin
> +						      (org--get-next-element-region-at-point elements
> +											  (point-min)
> +											  'previous))))
> +            (setq beg new-beg))))
> +      (when (memq 'headline elements)
> +	(when-let ((new-beg (save-excursion
> +			      (org-with-limited-levels (outline-previous-heading)))))
> +          (setq beg new-beg))))
> +    (org-with-point-at end
> +      (let ((new-end (org-element-property :end (org--get-element-region-at-point elements))))
> +	(if new-end
> +	    (setq end new-end)
> +          (when (and include-neighbours
> +		     (setq new-end (org-element-property :end
> +						      (org--get-next-element-region-at-point elements (point-max)))))
> +            (setq end new-end))))
> +      (when (memq 'headline elements)
> +	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
> +          (setq end (1- new-end))))))
> +  (save-excursion
> +    (save-restriction
> +      (narrow-to-region beg end)
> +      (goto-char (point-min))
> +      (let (result el)
> +	(while (setq el (org--get-next-element-region-at-point elements end))
> +          (push el result)
> +          (goto-char (org-element-property :end el)))
> +        result))))
> +
> +(defun org--drawer-or-block-unfold-maybe (el)
> +  "Update visibility of modified folded drawer/block EL.
> +If text was added to hidden drawer/block, make sure that the text is
> +also hidden, unless the change was done by a command listed in
> +`org-track-element-modification-default-sensitive-commands'.  If the
> +modification destroyed the drawer/block, reveal the hidden text in
> +former drawer/block.  If the modification shrinked/expanded the
> +drawer/block beyond the hidden text, reveal the affected
> +drawers/blocks as well.
> +Examples:
> +----------------------------------------------
> +----------------------------------------------
> +Case #1 (the element content is hidden):
> +----------------------------------------------
> +:PROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:ROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +Text is revealed, because we have drawer in place of property-drawer
> +----------------------------------------------
> +----------------------------------------------
> +Case #2 (the element content is hidden):
> +----------------------------------------------
> +:ROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:OPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +The text remains hidden since it is still a drawer.
> +----------------------------------------------
> +----------------------------------------------
> +Case #3: (the element content is hidden):
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:FOO:
> +bar
> +:END:
> +tmp
> +:END:
> +----------------------------------------------
> +The text is revealed because the drawer contents shrank.
> +----------------------------------------------
> +----------------------------------------------
> +Case #4: (the element content is hidden in both the drawers):
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:END:
> +:BAR:
> +jjd
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:BAR:
> +jjd
> +:END:
> +----------------------------------------------
> +The text is revealed in both the drawers because the drawers are merged
> +into a new drawer.
> +----------------------------------------------
> +----------------------------------------------
> +Case #5: (the element content is hidden)
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:drawer:
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +The text is revealed in the drawer because the drawer expended.
> +----------------------------------------------
> +----------------------------------------------
> +Case #6: (the element content is hidden):
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +Nam a sapien.
> +:end:
> +----------------------------------------------
> +The text remains hidden because drawer contents is always before the
> +first :end:."
> +  (save-match-data
> +    (save-excursion
> +      (save-restriction
> +	(goto-char (org-element-property :begin el))
> +	(let* ((newel (org--get-element-region-at-point
> +		       (mapcar (lambda (el)
> +				 (when (string-match-p (regexp-opt '("block" "drawer"))
> +						       (symbol-name (car el)))
> +                                   (car el)))
> +                               org-track-element-modifications)))
> +	       (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
> +			 'org-hide-block
> +		       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
> +			   'org-hide-drawer
> +			 t)))
> +               (toggle-func (if (eq spec 'org-hide-drawer)
> +				#'org-hide-drawer-toggle
> +                              (if (eq spec 'org-hide-block)
> +				  #'org-hide-block-toggle
> +                                #'ignore)))) ; this should not happen
> +	  (if (and (equal (org-element-type el) (org-element-type newel))
> +		   (equal (marker-position (org-element-property :begin el))
> +			  (org-element-property :begin newel))
> +		   (equal (marker-position (org-element-property :end el))
> +			  (org-element-property :end newel)))
> +	      (when (text-property-any (marker-position (org-element-property :begin el))
> +				       (marker-position (org-element-property :end el))
> +				       'invisible spec)
> +                (goto-char (org-element-property :begin newel))
> +		(if (memq this-command org-track-element-modification-default-sensitive-commands)
> +		    ;; reveal if change was made by typing
> +		    (funcall toggle-func 'off)
> +		  ;; re-hide the inserted text
> +		  ;; FIXME: opening the drawer before hiding should not be needed here
> +		  (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis
> +		  (funcall toggle-func 'hide)))
> +            ;; The element was destroyed. Reveal everything.
> +            (org-flag-region (marker-position (org-element-property :begin el))
> +			  (marker-position (org-element-property :end el))
> +			  nil spec)
> +            (when newel
> +              (org-flag-region (org-element-property :begin newel)
> +			    (org-element-property :end newel)
> +			    nil spec))))))))
> +
> +(defun org--before-element-change-function (beg end)
> +  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
> +  (save-match-data
> +    (save-excursion
> +      (save-restriction
> +        (widen)
> +	(dolist (el (org--find-elements-in-region beg
> +					       end
> +					       (mapcar #'car org-track-element-modifications)
> +					       'include-partial
> +                                               'include-neighbours))
> +	  (let* ((beg-marker (copy-marker (org-element-property :begin el) 't))
> +		 (end-marker (copy-marker (org-element-property :end el) 't)))
> +	    (when (and (marker-position beg-marker) (marker-position end-marker))
> +	      (org-element-put-property el :begin beg-marker)
> +	      (org-element-put-property el :end end-marker)
> +	      (add-to-list 'org--modified-elements el))))))))
> +
> +;; FIXME: this function may be called many times during routine modifications
> +;; The normal way to avoid this is `combine-after-change-calls' - not
> +;; the case in most org primitives.
> +(defun org--after-element-change-function (&rest _)
> +  "Handle changed elements from `org--modified-elements'."
> +  (dolist (el org--modified-elements)
> +    (save-match-data
> +      (save-excursion
> +        (save-restriction
> +          (widen)
> +	  (when-let* ((type (org-element-type el))
> +		      (change-func (plist-get (alist-get type org-track-element-modifications)
> +					      :after-change-function)))
> +	    (with-demoted-errors
> +		(funcall (symbol-function change-func) el)))))))
> +  (setq org--modified-elements nil))
> +
>  (defvar org-mode-map)
>  (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
>  (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
> @@ -4818,6 +5194,9 @@ The following commands are available:
>    ;; Activate before-change-function
>    (setq-local org-table-may-need-update t)
>    (add-hook 'before-change-functions 'org-before-change-function nil 'local)
> +  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
> +  ;; Activate after-change-function
> +  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
>    ;; Check for running clock before killing a buffer
>    (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
>    ;; Initialize macros templates.
> @@ -4869,6 +5248,10 @@ The following commands are available:
>    (setq-local outline-isearch-open-invisible-function
>  	      (lambda (&rest _) (org-show-context 'isearch)))
>  
> +  ;; Make isearch search in blocks hidden via text properties
> +  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
> +  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
> +
>    ;; Setup the pcomplete hooks
>    (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
>    (setq-local pcomplete-command-name-function #'org-command-at-point)
> @@ -5050,8 +5433,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
>  	      (when verbatim?
>  		(org-remove-flyspell-overlays-in
>  		 (match-beginning 0) (match-end 0))
> -		(remove-text-properties (match-beginning 2) (match-end 2)
> -					'(display t invisible t intangible t)))
> +		(org-remove-text-properties (match-beginning 2) (match-end 2)
> +					 '(display t invisible t intangible t)))
>  	      (add-text-properties (match-beginning 2) (match-end 2)
>  				   '(font-lock-multiline t org-emphasis t))
>  	      (when (and org-hide-emphasis-markers
> @@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links."
>  	    (if (not (eq 'bracket style))
>  		(add-text-properties start end properties)
>  	      ;; Handle invisible parts in bracket links.
> -	      (remove-text-properties start end '(invisible nil))
> +	      (org-remove-text-properties start end '(invisible nil))
>  	      (let ((hidden
>  		     (append `(invisible
>  			       ,(or (org-link-get-parameter type :display)
> @@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links."
>  (defun org-activate-code (limit)
>    (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
>      (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
> -    (remove-text-properties (match-beginning 0) (match-end 0)
> -			    '(display t invisible t intangible t))
> +    (org-remove-text-properties (match-beginning 0) (match-end 0)
> +			     '(display t invisible t intangible t))
>      t))
>  
>  (defcustom org-src-fontify-natively t
> @@ -5258,8 +5641,8 @@ by a #."
>  	    (setq block-end (match-beginning 0)) ; includes the final newline.
>  	    (when quoting
>  	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
> -	      (remove-text-properties beg end-of-endline
> -				      '(display t invisible t intangible t)))
> +	      (org-remove-text-properties beg end-of-endline
> +				       '(display t invisible t intangible t)))
>  	    (add-text-properties
>  	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
>  	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
> @@ -5313,8 +5696,8 @@ by a #."
>  	     '(font-lock-fontified t face org-document-info))))
>  	 ((string-prefix-p "+caption" dc1)
>  	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
> -	  (remove-text-properties (match-beginning 0) (match-end 0)
> -				  '(display t invisible t intangible t))
> +	  (org-remove-text-properties (match-beginning 0) (match-end 0)
> +				   '(display t invisible t intangible t))
>  	  ;; Handle short captions.
>  	  (save-excursion
>  	    (beginning-of-line)
> @@ -5336,8 +5719,8 @@ by a #."
>  	   '(font-lock-fontified t face font-lock-comment-face)))
>  	 (t ;; just any other in-buffer setting, but not indented
>  	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
> -	  (remove-text-properties (match-beginning 0) (match-end 0)
> -				  '(display t invisible t intangible t))
> +	  (org-remove-text-properties (match-beginning 0) (match-end 0)
> +				   '(display t invisible t intangible t))
>  	  (add-text-properties beg (match-end 0)
>  			       '(font-lock-fontified t face org-meta-line))
>  	  t))))))
> @@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group."
>  	 (inhibit-modification-hooks t)
>  	 deactivate-mark buffer-file-name buffer-file-truename)
>      (decompose-region beg end)
> -    (remove-text-properties beg end
> -			    '(mouse-face t keymap t org-linked-text t
> -					 invisible t intangible t
> -					 org-emphasis t))
> +    (org-remove-text-properties beg end
> +			     '(mouse-face t keymap t org-linked-text t
> +					  invisible t
> +                                          intangible t
> +					  org-emphasis t))
>      (org-remove-font-lock-display-properties beg end)))
>  
>  (defconst org-script-display  '(((raise -0.3) (height 0.7))
> @@ -5970,6 +6354,29 @@ open and agenda-wise Org files."
>  
>  ;;;; Headlines visibility
>  
> +(defun org-hide-entry ()
> +  "Hide the body directly following this heading."
> +  (interactive)
> +  (save-excursion
> +    (outline-back-to-heading)
> +    (outline-end-of-heading)
> +    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
> +
> +(defun org-hide-subtree ()
> +  "Hide everything after this heading at deeper levels."
> +  (interactive)
> +  (org-flag-subtree t))
> +
> +(defun org-hide-sublevels (levels)
> +  "Hide everything but the top LEVELS levels of headers, in whole buffer.
> +This also unhides the top heading-less body, if any.
> +
> +Interactively, the prefix argument supplies the value of LEVELS.
> +When invoked without a prefix argument, LEVELS defaults to the level
> +of the current heading, or to 1 if the current line is not a heading."
> +  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
> +    (org-hide-sublevels levels)))
> +
>  (defun org-show-entry ()
>    "Show the body directly following this heading.
>  Show the heading too, if it is currently invisible."
> @@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible."
>         'outline)
>        (org-cycle-hide-property-drawers 'children))))
>  
> +(defun org-show-heading ()
> +  "Show the current heading and move to its end."
> +  (org-flag-region (- (point)
> + 		   (if (bobp) 0
> + 		     (if (and outline-blank-line
> +                              (eq (char-before (1- (point))) ?\n))
> + 			 2 1)))
> +		(progn (outline-end-of-heading) (point))
> +		nil))
> +
>  (defun org-show-children (&optional level)
>    "Show all direct subheadings of this heading.
>  Prefix arg LEVEL is how many levels below the current level
> @@ -6031,6 +6448,11 @@ heading to appear."
>    (org-flag-region
>     (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
>  
> +(defun org-show-branches ()
> +  "Show all subheadings of this heading, but not their bodies."
> +  (interactive)
> +  (org-show-children 1000))
> +
>  ;;;; Blocks and drawers visibility
>  
>  (defun org--hide-wrapper-toggle (element category force no-error)
> @@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful."
>  	(unless (let ((eol (line-end-position)))
>  		  (and (> eol start) (/= eol end)))
>  	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
> -			     ((eq type 'property-drawer) 'outline)
> -			     (t 'org-hide-drawer)))
> +			     ((eq category 'drawer) 'org-hide-drawer)
> +			     (t 'outline)))
>  		 (flag
>  		  (cond ((eq force 'off) nil)
>  			(force t)
> @@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of
>  	       (when (org-at-property-drawer-p)
>  		 (let* ((case-fold-search t)
>  			(end (re-search-forward org-property-end-re)))
> -		   ;; Property drawers use `outline' invisibility spec
> -		   ;; so they can be swallowed once we hide the
> -		   ;; outline.
> -		   (org-flag-region start end t 'outline)))))))))))
> +		   (org-flag-region start end t 'org-hide-drawer)))))))))))
>  
>  ;;;; Visibility cycling
>  
> @@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level."
>  		     (org-narrow-to-subtree)
>  		     (org-content))))
>  		((or "all" "showall")
> -		 (outline-show-subtree))
> +		 (org-show-subtree))
>  		(_ nil)))
>  	    (org-end-of-subtree)))))))
>  
> @@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'."
>  	  (while (re-search-forward re nil t)
>  	    (when (and (not (org-invisible-p))
>  		       (org-invisible-p (line-end-position)))
> -	      (outline-hide-entry))))
> +	      (org-hide-entry))))
>  	(org-cycle-hide-property-drawers 'all)
>  	(org-cycle-show-empty-lines 'overview)))))
>  
> @@ -6683,8 +7102,13 @@ information."
>      ;; expose it.
>      (dolist (o (overlays-at (point)))
>        (when (memq (overlay-get o 'invisible)
> -		  '(org-hide-block org-hide-drawer outline))
> +		  '(outline))
>  	(delete-overlay o)))
> +    (when (memq (get-text-property (point) 'invisible)
> +		'(org-hide-block org-hide-drawer))
> +      (let ((spec (get-text-property (point) 'invisible))
> +	    (region (org--find-text-property-region (point) 'invisible)))
> +	(org-flag-region (car region) (cdr region) nil spec)))
>      (unless (org-before-first-heading-p)
>        (org-with-limited-levels
>         (cl-case detail
> @@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
>       (skip-chars-forward " \t\n\r")
>       (setq beg (point))
>       (when (and (org-invisible-p) visp)
> -       (save-excursion (outline-show-heading)))
> +       (save-excursion (org-show-heading)))
>       ;; Shift if necessary.
>       (unless (= shift 0)
>         (save-restriction
> @@ -8103,7 +8527,7 @@ function is being called interactively."
>  		       (point))
>  	    what "children")
>        (goto-char start)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (outline-next-heading))
>       (t
>        ;; we will sort the top-level entries in this file
> @@ -13150,7 +13574,7 @@ drawer is immediately hidden."
>  	   (inhibit-read-only t))
>         (unless (bobp) (insert "\n"))
>         (insert ":PROPERTIES:\n:END:")
> -       (org-flag-region (line-end-position 0) (point) t 'outline)
> +       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
>         (when (or (eobp) (= begin (point-min))) (insert "\n"))
>         (org-indent-region begin (point))))))
>  
> @@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer."
>  (defun org-show-branches-buffer ()
>    "Show all branches in the buffer."
>    (org-flag-above-first-heading)
> -  (outline-hide-sublevels 1)
> +  (org-hide-sublevels 1)
>    (unless (eobp)
> -    (outline-show-branches)
> +    (org-show-branches)
>      (while (outline-get-next-sibling)
> -      (outline-show-branches)))
> +      (org-show-branches)))
>    (goto-char (point-min)))
>  
>  (defun org-kill-note-or-show-branches ()
> @@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer."
>  	(t
>  	 (let ((beg (progn (org-back-to-heading) (point)))
>  	       (end (save-excursion (org-end-of-subtree t t) (point))))
> -	   (outline-hide-subtree)
> -	   (outline-show-branches)
> +	   (org-hide-subtree)
> +	   (org-show-branches)
>  	   (org-hide-archived-subtrees beg end)))))
>  
>  (defun org-delete-indentation (&optional arg)
> @@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
>      (if (org-before-first-heading-p)
>          (progn
>            (org-flag-above-first-heading)
> -          (outline-hide-sublevels (or arg 1))
> +          (org-hide-sublevels (or arg 1))
>            (goto-char (point-min)))
> -      (outline-hide-subtree)
> +      (org-hide-subtree)
>        (org-show-children arg))))
>  
>  (defun org-ctrl-c-star ()
> @@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'."
>            (t default-org-info-node))))))
>  
>  \f
> +
> +;;; Make isearch search in some text hidden via text propertoes
> +
> +(defvar org--isearch-overlays nil
> +  "List of overlays temporarily created during isearch.
> +This is used to allow searching in regions hidden via text properties.
> +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
> +Any text hidden via text properties is not revealed even if `search-invisible'
> +is set to 't.")
> +
> +;; Not sure if it needs to be a user option
> +;; One might want to reveal hidden text in, for example, hidden parts of the links.
> +;; Currently, hidden text in links is never revealed by isearch.
> +(defvar org-isearch-specs '(org-hide-block
> +			 org-hide-drawer)
> +  "List of text invisibility specs to be searched by isearch.
> +By default ([2020-05-09 Sat]), isearch does not search in hidden text,
> +which was made invisible using text properties. Isearch will be forced
> +to search in hidden text with any of the listed 'invisible property value.")
> +
> +(defun org--create-isearch-overlays (beg end)
> +  "Replace text property invisibility spec by overlays between BEG and END.
> +All the regions with invisibility text property spec from
> +`org-isearch-specs' will be changed to use overlays instead
> +of text properties. The created overlays will be stored in
> +`org--isearch-overlays'."
> +  (let ((pos beg))
> +    (while (< pos end)
> +      (when-let* ((spec (get-text-property pos 'invisible))
> +		  (spec (memq spec org-isearch-specs))
> +		  (region (org--find-text-property-region pos 'invisible)))
> +        (setq spec (get-text-property pos 'invisible))
> +        ;; Changing text properties is considered buffer modification.
> +        ;; We do not want it here.
> +	(with-silent-modifications
> +          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
> +          ;; overlay for 'outline blocks.
> +          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
> +	    (overlay-put o 'evaporate t)
> +	    (overlay-put o 'invisible spec)
> +            ;; `delete-overlay' here means that spec information will be lost
> +            ;; for the region. The region will remain visible.
> +	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
> +            (push o org--isearch-overlays))
> +	  (org-flag-region (car region) (cdr region) nil spec)))
> +      (setq pos (next-single-property-change pos 'invisible nil end)))))
> +
> +(defun org--isearch-filter-predicate (beg end)
> +  "Return non-nil if text between BEG and END is deemed visible by Isearch.
> +This function is intended to be used as `isearch-filter-predicate'.
> +Unlike `isearch-filter-visible', make text with 'invisible text property
> +value listed in `org-isearch-specs' visible to Isearch."
> +  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
> +  (isearch-filter-visible beg end))
> +
> +(defun org--clear-isearch-overlay (ov)
> +  "Convert OV region back into using text properties."
> +  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
> +    ;; Changing text properties is considered buffer modification.
> +    ;; We do not want it here.
> +    (with-silent-modifications
> +      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
> +  (when (member ov isearch-opened-overlays)
> +    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
> +  (delete-overlay ov))
> +
> +(defun org--clear-isearch-overlays ()
> +  "Convert overlays from `org--isearch-overlays' back into using text properties."
> +  (when org--isearch-overlays
> +    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
> +    (setq org--isearch-overlays nil)))
> +
> +\f
> +
>  ;;; Finish up
>  
>  (add-hook 'org-mode-hook     ;remove overlays when changing major mode
>
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Hello,
>>
>> [The patch itself will be provided in the following email]
>>
>> I have five updates from the previous version of the patch:
>>
>> 1. I implemented a simplified version of element parsing to detect
>> changes in folded drawers or blocks. No computationally expensive calls
>> of org-element-at-point or org-element-parse-buffer are needed now.
>>
>> 2. The patch is now compatible with master (commit 2e96dc639). I
>> reverted the earlier change in folding drawers and blocks. Now, they are
>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
>> achieve nothing when we use text properties.
>>
>> 3. 'invisible text property can now be nested. This is important, for
>> example, when text inside drawers contains fontified links (which also
>> use 'invisible text property to hide parts of the link). Now, the old
>> 'invisible spec is recovered after unfolding.
>>
>> 4. Some outline-* function calls in org referred to outline-flag-region
>> implementation, which is not in sync with org-flag-region in this patch.
>> I have implemented their org-* versions and replaced the calls
>> throughout .el files. Actually, some org-* versions were already
>> implemented in org, but not used for some reason (or not mentioned in
>> the manual). I have updated the relevant sections of manual. These
>> changes might be relevant to org independently of this feature branch.
>>
>> 5. I have managed to get a working version of outline folding via text
>> properties. However, that approach has a big downside - folding state
>> cannot be different in indirect buffer when we use text properties. I
>> have seen packages relying on this feature of org and I do not see any
>> obvious way to achieve different folding state in indirect buffer while
>> using text properties for outline folding.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on the new implementation for tracking changes:
>>
>>> Of course we can. It is only necessary to focus on changes that would
>>> break the structure of the element. This does not entail a full parsing.
>>
>> I have limited parsing to matching beginning and end of a drawer/block.
>> The basic functions are org--get-element-region-at-point,
>> org--get-next-element-region-at-point, and org--find-elements-in-region.
>> They are simplified versions of org-element-* parsers and do not require
>> parsing everything from the beginning of the section.
>>
>> For now, I keep everything in org.el, but those simplified parsers
>> probably belong to org-element.el.
>>
>>> If we can stick with `after-change-functions' (or local equivalent),
>>> that's better. It is more predictable than `before-change-functions' and
>>> alike.
>>
>> For now, I still used before/after-change-functions combination.
>> I see the following problems with using only after-change-functions: 
>>
>> 1. They are not guaranteed to be called after every single change:
>>
>> From (elisp) Change Hooks:
>> "... some complex primitives call ‘before-change-functions’ once before
>> making changes, and then call ‘after-change-functions’ zero or more
>> times"
>>
>> The consequence of it is a possibility that region passed to the
>> after-change-functions is quite big (including all the singular changes,
>> even if they are distant). This region may contain changed drawers as
>> well and unchanged drawers and needs to be parsed to determine which
>> drawers need to be re-folded.
>>
>>> And, more importantly, they are not meant to be used together, i.e., you
>>> cannot assume that a single call to `before-change-functions' always
>>> happens before calling `after-change-functions'. This can be tricky if
>>> you want to use the former to pass information to the latter.
>>
>> The fact that before-change-functions can be called multiple times
>> before after-change-functions, is trivially solved by using buffer-local
>> changes register (see org--modified-elements). The register is populated
>> by before-change-functions and cleared by after-change-functions.
>>
>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>> clean at all: you modify an unrelated part of the buffer, but still call
>>> those to check if a drawer needs to be unfolded somewhere.
>>
>> 2. As you pointed, instead of global before-change-functions, we can use
>> modification-hooks text property on sensitive parts of the
>> drawers/blocks. This would work, but I am concerned about one annoying
>> special case:
>>
>> -------------------------------------------------------------------------
>> :BLAH: <inserted outside any of the existing drawers>
>>
>> <some text>
>>
>> :DRAWER: <folded>
>> Donec at pede.
>> :END:
>> -------------------------------------------------------------------------
>> In this example, the user would not be able to unfold the folder DRAWER
>> because it will technically become a part of a new giant BLAH drawer.
>> This may be especially annoying if <some text> is more than one screen
>> long and there is no easy way to identify why unfolding does not work
>> (with point at :DRAWER:).
>>
>> Because of this scenario, limiting before-change-functions to folded
>> drawers is not sufficient. Any change in text may need to trigger
>> unfolding.
>>
>> In the patch, I always register possible modifications in the
>> blocks/drawers intersecting with the modified region + a drawer/block
>> right next to the region.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on the nested 'invisible text property implementation.
>>
>> The idea is to keep 'invisible property stack push and popping from it
>> as we add/remove 'invisible text property. All the work is done in
>> org-flag-region.
>>
>> This was originally intended for folding outlines via text properties.
>> Since using text properties for folding outlines is not a good idea,
>> nested text properties have much less use. As I mentioned, they do
>> preserve link fontification, but I am not sure if it worth it for the
>> overhead to org-flag-region. Keeping this here mostly in the case if
>> someone has any ideas how it can be useful.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on replaced outline-* -> org-* function calls.
>>
>> I have implemented org-* versions of the following functions:
>>
>> - outline-hide-entry
>> - outline-hide-subtree
>> - outline-hide-sublevels
>> - outline-show-heading
>> - outline-show-branches
>>
>> The org-* versions trivially use org-flag-region instead of
>> outline-flag-region.
>>
>> Replaced outline-* calls where org- versions were already available:
>>
>> - outline-show-entry
>> - outline-show-all
>> - outline-show-subtree
>>
>> I reflected the new (including already available) functions in the
>> manual and removed some defalias from org-compat.el where they are not
>> needed. 
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> Further work:
>>
>> 1. after-change-functions use org-hide-drawer/block-toggle to
>> fold/unfold after modification. However, I just found that they call
>> org-element-at-point, which slows down modifications in folded
>> drawers/blocks. For example, realigning a long table inside folded
>> drawer takes >1sec, while it is instant in the unfolded drawer.
>>
>> 2. org-toggle-custom-properties is terribly slow on large org documents,
>> similarly to folded drawers on master. It should be relatively easy to
>> use text properties there instead of overlays.
>>
>> 3. Multiple calls to before/after-change-functions is still a problem. I
>> am looking into following ways to reduce this number:
>>  - reduce the number of elements registered as potentially modified
>>    + do not add duplicates to org--modified-elements
>>    + do not add unfolded elements to org--modified-elements
>>    + register after-change-function as post-command hook and remove it
>>      from global after-change-functions. This way, it will be called
>>      twice per command only.
>>  - determine common region containing org--modified-elements. if change
>>    is happening within that region, there is no need to parse
>>    drawers/blocks there again.
>>
>> P.S.
>>
>>>> It was mostly an annoyance, because they returned different results on
>>>> the same element. Specifically, they returned different :post-blank and
>>>> :end properties, which does not sound right.
>>>
>>> OK. If you have a reproducible recipe, I can look into it and see what
>>> can be done.
>>
>> Recipe to have different (org-element-at-point) and
>> (org-element-parse-buffer 'element)
>> -------------------------------------------------------------------------
>> <point-min>
>> :PROPERTIES:
>> :CREATED:  [2020-05-23 Sat 02:32]
>> :END:
>>
>>
>> <point-max>
>> -------------------------------------------------------------------------
>>
>>
>> Best,
>> Ihor
>>
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>>> Hello,
>>>
>>> Ihor Radchenko <yantar92@gmail.com> writes:
>>>
>>>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>>>>> element is a O(N) operation by the number of elements before it in
>>>>> a section. In particular, it is not bounded, and not mitigated by
>>>>> a cache. For large documents, it is going to be unbearably slow, too.
>>>>
>>>> Ouch. I thought it is faster.
>>>> What do you mean by "not mitigated by a cache"?
>>>
>>> Parsing starts from the closest headline, every time. So, if Org parses
>>> the Nth element in the entry two times, it really parses 2N elements.
>>>
>>> With a cache, assuming the buffer wasn't modified, Org would parse
>>> N elements only. With a smarter cache, with fine grained cache
>>> invalidation, it could also reduce the number of subsequent parsed
>>> elements.
>>>
>>>> The reason I would like to utilise org-element parser to make tracking
>>>> modifications more robust. Using details of the syntax would make the
>>>> code fragile if any modifications are made to syntax in future.
>>>
>>> I don't think the code would be more fragile. Also, the syntax we're
>>> talking about is not going to be modified anytime soon. Moreover, if
>>> folding breaks, it is usually visible, so the bug will not be unnoticed.
>>>
>>> This code is going to be as low-level as it can be.
>>>
>>>> Debugging bugs in modification functions is not easy, according to my
>>>> experience.
>>>
>>> No, it's not. 
>>>
>>> But this is not really related to whether you use Element or not.
>>>
>>>> One possible way to avoid performance issues during modification is
>>>> running parser in advance. For example, folding an element may
>>>> as well add information about the element to its text properties.
>>>> This will not degrade performance of folding since we are already
>>>> parsing the element during folding (at least, in
>>>> org-hide-drawer-toggle).
>>>
>>> We can use this information stored at fold time. But I'm not even sure
>>> we need it.
>>>
>>>> The problem with parsing an element during folding is that we cannot
>>>> easily detect changes like below without re-parsing.
>>>
>>> Of course we can. It is only necessary to focus on changes that would
>>> break the structure of the element. This does not entail a full parsing.
>>>
>>>> :PROPERTIES: <folded>
>>>> :CREATED: [2020-05-18 Mon]
>>>> :END: <- added line
>>>> :ID: test
>>>> :END:
>>>>
>>>> or even
>>>>
>>>> :PROPERTIES:
>>>> :CREATED: [2020-05-18 Mon]
>>>> :ID: test
>>>> :END: <- delete this line
>>>>
>>>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
>>>> test
>>>> :END:
>>>
>>> Please have a look at the "sensitive parts" I wrote about. This takes
>>> care of this kind of breakage.
>>>
>>>> The re-parsing can be done via regexp, as you suggested, but I don't
>>>> like this idea, because it will end up re-implementing
>>>> org-element-*-parser.
>>>
>>> You may have misunderstood my suggestion. See below.
>>>
>>>> Would it be acceptable to run org-element-*-parser
>>>> in after-change-functions?
>>>
>>> I'd rather not do that. This is unnecessary consing, and matching, etc.
>>>
>>>> If I understand correctly, it is not as easy.
>>>> Consider the following example:
>>>>
>>>> :PROPERTIES:
>>>> :CREATED: [2020-05-18 Mon]
>>>> <region-beginning>
>>>> :ID: example
>>>> :END:
>>>>
>>>> <... a lot of text, maybe containing other drawers ...>
>>>>
>>>> Nullam rutrum.
>>>> Pellentesque dapibus suscipit ligula.
>>>> <region-end>
>>>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>>>>
>>>> If the region gets deleted, the modification hooks from chars inside
>>>> drawer will be called as (hook-function <region-beginning>
>>>> <region-end>). So, there is still a need to find the drawer somehow to
>>>> mark it as about to be modified (modification hooks are ran before
>>>> actual modification).
>>>
>>> If we can stick with `after-change-functions' (or local equivalent),
>>> that's better. It is more predictable than `before-change-functions' and
>>> alike.
>>>
>>> If it is a deletion, here is the kind of checks we could do, depending
>>> on when they are performed.
>>>
>>> Before actual changes :
>>>
>>>   1. The deletion is happening within a folded drawer (unnecessary step
>>>      in local functions).
>>>   2. The change deleted the sensitive line ":END:".
>>>   3. Conclusion : unfold.
>>>
>>> Or, after actual changes :
>>>
>>>   1. The deletion involves a drawer.
>>>   2. Text properties indicate that the beginning of the propertized part
>>>      of the buffer start with org-drawer-regexp, but doesn't end with
>>>      `org-property-end-re'. A "sensitive part" disappeared!
>>>   3. Conclusion : unfold
>>>
>>> This is far away from parsing. IMO, a few checks cover all cases. Let me
>>> know if you have questions about it.
>>>
>>> Also, note that the kind of change you describe will happen perhaps
>>> 0.01% of the time. Most change are about one character, or a single
>>> line, long.
>>>
>>>> The only difference between using modification hooks and
>>>> before-change-functions is that modification hooks will trigger less
>>>> frequently. 
>>>
>>> Exactly. Much less frequently. But extra care is required, as you noted
>>> already.
>>>
>>>> Considering the performance of org-element-at-point, it is
>>>> probably worth doing. Initially, I wanted to avoid it because setting a
>>>> single before-change-functions hook sounded cleaner than setting
>>>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
>>>
>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>> clean at all: you modify an unrelated part of the buffer, but still call
>>> those to check if a drawer needs to be unfolded somewhere.
>>>
>>> And, more importantly, they are not meant to be used together, i.e., you
>>> cannot assume that a single call to `before-change-functions' always
>>> happens before calling `after-change-functions'. This can be tricky if
>>> you want to use the former to pass information to the latter.
>>>
>>> But I understand that they are easier to use than their local
>>> counterparts. If you stick with (before|after)-change-functions, the
>>> function being called needs to drop the ball very quickly if the
>>> modification is not about folding changes. Also, I very much suggest to
>>> stick to only `after-change-functions', if feasible (I think it is), per
>>> above.
>>>
>>>> Moreover, these text properties would be copied by default if one uses 
>>>> buffer-substring. Then, the hooks will also trigger later in the yanked
>>>> text, which may cause all kinds of bugs.
>>>
>>> Indeed, that would be something to handle specifically. I.e.,
>>> destructive modifications (i.e., those that unfold) could clear such
>>> properties.
>>>
>>> It is more work. I don't know if it is worth the trouble if we can get
>>> out quickly of `after-change-functions' for unrelated changes.
>>>
>>>> It was mostly an annoyance, because they returned different results on
>>>> the same element. Specifically, they returned different :post-blank and
>>>> :end properties, which does not sound right.
>>>
>>> OK. If you have a reproducible recipe, I can look into it and see what
>>> can be done.
>>>
>>> Regards,
>>>
>>> -- 
>>> Nicolas Goaziou
>>
>> -- 
>> Ihor Radchenko,
>> PhD,
>> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
>> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
>> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg