[9.4] LOGBOOK visibility

[9.4] LOGBOOK visibility

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

Hi,

I've noticed that LOGBOOK drawers are now (9.4) open by default when
visiting an Org file, AFAICT as a result of org-startup-folded now
defaulting to showeverything.

That's not much of a problem by itself; what I find more of a hindrance
is that even with "#+STARTUP overview", when one starts isearching for
something, all "nearby" LOGBOOK drawers are opened; this adds a lot of
visual noise that hides the document structure.

For example, in the attached file, if I search for "bug#5":

- Org 9.3 keeps the drawers closed,
- Org 9.4 opens every drawer for bugs 4, 5 and 6.

AFAICT Org 9.3 never opened these drawers unless either
org-startup-folded was showeverything, the user explicitly opened one
individually, or their /content/ matched a search.

Note that when cycling with S-TAB, even when "[SHOWING] ALL", Org 9.4
keeps these drawers closed.

Is there a knob I've missed that would allow me to tell Org to keep
LOGBOOKs closed?


BTW, while reading ORG-NEWS, I came across this bit, added in 074ea1629:

> *** Deprecated ~org-cycle-hide-drawers~ function
> 
> Use the new function ~org-hide-drawer-all~ instead.  Note that there
> is also a new ~org-cycle-hide-property-drawers~ function.

AFAICT org-cycle-hide-property-drawers has been removed in 1aa095ccf?


Thank you for your time, and for your work with Org 9.4.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: logbooks.org --]
[-- Type: text/x-org, Size: 4010 bytes --]

#+STARTUP: overview
* project 1
** bug#1
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
** bug#2
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
** bug#3
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
* project 2
** bug#4
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
** bug#5
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
** bug#6
*** investigate
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** fix
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:
*** document
:LOGBOOK:
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
CLOCK: [2020-07-02 Thu 11:43]--[2020-07-02 Thu 11:43] =>  0:00
:END:

Re: [9.4] LOGBOOK visibility

[Kévin Le Gouguec]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> For example, in the attached file, if I search for "bug#5":
>
> - Org 9.3 keeps the drawers closed,
> - Org 9.4 opens every drawer for bugs 4, 5 and 6.
>
> AFAICT Org 9.3 never opened these drawers unless either
> org-startup-folded was showeverything, the user explicitly opened one
> individually, or their /content/ matched a search.
>
> Note that when cycling with S-TAB, even when "[SHOWING] ALL", Org 9.4
> keeps these drawers closed.

I've tried to bisect this; if I'm not mistaken, the commit that
introduced this behaviour is 1027e0256.  I don't know why or how this
commit caused this change yet, so I'm not sure if it's deliberate or
not.

Re: [9.4] LOGBOOK visibility

[Kévin Le Gouguec]

I haven't reached the bottom of this rabbit hole yet.  Since I think
I've spent all the time I had to spend in this issue for the day, here's
where I'm at.

If I open my example file[1] with Org 9.3 and master (86fada6b5), and
compare the overlays covering the hidden part of the first LOGBOOK
drawer, like so:

#+begin_src sh
#!/bin/bash

set -eu

point=67

orgs=(
    "${path_to_org_9_3}"
    "${path_to_org_master}"
)

for o in "${orgs[@]}"
do
    emacs -Q -batch \
          -L "${o}"/lisp \
          -eval "(find-file \"logbooks.org\")" \
          -eval "(message org-version)" \
          -eval "(dolist (o (overlays-at ${point}))
                   (message \"from %s to %s\" (overlay-start o) (overlay-end o))
                   (message \"%s\" (overlay-properties o)))"
    echo
done
#+end_src

Here is what I get:

> 9.3
> from 32 to 2015
> (evaporate t invisible outline isearch-open-invisible #[lambda gibberish involving (org-show-context 'isearch)])
> from 67 to 262
> (isearch-open-invisible delete-overlay invisible org-hide-drawer evaporate t)
> 
> 9.3.7
> from 32 to 2015
> (isearch-open-invisible delete-overlay invisible outline evaporate t)

- The LOGBOOK overlay (from 67 to 262) vanished.

- The isearch-open-invisible property of the "project 1" overlay (from
  32 to 2015) changed from a lambda[2] to just delete-overlay.


Again, not sure this shows there's an issue; for all I know Org 9.4 just
works differently and nothing's wrong there.  I hope someone can chime
in and take it up from here; otherwise I'll resume my investigations
sometime later.


[1] https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/2-logbooks.org

[2] Which I guess is the value of
    outline-isearch-open-invisible-function, set locally by org-mode in
    the major mode function?

Re: [9.4] LOGBOOK visibility

[Ihor Radchenko]

> I haven't reached the bottom of this rabbit hole yet.  Since I think
> I've spent all the time I had to spend in this issue for the day, here's
> where I'm at.

There used to be several types of overlays for headline folding, drawer
folding, and block folding. One of the recent commits made all the
overlays use headline type. As a result all the overlays are merged into
a single overlay upon folding. This was done to reduce the total number
of overlays present in an org buffer, which degrades Emacs performance
on huge org files (see the discussion in [1]). 

What you observe is a consequence of that change. Now, all the drawer
overlays are destroyed when you fold a heading. Isearch would
only see a single huge overlay and unfold it alltogether.

I am currently working on a patch to rewrite the whole folding system.
Your issue should disappear once it is applied.

Meanwhile, you may, for example, advice
org-fold--isearch-filter-predicate to re-fold drawers during isearch. 

Best,
Ihor

[1] https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> I haven't reached the bottom of this rabbit hole yet.  Since I think
> I've spent all the time I had to spend in this issue for the day, here's
> where I'm at.
>
> If I open my example file[1] with Org 9.3 and master (86fada6b5), and
> compare the overlays covering the hidden part of the first LOGBOOK
> drawer, like so:
>
> #+begin_src sh
> #!/bin/bash
>
> set -eu
>
> point=67
>
> orgs=(
>     "${path_to_org_9_3}"
>     "${path_to_org_master}"
> )
>
> for o in "${orgs[@]}"
> do
>     emacs -Q -batch \
>           -L "${o}"/lisp \
>           -eval "(find-file \"logbooks.org\")" \
>           -eval "(message org-version)" \
>           -eval "(dolist (o (overlays-at ${point}))
>                    (message \"from %s to %s\" (overlay-start o) (overlay-end o))
>                    (message \"%s\" (overlay-properties o)))"
>     echo
> done
> #+end_src
>
> Here is what I get:
>
>> 9.3
>> from 32 to 2015
>> (evaporate t invisible outline isearch-open-invisible #[lambda gibberish involving (org-show-context 'isearch)])
>> from 67 to 262
>> (isearch-open-invisible delete-overlay invisible org-hide-drawer evaporate t)
>> 
>> 9.3.7
>> from 32 to 2015
>> (isearch-open-invisible delete-overlay invisible outline evaporate t)
>
> - The LOGBOOK overlay (from 67 to 262) vanished.
>
> - The isearch-open-invisible property of the "project 1" overlay (from
>   32 to 2015) changed from a lambda[2] to just delete-overlay.
>
>
> Again, not sure this shows there's an issue; for all I know Org 9.4 just
> works differently and nothing's wrong there.  I hope someone can chime
> in and take it up from here; otherwise I'll resume my investigations
> sometime later.
>
>
> [1] https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/2-logbooks.org
>
> [2] Which I guess is the value of
>     outline-isearch-open-invisible-function, set locally by org-mode in
>     the major mode function?
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: [9.4] LOGBOOK visibility

[Ihor Radchenko]

> Meanwhile, you may, for example, advice
> org-fold--isearch-filter-predicate to re-fold drawers during isearch. 

org-fold--isearch-filter-predicate->isearch-filter-visible


Ihor Radchenko <yantar92@gmail.com> writes:

>> I haven't reached the bottom of this rabbit hole yet.  Since I think
>> I've spent all the time I had to spend in this issue for the day, here's
>> where I'm at.
>
> There used to be several types of overlays for headline folding, drawer
> folding, and block folding. One of the recent commits made all the
> overlays use headline type. As a result all the overlays are merged into
> a single overlay upon folding. This was done to reduce the total number
> of overlays present in an org buffer, which degrades Emacs performance
> on huge org files (see the discussion in [1]). 
>
> What you observe is a consequence of that change. Now, all the drawer
> overlays are destroyed when you fold a heading. Isearch would
> only see a single huge overlay and unfold it alltogether.
>
> I am currently working on a patch to rewrite the whole folding system.
> Your issue should disappear once it is applied.
>
> Meanwhile, you may, for example, advice
> org-fold--isearch-filter-predicate to re-fold drawers during isearch. 
>
> Best,
> Ihor
>
> [1] https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html
>
> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> I haven't reached the bottom of this rabbit hole yet.  Since I think
>> I've spent all the time I had to spend in this issue for the day, here's
>> where I'm at.
>>
>> If I open my example file[1] with Org 9.3 and master (86fada6b5), and
>> compare the overlays covering the hidden part of the first LOGBOOK
>> drawer, like so:
>>
>> #+begin_src sh
>> #!/bin/bash
>>
>> set -eu
>>
>> point=67
>>
>> orgs=(
>>     "${path_to_org_9_3}"
>>     "${path_to_org_master}"
>> )
>>
>> for o in "${orgs[@]}"
>> do
>>     emacs -Q -batch \
>>           -L "${o}"/lisp \
>>           -eval "(find-file \"logbooks.org\")" \
>>           -eval "(message org-version)" \
>>           -eval "(dolist (o (overlays-at ${point}))
>>                    (message \"from %s to %s\" (overlay-start o) (overlay-end o))
>>                    (message \"%s\" (overlay-properties o)))"
>>     echo
>> done
>> #+end_src
>>
>> Here is what I get:
>>
>>> 9.3
>>> from 32 to 2015
>>> (evaporate t invisible outline isearch-open-invisible #[lambda gibberish involving (org-show-context 'isearch)])
>>> from 67 to 262
>>> (isearch-open-invisible delete-overlay invisible org-hide-drawer evaporate t)
>>> 
>>> 9.3.7
>>> from 32 to 2015
>>> (isearch-open-invisible delete-overlay invisible outline evaporate t)
>>
>> - The LOGBOOK overlay (from 67 to 262) vanished.
>>
>> - The isearch-open-invisible property of the "project 1" overlay (from
>>   32 to 2015) changed from a lambda[2] to just delete-overlay.
>>
>>
>> Again, not sure this shows there's an issue; for all I know Org 9.4 just
>> works differently and nothing's wrong there.  I hope someone can chime
>> in and take it up from here; otherwise I'll resume my investigations
>> sometime later.
>>
>>
>> [1] https://orgmode.org/list/87eepuz0bj.fsf@gmail.com/2-logbooks.org
>>
>> [2] Which I guess is the value of
>>     outline-isearch-open-invisible-function, set locally by org-mode in
>>     the major mode function?
>>
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: [9.4] LOGBOOK visibility

[Kévin Le Gouguec]

Ihor Radchenko <yantar92@gmail.com> writes:

>                   (see the discussion in [1]). 

Ah, that makes sense!  No idea how I managed to miss that thread.

> I am currently working on a patch to rewrite the whole folding system.
> Your issue should disappear once it is applied.

Good to know!

> Meanwhile, you may, for example, advice
> org-fold--isearch-filter-predicate to re-fold drawers during isearch. 

Duly noted, thanks for the advice.

And thank you and Nicolas for all the hard work!

Re: [9.4] LOGBOOK visibility

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> There used to be several types of overlays for headline folding, drawer
> folding, and block folding. One of the recent commits made all the
> overlays use headline type. As a result all the overlays are merged into
> a single overlay upon folding. This was done to reduce the total number
> of overlays present in an org buffer, which degrades Emacs performance
> on huge org files (see the discussion in [1]). 
>
> What you observe is a consequence of that change. Now, all the drawer
> overlays are destroyed when you fold a heading. Isearch would
> only see a single huge overlay and unfold it alltogether.

For completeness, at the beginning drawers and outline shared the same
invisibility value. 

At some point during 9.X development, I gave drawers their own
invisibility value. It provided an overall nicer behaviour, but also
introduced some serious slowdown in large files.

Recently, as pointed out by Ihor, I switched drawers back to their
initial state, i.e., they now share the same invisibility value as the
outline. At least, Org on very large files is slightly more manageable,
but the behaviour is less nice. With overlays, you can't have your cake
and eat it too.

> I am currently working on a patch to rewrite the whole folding system.
> Your issue should disappear once it is applied.

Yes, hopefully, switching to text properties will get us out of this sad
situation.

Regards,
-- 
Nicolas Goaziou

Re: [9.4] LOGBOOK visibility

[Kévin Le Gouguec]

Thank you for taking the time to make this write-up.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>                                 With overlays, you can't have your cake
> and eat it too.

FWIW, I think Emacs has a branch (feature/noverlay) which has been
reported to improve performance with overlays.  AFAICT it's just hanging
there waiting to be merged (though a naive "git merge" reveals some
conflicts); some quick Googling suggests it has last been discussed two
years ago:

https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html

Re: [9.4] LOGBOOK visibility

[Ihor Radchenko]

> FWIW, I think Emacs has a branch (feature/noverlay) which has been
> reported to improve performance with overlays.  AFAICT it's just hanging
> there waiting to be merged (though a naive "git merge" reveals some
> conflicts); some quick Googling suggests it has last been discussed two
> years ago:

That branch is around for a very long time. The last attempt on it was
back in December [1]. However, all the progress was only in tests, not in
actual code. As I mentioned in the discussion about using properties
instead of overlays, I prefer to modify org now rather than waiting for
someone to finalise and merge that branch.

Also, text properties appear to use less memory.

Best,
Ihor

[1] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00115.html

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Thank you for taking the time to make this write-up.
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>>                                 With overlays, you can't have your cake
>> and eat it too.
>
> FWIW, I think Emacs has a branch (feature/noverlay) which has been
> reported to improve performance with overlays.  AFAICT it's just hanging
> there waiting to be merged (though a naive "git merge" reveals some
> conflicts); some quick Googling suggests it has last been discussed two
> years ago:
>
> https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg