Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     http://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org-mode mailing list.
------------------------------------------------------------------------

Since recently, after starting up emacs and bringing up an org agenda
which loads all my agenda files into buffers as a side-effect, all
entries in all files are fully expanded, although org-startup-folded is
set to t.

I've tried adding a

  #+STARTUP: fold

or a

  #+STARTUP: overview

header to the top of my agenda files, but that doesn't change anything.
Everything is completely expanded after the initial startup.

If needed, I could try a bisection.

Emacs  : GNU Emacs 24.3.50.1 (x86_64-pc-linux-gnu, GTK+ Version 3.6.3)
 of 2013-02-12 on thinkpad
Package: Org-mode version 7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)

current state:
==============
(setq
 org-tab-first-hook '(org-hide-block-toggle-maybe org-src-native-tab-command-maybe
		      org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 outline-minor-mode-hook '(th-outline-minor-mode-init)
 org-speed-command-hook '(org-speed-command-default-hook org-babel-speed-command-hook)
 org-gnus-prefer-web-links t
 org-special-ctrl-k t
 org-occur-hook '(org-first-headline-recenter)
 org-src-tab-acts-natively t
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-log-done 'time
 org-confirm-shell-link-function 'yes-or-no-p
 org-complete-tags-always-offer-all-agenda-tags t
 org-id-link-to-org-use-id t
 org-finalize-agenda-hook '(th-org-agenda-to-appt)
 org-columns-default-format "%50ITEM %TODO %ALLTAGS %SCHEDULED %DEADLINE"
 org-agenda-restore-windows-after-quit t
 org-special-ctrl-a/e 'reversed
 org-agenda-custom-commands '(("n" "Next 21 days" agenda "" ((org-agenda-span 21)))
			      ("^" "Calfw Month Calendar" th-calfw-open-calendar))
 org-return-follows-link t
 org-agenda-time-leading-zero t
 org-capture-templates '(("n" "NORMAL" entry (file "/home/horn/Repos/org/remember.org")
			  "* %?\n  :PROPERTIES:\n  :created: %U\n  :link: %a\n  :END:\n	%i")
			 ("t" "TODO" entry (file "/home/horn/Repos/org/remember.org")
			  "* TODO %?\n  :PROPERTIES:\n  :created: %U\n  :link: %a\n  :END:\n  %i")
			 ("i" "IDEA" entry (file "/home/horn/Repos/org/remember.org")
			  "* IDEA %?\n  :PROPERTIES:\n  :created: %U\n  :link: %a\n  :END:\n  %i")
			 )
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer)
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-babel-pre-tangle-hook '(save-buffer)
 org-mode-hook '(th-org-mode-init
		 #[nil "\300\301\302\303\304$\207"
		   [org-add-hook change-major-mode-hook org-show-block-all append local] 5]
		 #[nil "\300\301\302\303\304$\207"
		   [org-add-hook change-major-mode-hook org-babel-show-result-all append local] 5]
		 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-refile-targets '((org-agenda-files :maxlevel . 5))
 org-attach-method 'mv
 org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point org-babel-execute-safely-maybe)
 org-refile-use-outline-path 'file
 org-directory "/home/horn/Repos/org"
 org-enforce-todo-dependencies t
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers
		  org-cycle-hide-inline-tasks org-cycle-show-empty-lines
		  org-optimize-window-after-visibility-change)
 org-agenda-finalize-hook '(th-org-agenda-to-appt)
 org-attach-directory "/home/horn/Repos/org/attachments"
 org-archive-default-command 'org-archive-set-tag
 org-refile-allow-creating-parent-nodes 'confirm
 org-todo-keywords '((sequence "TODO(t)" "STARTED(s)" "DELEGATED(g)" "IDEA(i)" "|" "DONE(d)"
		      "CANCELLED(c)")
		     )
 org-modules '(org-icalendar org-attach org-protocol org-id org-bibtex org-docview org-gnus
	       org-info org-irc org-capture org-mobile)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-log-into-drawer "LOGBOOK"
 org-icalendar-store-UID t
 org-blocker-hook '(org-block-todo-from-children-or-siblings-or-parent)
 org-mobile-directory "~/Dropbox/MobileOrg"
 org-agenda-mode-hook '(th-org-agenda-mode-init)
 org-agenda-files '("/home/horn/Repos/org")
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-link-frame-setup '((vm . vm-visit-folder) (gnus . org-gnus-no-new-news)
			(file . find-file-other-window))
 org-mobile-inbox-for-pull "~/Repos/org/from-org-mobile.org"
 org-src-fontify-natively t
 )

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Sebastien Vauban]

Hi Tassilo,

Tassilo Horn wrote:
> Remember to cover the basics, that is, what you expected to happen and
> what in fact did happen.  You don't know how to make a good report?  See
>
>      http://orgmode.org/manual/Feedback.html#Feedback
>
> Your bug report will be posted to the Org-mode mailing list.
> ------------------------------------------------------------------------
>
> Since recently, after starting up emacs and bringing up an org agenda
> which loads all my agenda files into buffers as a side-effect, all
> entries in all files are fully expanded, although org-startup-folded is
> set to t.
>
> I've tried adding a
>
>   #+STARTUP: fold
>
> or a
>
>   #+STARTUP: overview
>
> header to the top of my agenda files, but that doesn't change anything.
> Everything is completely expanded after the initial startup.
>
> If needed, I could try a bisection.

It's an optimization done by Bastien. There is a variable to inhibit it.

That subject has been discussed here a couple of days ago. Search for "agenda"
and "performance" in Bastien's emails. Sorry, not time to do it, but I wanted
to avoid you loosing more time searching after this feature.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Bastien]

Hi Tassilo,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> It's an optimization done by Bastien. 

I think it's fine to have this by default, the time spared is worth
it IMHO.  But I understand many people may want to turn this off.

> There is a variable to inhibit it.

(setq org-agenda-inhibit-startup nil)

If you don't want the optimization.

In general, it's worth sticking to the default behavior if you have
many agenda files.  If you have only a few, the setting above may be
preferable.

HTH,

-- 
 Bastien

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Bastien <bzg@altern.org> writes:

Hi Bastien,

>> It's an optimization done by Bastien.
>
> I think it's fine to have this by default, the time spared is worth it
> IMHO.  But I understand many people may want to turn this off.
>
>> There is a variable to inhibit it.
>
> (setq org-agenda-inhibit-startup nil)
>
> If you don't want the optimization.

I've done the performance measurements you've requested in Message-ID:
<87ehgjgmiv.fsf@thinkpad.tsdh.de>.  For me, the "optimized" version is
by no means faster than the `org-agenda-inhibit-startup' set to nil.

If you reproduce the measurements on your machine and it turns out that
you get results inverse to mine, I'm happy to help debugging where the
differences come from.

> In general, it's worth sticking to the default behavior if you have
> many agenda files.  If you have only a few, the setting above may be
> preferable.

My measurements were performed with my 7 normal agenda files + one
synthetically constructed.  1 top-level headline with 8000 child TODO
items, and then triggering the TODO agenda (and also the week agenda).
That's what you've requested in <87vc9vdwx7.fsf@bzg.ath.cx>.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

"Sebastien Vauban"
<wxhgmqzgwmuf@spammotel.com> writes:

>> Everything is completely expanded after the initial startup.
>
> It's an optimization done by Bastien. There is a variable to inhibit
> it.

Ah, ok.  I'm not a big fan of optimizations that break correctness, and
the current behavior is simply wrong wrt. the docs of
`org-startup-folded' and the #+STARTUP property.

> That subject has been discussed here a couple of days ago. Search for
> "agenda" and "performance" in Bastien's emails.  Sorry, not time to do
> it, but I wanted to avoid you loosing more time searching after this
> feature.

Thanks a lot, I'll look it up myself.  I already searched for "folded",
"collapsed", and "expanded" before writing the bug report, but those
searches didn't find any recent messages.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Bastien]

Hi Tassilo,

Tassilo Horn <tsdh@gnu.org> writes:

> Ah, ok.  I'm not a big fan of optimizations that break correctness, and
> the current behavior is simply wrong wrt. the docs of
> `org-startup-folded' and the #+STARTUP property.

I just updated the docstring of org-startup-folded and mentioned
`org-agenda-inhibit-startup' in the manual (maint branch.)

I'm not convinced nil would be better as a default for
`org-agenda-inhibit-startup' -- even when users use only one big
.org file as their agenda, if they call (org-agenda nil "a") in
their .emacs.el then the optimization is also worth it.

-- 
 Bastien

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Bastien <bzg@altern.org> writes:

Hi Bastien,

>> Ah, ok.  I'm not a big fan of optimizations that break correctness,
>> and the current behavior is simply wrong wrt. the docs of
>> `org-startup-folded' and the #+STARTUP property.
>
> I just updated the docstring of org-startup-folded and mentioned
> `org-agenda-inhibit-startup' in the manual (maint branch.)

Thanks.

> I'm not convinced nil would be better as a default for
> `org-agenda-inhibit-startup' -- even when users use only one big .org
> file as their agenda, if they call (org-agenda nil "a") in their
> .emacs.el then the optimization is also worth it.

I have seven agenda files, all not too big (~10K on average), and I use
an averagely performant computer (5 years old dual core laptop).  For
me, the difference between `org-agenda-inhibit-startup' set to t or nil
is not noticable at all.  In both cases, it's less than a second, and
that's a time I can easily live with.

So if one asked me, I'd say the default should be nil.  Probably, it
would be a good idea to list and explain all such performance
optimization options and their implications in one central place in the
manual ("Speeding up org on slower computers"), so that people having
performance issues can easily find these knobs.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Bastien]

Hi Tassilo,

Tassilo Horn <tsdh@gnu.org> writes:

> I have seven agenda files, all not too big (~10K on average), and I use
> an averagely performant computer (5 years old dual core laptop).  For
> me, the difference between `org-agenda-inhibit-startup' set to t or nil
> is not noticable at all.  In both cases, it's less than a second, and
> that's a time I can easily live with.

It really depends on what your agenda views are.

The real test is this: create a 50K file with folded one headline and
many second level TODO entries.  Then have an agenda view search for
those TODO items.  Do you get the same difference in this case?

If this difference is closer to 0 than to 1 sec, yes maybe we should
switch back to the default behavior.  Tests are very welcome for this.

Let me know,

-- 
 Bastien

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Bastien <bzg@altern.org> writes:

>> I have seven agenda files, all not too big (~10K on average), and I
>> use an averagely performant computer (5 years old dual core laptop).
>> For me, the difference between `org-agenda-inhibit-startup' set to t
>> or nil is not noticable at all.  In both cases, it's less than a
>> second, and that's a time I can easily live with.
>
> It really depends on what your agenda views are.
>
> The real test is this: create a 50K file with folded one headline and
> many second level TODO entries.

Ok, the testfile looks like

--8<---------------cut here---------------start------------->8---
* The folded top-level headline

** TODO Some test todo headline 0

   - also some contents

** TODO Some test todo headline 1

   - also some contents
--8<---------------cut here---------------end--------------->8---

and has 2000 such subheadlines.  File size is 127K.
`org-startup-folded' is t.

> Then have an agenda view search for those TODO items.  Do you get the
> same difference in this case?

It takes approximately 9-10 seconds to build a TODO agenda (C-c o a t),
no matter if `org-agenda-inhibit-startup' is t or nil.  The week agenda
takes less than a second with both settings.  Before each agenda
invocation, I've deleted all org buffers of course.

I must admit I didn't really profile but just counted in my mind.  But I
did it several times, so it's super-scientific and you should trust me.
;-)

So now I've increased the number of subentries to 8000 resulting in a
file size of 492K, and I've used a stop-watch for measuring.

| org-agenda-inhibit-startup | time TODO agenda | time week agenda |
|----------------------------+------------------+------------------|
| nil                        | 2min1sec         | 1sec / 9sec      |
| t                          | 2min18sec        | 1sec / 29sec     |

I've performed these tests several times.  The times with respect to
TODO agenda creation were pretty much consistent.  However, the times
with respect to week agenda creation were usually around 1 second, but
there were some strange outliers I have no explanation for (the 9 and
the 29 seconds one).

But in any case, if there was a noticable difference, the
"org-agenda-inhibit-startup set to nil"-version was even faster than the
"optimized" version.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Bastien]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

Hi Tassilo,

Tassilo Horn <tsdh@gnu.org> writes:

> Bastien <bzg@altern.org> writes:
>
>>> I have seven agenda files, all not too big (~10K on average), and I
>>> use an averagely performant computer (5 years old dual core laptop).
>>> For me, the difference between `org-agenda-inhibit-startup' set to t
>>> or nil is not noticable at all.  In both cases, it's less than a
>>> second, and that's a time I can easily live with.
>>
>> It really depends on what your agenda views are.
>>
>> The real test is this: create a 50K file with folded one headline and
>> many second level TODO entries.
>
> Ok, the testfile looks like
>
> * The folded top-level headline
>
> ** TODO Some test todo headline 0
>
>    - also some contents
>
> ** TODO Some test todo headline 1
>
>    - also some contents
>
> and has 2000 such subheadlines.  File size is 127K.
> `org-startup-folded' is t.

That's not a very good test file, because you just need to unfold the
main headline to get all headlines visible.

I tested with a test.org file using this structure:

* Test
** TODO Test
*** Test

repeated 10k times.  Then run emacs -Q -l ~/test.el with the
attached test.el file, I get sensible results here.

I don't think it varies depending on the Emacs 23 vs 24 version,
but I believe we are using the same.

PS: I will revert `org-agenda-inhibit-startup' to nil as a default
and will explain why in another mail.


[-- Attachment #2: test.el --]
[-- Type: application/emacs-lisp, Size: 543 bytes --]

[-- Attachment #3: Type: text/plain, Size: 14 bytes --]


-- 
 Bastien

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Hi Bastien,

>> Ok, the testfile looks like
>>
>> * The folded top-level headline
>>
>> ** TODO Some test todo headline 0
>>
>>    - also some contents
>>
>> ** TODO Some test todo headline 1
>>
>>    - also some contents
>>
>> and has 2000 such subheadlines.  File size is 127K.
>> `org-startup-folded' is t.
>
> That's not a very good test file, because you just need to unfold the
> main headline to get all headlines visible.
>
> I tested with a test.org file using this structure:
>
> * Test
> ** TODO Test
> *** Test
>
> repeated 10k times.  Then run emacs -Q -l ~/test.el with the
> attached test.el file, I get sensible results here.

Indeed, with this test file, `org-agenda-inhibit-startup' does have a
major performance implication.

> PS: I will revert `org-agenda-inhibit-startup' to nil as a default
> and will explain why in another mail.

I'm keen to hear the explanation.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Tassilo Horn <thorn@fastmail.fm> writes:

> Since recently, after starting up emacs and bringing up an org agenda
> which loads all my agenda files into buffers as a side-effect, all
> entries in all files are fully expanded, although org-startup-folded
> is set to t.
>
> I've tried adding a
>
>   #+STARTUP: fold
>
> or a
>
>   #+STARTUP: overview
>
> header to the top of my agenda files, but that doesn't change anything.
> Everything is completely expanded after the initial startup.

Just a short addon:

When I do M-x revert-buffer RET in some agenda file (currently being
completely expanded), it's displayed folded (with the exception of the
entry containing point).  That's the correct behavior.

If I invoke `org-agenda-list' again to generate a new agenda, the
reverted agenda files stay in their folded state, too.

Now if I kill the buffer of some agenda file and invoke
`org-agenda-list' which will reopen the file, it's completely expanded
again.

So the problem seems to be that only if an agenda file is opened during
execution of `org-agenda-list', it's expanded completely.  Else,
`org-startup-folded' works correctly.

Bye,
Tassilo

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Bastien]

Hi Tassilo,

Tassilo Horn <tsdh@gnu.org> writes:

> So the problem seems to be that only if an agenda file is opened during
> execution of `org-agenda-list', it's expanded completely.  Else,
> `org-startup-folded' works correctly.

Exactly -- hence this commit:

http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=981c6d

I hope it's clear enough, let me know otherwise.

Thanks!

-- 
 Bastien

Re: Bug: #+STARTUP: overview and org-startup-folded have no effect [7.9.3e (7.9.3e-1032-g791a8d @ /home/horn/Repos/el/org-mode/lisp/)]

[Tassilo Horn]

Bastien <bzg@altern.org> writes:

Hi Bastien,

>> So the problem seems to be that only if an agenda file is opened
>> during execution of `org-agenda-list', it's expanded completely.
>> Else, `org-startup-folded' works correctly.
>
> Exactly -- hence this commit:
>
> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=981c6d
>
> I hope it's clear enough, let me know otherwise.

Yes, now the docs are clear again.  But still the change itself seems
like a premature optimization to me.  It might make a difference when
you have megabyte large agenda files or a very slow computer, but is
that the norm among users justifying such a change?

Well, anyway.  I've set `org-agenda-inhibit-startup' to nil, and now I'm
satisfied again.

BTW, that reminds me of http://xkcd.com/1172/. ;-)

Bye,
Tassilo