Slow movement in large buffers

Slow movement in large buffers

[Matt Lundin]

I've been navigating the org-issues file (14000+ lines) and have found
movement within the file to be fairly slow. Sometimes Emacs will lock up
for several seconds.

For instance, to move from the level one heading "* Other" to "* Closed
issues" when the outline is folded takes over three seconds:

--8<---------------cut here---------------start------------->8---
next-line     1           3.015289      3.015289
--8<---------------cut here---------------end--------------->8---

In my experience, the maximum workable size of org files is around
10,000 lines. Beyond that, Emacs starts to spin its wheels.

Do others have the same experience? If so, does anyone have any tips on
how to diagnose this further?

(insert "\n" emacs-version)
23.3.1

(insert "\n" org-version)
7.5

Best,
Matt

 

Re: Slow movement in large buffers

[Carsten Dominik]

DO you have flyspell turned on?

- Carsten

On 15.3.2011, at 04:25, Matt Lundin wrote:

> I've been navigating the org-issues file (14000+ lines) and have found
> movement within the file to be fairly slow. Sometimes Emacs will lock up
> for several seconds.
> 
> For instance, to move from the level one heading "* Other" to "* Closed
> issues" when the outline is folded takes over three seconds:
> 
> --8<---------------cut here---------------start------------->8---
> next-line     1           3.015289      3.015289
> --8<---------------cut here---------------end--------------->8---
> 
> In my experience, the maximum workable size of org files is around
> 10,000 lines. Beyond that, Emacs starts to spin its wheels.
> 
> Do others have the same experience? If so, does anyone have any tips on
> how to diagnose this further?
> 
> (insert "\n" emacs-version)
> 23.3.1
> 
> (insert "\n" org-version)
> 7.5
> 
> Best,
> Matt
> 
> 
> 

Re: Slow movement in large buffers

[Eric S Fraga]

Matt Lundin <mdl@imapmail.org> writes:

> I've been navigating the org-issues file (14000+ lines) and have found
> movement within the file to be fairly slow. Sometimes Emacs will lock up
> for several seconds.
>
> For instance, to move from the level one heading "* Other" to "* Closed
> issues" when the outline is folded takes over three seconds:
>
> next-line     1           3.015289      3.015289
>
> In my experience, the maximum workable size of org files is around
> 10,000 lines. Beyond that, Emacs starts to spin its wheels.
>
> Do others have the same experience? If so, does anyone have any tips on
> how to diagnose this further?
>
> (insert "\n" emacs-version)
> 23.3.1
>
> (insert "\n" org-version)
> 7.5
>
> Best,
> Matt
>
>  

Interesting.  It doesn't take 3 seconds on my system (Intel(R)
Pentium(R) D CPU 2.80GHz) but there is a noticeable delay using C-n to
move from =* Other= to =* Closed issues=.  However, using C-cC-n to move
has no delay at all and moving backwards (either C-p or C-cC-p) also
exhibits no delay.

I have noticed significant slowness in some of my own files lately when
including babel source code blocks, especially latex ones.  But I
haven't been able to reproduce this slowness predictably enough to
instrument it and hence report it in any useful sense... still
experimenting.

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.38.gf8c6.dirty)

Re: Slow movement in large buffers

[Carsten Dominik]

On Mar 15, 2011, at 4:25 AM, Matt Lundin wrote:

> I've been navigating the org-issues file (14000+ lines) and have found
> movement within the file to be fairly slow. Sometimes Emacs will lock up
> for several seconds.
> 
> For instance, to move from the level one heading "* Other" to "* Closed
> issues" when the outline is folded takes over three seconds:
> 
> --8<---------------cut here---------------start------------->8---
> next-line     1           3.015289      3.015289
> --8<---------------cut here---------------end--------------->8---
> 

Wow, this is really bad.
Could you use elp and instrument org, outline,
and font-lock, and do that motion and report
the results?

> In my experience, the maximum workable size of org files is around
> 10,000 lines. Beyond that, Emacs starts to spin its wheels.

Not good at all.

- Carsten

> 
> Do others have the same experience? If so, does anyone have any tips on
> how to diagnose this further?
> 
> (insert "\n" emacs-version)
> 23.3.1
> 
> (insert "\n" org-version)
> 7.5
> 
> Best,
> Matt
> 
> 
> 

- Carsten

Re: Slow movement in large buffers

[Lawrence Mitchell]

Carsten Dominik wrote:
> On Mar 15, 2011, at 4:25 AM, Matt Lundin wrote:

>> I've been navigating the org-issues file (14000+ lines) and have found
>> movement within the file to be fairly slow. Sometimes Emacs will lock up
>> for several seconds.

>> For instance, to move from the level one heading "* Other" to "* Closed
>> issues" when the outline is folded takes over three seconds:

>> --8<---------------cut here---------------start------------->8---
>> next-line     1           3.015289      3.015289
>> --8<---------------cut here---------------end--------------->8---


> Wow, this is really bad.
> Could you use elp and instrument org, outline,
> and font-lock, and do that motion and report
> the results?

For me, all the time is spent in vertical-motion (which is C
level, so I can't get a more detailed breakdown).

>> In my experience, the maximum workable size of org files is around
>> 10,000 lines. Beyond that, Emacs starts to spin its wheels.

> Not good at all.

I believe it is likely to be due to org's large use of overlays.
Here's a quote from the elisp manual:

| (elisp)Overlays
|    The visual effect of an overlay is the same as of the corresponding
| text property ....  However, due to a different
| implementation, *overlays generally don't scale well* (many operations
| take a time that is proportional to the number of overlays in the
| buffer).  If you need to affect the visual appearance of many portions
| in the buffer, we recommend using text properties.

Emphasis added.

For example, in org-issues.org:

M-: (loop for l in (overlay-lists) sum (length l)) RET => 3823

And vertical motion (with C-n, C-p) takes a long time.

remove all the overlays

M-: (remove-overlays) RET

collapse the buffer again

M-x org-shifttab RET

Now vertical motion is rapid.

This suggests that one should try and see if overlays in org
buffers can be replaced by text properties.  Which do not cause
the same slowdown.

Lawrence
-- 
Lawrence Mitchell <wence@gmx.li>

Re: Slow movement in large buffers

[Matt Lundin]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> On Mar 15, 2011, at 4:25 AM, Matt Lundin wrote:
>
>> I've been navigating the org-issues file (14000+ lines) and have found
>> movement within the file to be fairly slow. Sometimes Emacs will lock up
>> for several seconds.
>> 
>> For instance, to move from the level one heading "* Other" to "* Closed
>> issues" when the outline is folded takes over three seconds:
>> 
>> --8<---------------cut here---------------start------------->8---
>> next-line     1           3.015289      3.015289
>> --8<---------------cut here---------------end--------------->8---
>> 
>
> Wow, this is really bad.
> Could you use elp and instrument org, outline,
> and font-lock, and do that motion and report
> the results?
>

I instrumented those packages (along with next-line and previous-line)
and turned off flyspell. Unfortunately, it seems there is something else
involved, as previous-line is the only function registered by
elp-results:

--8<---------------cut here---------------start------------->8---
previous-line   1           2.524475      2.524475
--8<---------------cut here---------------end--------------->8---

This time, moving forward was faster than moving backward. The slowness
occurred when moving back from "* Other" to "* Development Tasks" in the
folded org-issues.org buffer. The movement jumped over a region
containing 4000 lines.

Since nothing from org or outline showed up in the results, I
instrumented the various functions called by previous-line. The culprit
seems to be a function written in C: vertical-motion. So this seems to
be an Emacs issue, rather than an org or outline issue.

--8<---------------cut here---------------start------------->8---
previous-line           1           2.5365349999  2.5365349999
line-move               1           2.536489      2.536489
line-move-visual        1           2.536436      2.536436
vertical-motion         1           2.508419      2.508419
font-lock-mode          1           3.7e-05       3.7e-05
line-move-partial       1           1.4e-05       1.4e-05
font-lock-default-function      1           9e-06         9e-06
window-hscroll          1           4e-06         4e-06
--8<---------------cut here---------------end--------------->8---

Best,
Matt

Re: Slow movement in large buffers

[Matt Lundin]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> DO you have flyspell turned on?
>
Yes, but the same results occur with flyspell turned off. (See my
response to your other email.)

Best,
Matt 

Re: Slow movement in large buffers

[Matt Lundin]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Matt Lundin <mdl@imapmail.org> writes:

> Interesting.  It doesn't take 3 seconds on my system (Intel(R)
> Pentium(R) D CPU 2.80GHz) but there is a noticeable delay using C-n to
> move from =* Other= to =* Closed issues=.  However, using C-cC-n to move
> has no delay at all and moving backwards (either C-p or C-cC-p) also
> exhibits no delay.

You are right. The slowness only occurs when using next-line and
previous-line. When I use the functions for navigating outlines (e.g.,
outline-next-visible-heading), the movement is instantaneous.

Best,
Matt

Re: Slow movement in large buffers

[Eric S Fraga]

Hello,

following up on this issue, I have just run into it again.  I'm editing
a not very large document and suddenly things slowed down, mostly but
not exclusively for "next-line":

--8<---------------cut here---------------start------------->8---
next-line                                                     18          2.1547069999  0.1197059444
previous-line                                                 19          0.4066669999  0.0214035263
org-mode-flyspell-verify                                      16          5.299...e-05  3.312...e-06
--8<---------------cut here---------------end--------------->8---

This happened when I started a new source code block (gnuplot, to be
exact) but didn't type in the end_src line for a while.  The problem
seems to be due to font-locking and it tries to font-lock the whole
document initially.  When I eventually get around to typing the end_src
line, it font-locks correctly but things are slow thereafter.  There
seems to be some hysteresis loop in the code...

If I kill the buffer and reload the file, everything is fine.

--8<---------------cut here---------------start------------->8---
next-line                                                     17          0.0655900000  0.0038582352
previous-line                                                 17          0.0115249999  0.0006779411
org-mode                                                      1           0.007178      0.007178
org-fontify-meta-lines-and-blocks                             25          0.0022920000  9.168...e-05
org-set-startup-visibility                                    1           0.001619      0.001619
org-raise-scripts                                             25          0.0013889999  5.555...e-05
--8<---------------cut here---------------end--------------->8---

Dramatic difference!

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.38.gf8c6.dirty)

Re: Slow movement in large buffers

[Nick Dokos]

Carsten Dominik <carsten.dominik@gmail.com> wrote:

> 
> On Mar 15, 2011, at 4:25 AM, Matt Lundin wrote:
> 
> > I've been navigating the org-issues file (14000+ lines) and have found
> > movement within the file to be fairly slow. Sometimes Emacs will lock up
> > for several seconds.
> > 
> > For instance, to move from the level one heading "* Other" to "* Closed
> > issues" when the outline is folded takes over three seconds:
> > 
> > --8<---------------cut here---------------start------------->8---
> > next-line     1           3.015289      3.015289
> > --8<---------------cut here---------------end--------------->8---
> > 
> 
> Wow, this is really bad.
> Could you use elp and instrument org, outline,
> and font-lock, and do that motion and report
> the results?
> 
> > In my experience, the maximum workable size of org files is around
> > 10,000 lines. Beyond that, Emacs starts to spin its wheels.
> 
> Not good at all.
> 
> - Carsten
> 
> > 
> > Do others have the same experience? If so, does anyone have any tips on
> > how to diagnose this further?
> > 
> > (insert "\n" emacs-version)
> > 23.3.1
> > 
> > (insert "\n" org-version)
> > 7.5
> > 

FWIW, I opened org-issues.org and followed Matt's lead: at first, I got
almost instant response going from Other to Closed (using arrow-down)
and a slight hesitation (< 0.5s) going from Other to the previous
headline (Development Tasks).  After noodling around for a while
(including the motions that Eric F. suggested and getting no delays
there), I tried it again and could discern no delay going either
way. And I do run flyspell in org buffers.

This is on a fairly recent vintage laptop with an i7 quad core processor
(2.67GHz) and 4G of memory. That probably explains some of the difference
I see. Matt, what kind of hardware are you using?

Nick

Re: Slow movement in large buffers

[Matt Lundin]

Nick Dokos <nicholas.dokos@hp.com> writes:

> FWIW, I opened org-issues.org and followed Matt's lead: at first, I got
> almost instant response going from Other to Closed (using arrow-down)
> and a slight hesitation (< 0.5s) going from Other to the previous
> headline (Development Tasks).  After noodling around for a while
> (including the motions that Eric F. suggested and getting no delays
> there), I tried it again and could discern no delay going either
> way. And I do run flyspell in org buffers.
>
> This is on a fairly recent vintage laptop with an i7 quad core processor
> (2.67GHz) and 4G of memory. That probably explains some of the difference
> I see. Matt, what kind of hardware are you using?

Yes, hardware is indeed a factor here. I'm using a dual-core Atom processor
with 2GB of memory.

Matt

Re: Slow movement in large buffers

[Chris Randle]

Hi Matt

On 2011-03-15 03:25, Matt Lundin wrote:
> I've been navigating the org-issues file (14000+ lines) and have
> found movement within the file to be fairly slow. Sometimes Emacs
> will lock up for several seconds.
<snip>
> Do others have the same experience? If so, does anyone have any tips
> on how to diagnose this further?

I keep all my info in one big Org-mode file which is currently just shy
of 115,000 lines. There's the occasional "stutter" of a fraction of a
second when I move across closed nodes containing large chunks, but it's
still perfectly acceptable (to me, anyway!).

My PC is an Intel dual-core 2.66GHz with 4GB RAM, so nothing
earth-shattering.

-- 
Chris Randle

Re: Slow movement in large buffers

[MidLifeXis at PerlMonks]

I have seen this happen when I have largish files with tables.  I also have a 
file
setting of:

#+begin_src org
  #+STARTUP: align
#+end_src


which turns on automatic alignment of things recognized as table-like.  If I 
remove
this startup directive, things tend to run more smoothly.

I have not formally reproduced this problem, and am currently going by memory as
to when I have seen it, so YMMV (and so might mine).

MidLifeXis


----- Original Message ----
From: Chris Randle <chris@amlog.co.uk>
To: emacs-orgmode@gnu.org
Sent: Tue, March 15, 2011 11:11:23 AM
Subject: Re: [O] Slow movement in large buffers

Hi Matt

On 2011-03-15 03:25, Matt Lundin wrote:
> I've been navigating the org-issues file (14000+ lines) and have
> found movement within the file to be fairly slow. Sometimes Emacs
> will lock up for several seconds.
<snip>
> Do others have the same experience? If so, does anyone have any tips
> on how to diagnose this further?

I keep all my info in one big Org-mode file which is currently just shy
of 115,000 lines. There's the occasional "stutter" of a fraction of a
second when I move across closed nodes containing large chunks, but it's
still perfectly acceptable (to me, anyway!).

My PC is an Intel dual-core 2.66GHz with 4GB RAM, so nothing
earth-shattering.

-- Chris Randle

Re: Slow movement in large buffers

[Scott Randby]

On 03/15/2011 12:11 PM, Chris Randle wrote:
> Hi Matt
> 
> On 2011-03-15 03:25, Matt Lundin wrote:
>> I've been navigating the org-issues file (14000+ lines) and have
>> found movement within the file to be fairly slow. Sometimes Emacs
>> will lock up for several seconds.
> <snip>
>> Do others have the same experience? If so, does anyone have any tips
>> on how to diagnose this further?
> 
> I keep all my info in one big Org-mode file which is currently just shy
> of 115,000 lines. There's the occasional "stutter" of a fraction of a
> second when I move across closed nodes containing large chunks, but it's
> still perfectly acceptable (to me, anyway!).
> 
> My PC is an Intel dual-core 2.66GHz with 4GB RAM, so nothing
> earth-shattering.
> 

I have a netbook running an Intel dual-core 1.66GHz and 2GB RAM. I
opened up org-issues.org and navigated through it without difficulty.
There was a delay for about a second when I unfolded the "Development
Tasks" headline, but there were no delays after that. Vertical movement
is instantaneous.

Scott Randby

Re: Slow movement in large buffers

[Erik Iverson]

Scott Randby wrote:
> On 03/15/2011 12:11 PM, Chris Randle wrote:
>> On 2011-03-15 03:25, Matt Lundin wrote:
>>> I've been navigating the org-issues file (14000+ lines) and have
>>> found movement within the file to be fairly slow. Sometimes Emacs
>>> will lock up for several seconds.
>> <snip>
>>> Do others have the same experience? If so, does anyone have any tips
>>> on how to diagnose this further?

<snip>

> I have a netbook running an Intel dual-core 1.66GHz and 2GB RAM. I
> opened up org-issues.org and navigated through it without difficulty.
> There was a delay for about a second when I unfolded the "Development
> Tasks" headline, but there were no delays after that. Vertical movement
> is instantaneous.

Do those who experience slowness have a line-numbering mode in Emacs
turned on?  (linum-mode perhaps?)

Re: Slow movement in large buffers

[Carsten Dominik]

On 15.3.2011, at 18:14, Scott Randby wrote:

> On 03/15/2011 12:11 PM, Chris Randle wrote:
>> Hi Matt
>> 
>> On 2011-03-15 03:25, Matt Lundin wrote:
>>> I've been navigating the org-issues file (14000+ lines) and have
>>> found movement within the file to be fairly slow. Sometimes Emacs
>>> will lock up for several seconds.
>> <snip>
>>> Do others have the same experience? If so, does anyone have any tips
>>> on how to diagnose this further?
>> 
>> I keep all my info in one big Org-mode file which is currently just shy
>> of 115,000 lines. There's the occasional "stutter" of a fraction of a
>> second when I move across closed nodes containing large chunks, but it's
>> still perfectly acceptable (to me, anyway!).
>> 
>> My PC is an Intel dual-core 2.66GHz with 4GB RAM, so nothing
>> earth-shattering.
>> 
> 
> I have a netbook running an Intel dual-core 1.66GHz and 2GB RAM. I
> opened up org-issues.org and navigated through it without difficulty.
> There was a delay for about a second when I unfolded the "Development
> Tasks" headline, but there were no delays after that. Vertical movement
> is instantaneous.


One reason why the motion in a folded buffer is slow:
calling next line on the heading of a folded tree must
move down potentially thousands of buffer lines.  In each buffer
line it checks if the text is still invisible and then moves one.
So this kind of buffer motion is not optimized for outline
buffers.  Still, what Matt and others report is unacceptable,
and there must be a solution.  I have very slight delays
in org-issues.org, but nothing like 5 seconds.   Fractions
of a second, at most.

- Carsten

> 
> Scott Randby
> 

Re: Slow movement in large buffers

[Carsten Dominik]

On 15.3.2011, at 16:51, Matt Lundin wrote:

> Nick Dokos <nicholas.dokos@hp.com> writes:
> 
>> FWIW, I opened org-issues.org and followed Matt's lead: at first, I got
>> almost instant response going from Other to Closed (using arrow-down)
>> and a slight hesitation (< 0.5s) going from Other to the previous
>> headline (Development Tasks).  After noodling around for a while
>> (including the motions that Eric F. suggested and getting no delays
>> there), I tried it again and could discern no delay going either
>> way. And I do run flyspell in org buffers.
>> 
>> This is on a fairly recent vintage laptop with an i7 quad core processor
>> (2.67GHz) and 4G of memory. That probably explains some of the difference
>> I see. Matt, what kind of hardware are you using?
> 
> Yes, hardware is indeed a factor here. I'm using a dual-core Atom processor
> with 2GB of memory.

Well, it should not be a hardware issue to move the cursor down a line.

- Carsten

> 
> Matt

Re: Re: Slow movement in large buffers

[Carsten Dominik]

On 15.3.2011, at 15:09, Eric S Fraga wrote:

> Hello,
> 
> following up on this issue, I have just run into it again.  I'm editing
> a not very large document and suddenly things slowed down, mostly but
> not exclusively for "next-line":
> 
> --8<---------------cut here---------------start------------->8---
> next-line                                                     18          2.1547069999  0.1197059444
> previous-line                                                 19          0.4066669999  0.0214035263
> org-mode-flyspell-verify                                      16          5.299...e-05  3.312...e-06
> --8<---------------cut here---------------end--------------->8---
> 
> This happened when I started a new source code block (gnuplot, to be
> exact) but didn't type in the end_src line for a while.  The problem
> seems to be due to font-locking and it tries to font-lock the whole
> document initially.  When I eventually get around to typing the end_src
> line, it font-locks correctly but things are slow thereafter.  There
> seems to be some hysteresis loop in the code...

This sounds like a bug that needs to be fixed in the block fontifications,
maybe a limit for how far to search for the end line.  regular expressions
that match many lines need to be carefully constructed - there are possible
backtracking traps that can make the matching time scale as the number of
characters squared.

- Carsten

> 
> If I kill the buffer and reload the file, everything is fine.
> 
> --8<---------------cut here---------------start------------->8---
> next-line                                                     17          0.0655900000  0.0038582352
> previous-line                                                 17          0.0115249999  0.0006779411
> org-mode                                                      1           0.007178      0.007178
> org-fontify-meta-lines-and-blocks                             25          0.0022920000  9.168...e-05
> org-set-startup-visibility                                    1           0.001619      0.001619
> org-raise-scripts                                             25          0.0013889999  5.555...e-05
> --8<---------------cut here---------------end--------------->8---
> 
> Dramatic difference!
> 
> -- 
> : Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
> : using Org-mode version 7.5 (release_7.5.38.gf8c6.dirty)
> 

Re: Re: Slow movement in large buffers

[Christian Moe]

> Yes, hardware is indeed a factor here. I'm using a dual-core Atom processor
> with 2GB of memory.
>
> Matt
>

I'm not so sure about the hardware factor.

I'm on a seven-year-old G4 Mac PPC laptop with 768 MB RAM.

Over the same folded headings in a freshly pulled org-issues (Other to 
Development Tasks), I get

previous-line      5       5.89456       1.178912

Definitely a lag, but not as bad as you're seeing.


Yours,
Christian

Re: Re: Slow movement in large buffers

[Nick Dokos]

Christian Moe <mail@christianmoe.com> wrote:

> 
> > Yes, hardware is indeed a factor here. I'm using a dual-core Atom processor
> > with 2GB of memory.
> >
> > Matt
> >
> 
> I'm not so sure about the hardware factor.
> 
> I'm on a seven-year-old G4 Mac PPC laptop with 768 MB RAM.
> 
> Over the same folded headings in a freshly pulled org-issues (Other to
> Development Tasks), I get
> 
> previous-line      5       5.89456       1.178912
> 
> Definitely a lag, but not as bad as you're seeing.
> 
> 

True, but it's still a factor: on my laptop I get

--8<---------------cut here---------------start------------->8---
previous-line  2           1.435139      0.7175695
next-line      1           0.446293      0.446293
--8<---------------cut here---------------end--------------->8---

being very careful to avoid next-line/previous-line in any other
context:  that was one next-line from "Other" to "Closed issues" and two
previous-lines from "Closed Issues" to "Other" to "Development
Tasks". Given the five previous-lines in your profile, I suspect that
one or two were much longer than the others which skewed the average.

Given the evidence that Lawrence Mitchell provided however, it seems clear
that most of the blame can be placed on overlays - no?

Nick

Re: Re: Slow movement in large buffers

[Christian Moe]

On 3/15/11 11:56 PM, Nick Dokos wrote:
  Given the five previous-lines in your profile, I suspect that
> one or two were much longer than the others which skewed the average.

Actually not, I was going back and forth over the same two lines, and 
previous-line was fairly stable at around 1.2 seconds each time. 
Anyway, that's moot since, as you say,

> Given the evidence that Lawrence Mitchell provided however, it seems clear
> that most of the blame can be placed on overlays

Re: Re: Slow movement in large buffers

[Bastien]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Given the evidence that Lawrence Mitchell provided however, it seems clear
> that most of the blame can be placed on overlays - no?

Yes, probably.

I've also experience slow motion in some big files of mine.

One factor was the use of the :ARCHIVE: tag.  I tried to remove this
tag, and things were fast again.  Another factor might be the use of
folded drawers and #+begin_src environments.

So I guess the problem boils down to 2 factors: the number of *nested
overlays*, and the number of lines in each.  How these factors interact
is hard to guess.

Instead of testing from org-issues.org (which is pretty messy), maybe 
we can have a testing/overlays.org file with a systematic structure?

#+begin_src org
* headline 1

** A subheading with 100 lines

lines ... lines (x 100)

* headline 2

** A subheading with 100 lines (x 100)
   :PROPERTIES:
   :LOCATION: Erewhon
   :END:

lines ... lines (x 100)

* headline 3

** A subheading with begin_src env and 100 lines (x 100)

  #+begin_src emacs-lisp
  (defun fixme ()
    "Some random defun"
    (message "This is not a message")
  #+begin_src

lines ... lines (x 100)

* headline 4

** subheading with archive tag and 100 lines :ARCHIVE: (x 100)

lines ... lines (x 100)
#+end_src

Matt, can you build and locally test such a file?  Then instrument
next-line when jumping from headline 1 to 2, to 3, to 4?

-- 
 Bastien

Re: Re: Slow movement in large buffers

[Nick Dokos]

Christian Moe <mail@christianmoe.com> wrote:

> On 3/15/11 11:56 PM, Nick Dokos wrote:
>   Given the five previous-lines in your profile, I suspect that
> > one or two were much longer than the others which skewed the average.
> 
> Actually not, I was going back and forth over the same two lines, and 
> previous-line was fairly stable at around 1.2 seconds each time. 

My apologies. It took me a couple of tries to get things right and
I made an unwarranted generalization.

Thanks,
Nick

> Anyway, that's moot since, as you say,
> 
> > Given the evidence that Lawrence Mitchell provided however, it seems clear
> > that most of the blame can be placed on overlays
> 
> 

Re: Slow movement in large buffers

[Matt Lundin]

Bastien <bzg@altern.org> writes:

> Matt, can you build and locally test such a file?  Then instrument
> next-line when jumping from headline 1 to 2, to 3, to 4?

I'm on the job. :)

I'll write back with an official report. Suffice it to say that my early
experiments show org-mode cruising through the examples you gave me, but
slowing considerably whenever, as you suspected, there are numerous
nested overlays.

I tried, unwisely, to create a tree consisting of 100 subtrees, each of
which contained a quote, a drawer, and 100 additional subtrees, each of
which, in turn contained a drawer. It not only took the better part of a
minute to cycle the tree, but it sent my emacs memory usage
skyrocketing. Needless to say, 10000 nested entries, each with quotes
and drawers, is an extreme case. :)

Best,
Matt

Re: Slow movement in large buffers

[Carmine Casciato]

Carsten Dominik <carsten.dominik <at> gmail.com> writes:
 
> >> On 2011-03-15 03:25, Matt Lundin wrote:
> >>> I've been navigating the org-issues file (14000+ lines) and have
> >>> found movement within the file to be fairly slow. Sometimes Emacs
> >>> will lock up for several seconds.
> >> <snip>
> >>> Do others have the same experience? If so, does anyone have any tips
> >>> on how to diagnose this further?
> >> 
> >> I keep all my info in one big Org-mode file which is currently just shy
> >> of 115,000 lines. There's the occasional "stutter" of a fraction of a
> >> second when I move across closed nodes containing large chunks, but it's
> >> still perfectly acceptable (to me, anyway!).
> >> 
> >> My PC is an Intel dual-core 2.66GHz with 4GB RAM, so nothing
> >> earth-shattering.
> >> 
I am have been seeing this for a while now on only one of about 8 
org files that I have, specifically the longest one which also has the 
most babel snippets (of pretty long sql). This behaviour is present on 
an Ubuntu VM with a scant 300Mbs as well as a Mac Pro with 6Gbs 
(although less of course). On the Ubuntu, I specifically see the CPU 
spike when typing! Very bizarre.
-C.

Re: Slow movement in large buffers

[Eric S Fraga]

Carmine Casciato <casciato@gmail.com> writes:

[...]

> I am have been seeing this for a while now on only one of about 8 
> org files that I have, specifically the longest one which also has the 
> most babel snippets (of pretty long sql). This behaviour is present on 
> an Ubuntu VM with a scant 300Mbs as well as a Mac Pro with 6Gbs 
> (although less of course). On the Ubuntu, I specifically see the CPU 
> spike when typing! Very bizarre.
> -C.

What versions of org & emacs are you using?  Carsten recently (a month
ago or so?) put in a fix that seems to have addressed this problem, at
least for me.  I've been doing a lot of babel work lately and haven't
seen the slowdown since Carsten's fix.

Before this fix, the solution was to turn off mode specific fontification
of the source blocks: C-h v org-src-fontify-natively RET

HTH,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.274.gd6aba)