Largest org file you have + performance

Largest org file you have + performance

[Marcelo de Moraes Serpa]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

Hey list,

What's the largest orgmode file you have, and what's the performance you get
while manipulating/navigating it? Mine is a 5k lines file called
reference.org, and I basically keep all kind of notes and attachments there.
It's a bit slow to navigate on my Emacs 240.50.1, orgmode 7.4, although
still usable. I'm just afraid that it will keep getting slower with time at
the point of being unusable inside emacs+orgmode. I'm curious if there are
users with larger orgmode files and what their experience is, and open to
suggestions on how to improve performance.

Cheers,

Marcelo.

[-- Attachment #2: Type: text/html, Size: 719 bytes --]

Re: Largest org file you have + performance

[suvayu ali]

Hi Marcelo,

On Mon, Aug 1, 2011 at 6:35 PM, Marcelo de Moraes Serpa
<celoserpa@gmail.com> wrote:
> What's the largest orgmode file you have, and what's the performance you get
> while manipulating/navigating it? Mine is a 5k lines file called
> reference.org, and I basically keep all kind of notes and attachments there.

My largest agenda file used to be ~2k lines. Now I have archived many
things, so my 2nd largest agenda file is ~1.5k lines. I don't see any
performance problems. All agenda commands are almost instantaneous.

Hope this helps.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Largest org file you have + performance

[Marcelo de Moraes Serpa]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

What platform? OSX?

On Mon, Aug 1, 2011 at 11:54 AM, suvayu ali <fatkasuvayu+linux@gmail.com>wrote:

> Hi Marcelo,
>
> On Mon, Aug 1, 2011 at 6:35 PM, Marcelo de Moraes Serpa
> <celoserpa@gmail.com> wrote:
> > What's the largest orgmode file you have, and what's the performance you
> get
> > while manipulating/navigating it? Mine is a 5k lines file called
> > reference.org, and I basically keep all kind of notes and attachments
> there.
>
> My largest agenda file used to be ~2k lines. Now I have archived many
> things, so my 2nd largest agenda file is ~1.5k lines. I don't see any
> performance problems. All agenda commands are almost instantaneous.
>
> Hope this helps.
>
> --
> Suvayu
>
> Open source is the future. It sets us free.
>

[-- Attachment #2: Type: text/html, Size: 1212 bytes --]

Re: Largest org file you have + performance

[Florian Beck]

Marcelo de Moraes Serpa <celoserpa@gmail.com> writes:

> Hey list,
>
>
> What's the largest orgmode file you have, and what's the performance you get while manipulating/navigating it? Mine is a 5k lines
> file called reference.org, and I basically keep all kind of notes and attachments there. It's a bit slow to navigate on my Emacs
> 240.50.1, orgmode 7.4, although still usable. I'm just afraid that it will keep getting slower with time at the point of being
> unusable inside emacs+orgmode. I'm curious if there are users with larger orgmode files and what their experience is, and open to
> suggestions on how to improve performance.

emacs-bzr, org-7.7

The largest file I have is 7.2M. Some 6300 entries, 137k lines. Agenda
commands are slow, of course, but otherwise the performance is
acceptable. Anything that involves completion is quite slow. Opening ID
links is also slower than I would like. Anything that works on the whole
buffer can take a long time (cycling from OVERVIEW to CONTENTS takes more
than a minute), so try to avoid that. (I keep old notes in that file, so
mostly I search or link to entries.)

If you are using the current bzr version of emacs, make sure to set
`bidi-display-reordering' to nil.

>
>
>
> Cheers,
>
>
>
> Marcelo.

-- 
Florian Beck

Re: Largest org file you have + performance

[Eric Abrahamsen]

On Mon, Aug 01 2011, Marcelo de Moraes Serpa wrote:

> Hey list,
>
> What's the largest orgmode file you have, and what's the performance
> you get while manipulating/navigating it? Mine is a 5k lines file
> called reference.org, and I basically keep all kind of notes and
> attachments there. It's a bit slow to navigate on my Emacs 240.50.1,
> orgmode 7.4, although still usable. I'm just afraid that it will keep
> getting slower with time at the point of being unusable inside
> emacs+orgmode. I'm curious if there are users with larger orgmode
> files and what their experience is, and open to suggestions on how to
> improve performance.

My largest file is 32k lines. It's a novel translation (so essentially
two very long novels in one file), and there's no perceptible lag in
navigation/folding. BUT! It's almost all plain text, very little
fontification besides the headlines, and I understand that fontification
is one of the major causes of lag. So, dunno if that's helpful, but it's
another data point.

Eric

(Using the Ubuntu version of Emacs, 23.3 I believe, and yesterday's org-mode)

Re: Largest org file you have + performance

[suvayu ali]

On Mon, Aug 1, 2011 at 7:06 PM, Marcelo de Moraes Serpa
<celoserpa@gmail.com> wrote:
> What platform? OSX?

Linux x86_64

I also have other files which are larger (~3k lines) but they are not
in the agenda files. These other files have lots of source code blocks
and are meant for exporting to pdf via latex.

-- 
Suvayu

Open source is the future. It sets us free.

Fwd: Largest org file you have + performance

[Marcelo de Moraes Serpa]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

---------- Forwarded message ----------
From: Marcelo de Moraes Serpa <celoserpa@gmail.com>
Date: Mon, Aug 1, 2011 at 3:33 PM
Subject: Re: [O] Largest org file you have + performance
To: suvayu ali <fatkasuvayu+linux@gmail.com>


I think emacs on OSX underperforms significantly, at least comparing to its
Linux counterpart. I might be wrong...

The reference file is slow by itself, agenda commands are slow, but
acceptable.


On Mon, Aug 1, 2011 at 3:27 PM, suvayu ali <fatkasuvayu+linux@gmail.com>wrote:

> On Mon, Aug 1, 2011 at 7:06 PM, Marcelo de Moraes Serpa
> <celoserpa@gmail.com> wrote:
> > What platform? OSX?
>
> Linux x86_64
>
> I also have other files which are larger (~3k lines) but they are not
> in the agenda files. These other files have lots of source code blocks
> and are meant for exporting to pdf via latex.
>
> --
> Suvayu
>
> Open source is the future. It sets us free.
>

[-- Attachment #2: Type: text/html, Size: 1631 bytes --]

Re: Largest org file you have + performance

[Bernt Hansen]

Marcelo de Moraes Serpa <celoserpa@gmail.com> writes:

> What's the largest orgmode file you have, and what's the performance
> you get while manipulating/navigating it? Mine is a 5k lines file
> called reference.org, and I basically keep all kind of notes and
> attachments there. It's a bit slow to navigate on my Emacs 240.50.1,
> orgmode 7.4, although still usable. I'm just afraid that it will keep
> getting slower with time at the point of being unusable inside
> emacs+orgmode. I'm curious if there are users with larger orgmode
> files and what their experience is, and open to suggestions on how to
> improve performance.

My largest .org file is 6120 lines, 306741 bytes but I have 51 files
contributing to my agenda for a total of 29488 lines, 1,263,120 bytes
total.

My archive files are much bigger but not normally accessed.  This is on
a 32 bit linux system and the only slow thing I have is 5 seconds to
generate my block agenda which includes all of my org files.

Archiving things you don't need is the best way to keep things fast I
think.  You can still search the archive from the agenda if necessary.

Regards,
Bernt

Performance of org-mode with BIDI (was: Largest org file you have + performance)

[Reiner Steib]

On Mon, Aug 01 2011, Florian Beck wrote:

> If you are using the current bzr version of emacs, make sure to set
> `bidi-display-reordering' to nil.

Please don't.  Instead, report problems caused by
`bidi-display-reordering' with M-x report-emacs-bug RET so the Emacs
developers can fix the problem.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Performance of org-mode with BIDI

[Florian Beck]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Mon, Aug 01 2011, Florian Beck wrote:
>
>> If you are using the current bzr version of emacs, make sure to set
>> `bidi-display-reordering' to nil.
>
> Please don't.  Instead, report problems caused by
> `bidi-display-reordering' with M-x report-emacs-bug RET so the Emacs
> developers can fix the problem.

And, as of now, the problem is already fixed.

-- 
Florian Beck

Re: Largest org file you have + performance

[Milan Zamazal]

My largest org file is an org-drill file of almost 1 MB size, containing
about 32000 lines (most of them are entry properties) and 2000 org-drill
entries.  It's well useable but tag and property editing is better to be
done by hand instead of using org commands and I have to use some
folding trick to prevent org-drill performance problems.

Re: Largest org file you have + performance

[jiangzuoyan]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

My largest org file is QuanSongCi.org, contains ci poetry of Song dynasty,
about 6MB size.

Editing in emacs is very slow, save needs to wait about minutes. convert to
html needs to wait about minutes, too. So, sometimes, sed like tools is
preferred to do editing and converting.

                                                      Changsheng Jiang


On Fri, Aug 5, 2011 at 04:18, Milan Zamazal <pdm@zamazal.org> wrote:

> My largest org file is an org-drill file of almost 1 MB size, containing
> about 32000 lines (most of them are entry properties) and 2000 org-drill
> entries.  It's well useable but tag and property editing is better to be
> done by hand instead of using org commands and I have to use some
> folding trick to prevent org-drill performance problems.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1141 bytes --]

Re: Largest org file you have + performance

[Matt Lundin]

"jiangzuoyan@gmail.com" <jiangzuoyan@gmail.com> writes:

> My largest org file is QuanSongCi.org, contains ci poetry of Song
> dynasty, about 6MB size.
>
> Editing in emacs is very slow, save needs to wait about
> minutes. convert to html needs to wait about minutes, too. So,
> sometimes, sed like tools is preferred to do editing and converting.

Thanks for these details. I can report similar experiences with large files.

As far as I understand, large org-mode buffers are slow because there
are many overlays, which (unlike text properties), don't scale well:

(info "(elisp) Overlays")

This is right now an inescapable limitation of org-mode, since
outline-mode relies on overlays.

There was brief talk on emacs-dev of rewriting text-properties so that
outline-mode could use them instead of overlays, but I do not believe
the proposal was pursued further:

http://permalink.gmane.org/gmane.emacs.devel/131304

Best,
Matt

Re: Largest org file you have + performance

[Bastien]

Hi Matt,

Matt Lundin <mdl@imapmail.org> writes:

> There was brief talk on emacs-dev of rewriting text-properties so that
> outline-mode could use them instead of overlays, but I do not believe
> the proposal was pursued further:
>
> http://permalink.gmane.org/gmane.emacs.devel/131304

Quite interesting, thanks for the link.  I would love to see some
optimization in this area.

-- 
 Bastien

Re: Largest org file you have + performance

[SW]

I'm not familiar enough with Emacs to comment about the performance of overlays,
etc., but I'm surprised that processing *text* can be so CPU intensive.

These days we have games running with millions of pixels and shading or whatever
(some use the GPU of course), and browsers with fancy Flash animation and dozens
of tabs open and spreadsheets with thousands of cells and word processor
documents with hundreds of formatted pages on dual cores and quad cores with GB
of RAM.

Is this performance overhead just a result of poorly-scalable overlays, rather
than an inherent shortcoming of Emacs and/or org-mode?

Re: Largest org file you have + performance

[Bastien]

Hi,

SW <sabrewolfy@gmail.com> writes:

> Is this performance overhead just a result of poorly-scalable overlays, rather
> than an inherent shortcoming of Emacs and/or org-mode?

I would say this is a combination of the implementation overlays
and the use of overlays in outline.el.

Perhaps we should dive into outline.el and see how we can improve
things from there.  (We tend to forget that Org is just a derived
mode from outline.el!)

-- 
 Bastien