Extremely slow org-table operations

Extremely slow org-table operations

[York Zhao]

[-- Attachment #1: Type: text/plain, Size: 1989 bytes --]

Hi list,

A few months ago I tried Emacs 24.4 and was disappointed because almost all of
the org-table operations became AT LEAST ten times slower. Because recently
Emacs 24.4 gets officially released, I finally switched over. However, the
problems I had several months ago were still there, i.e., in a big file,
org-table operations are at least ten times slower. I tried profiling it but
didn't help much. So I decided to report the issue here.

I've created an ECM, please find attached the zip file which includes four
files:

1. test-org: a bash simple script file having a single line:

   emacs -Q --load test.el

2. test.el, with the following contents:
   (find-file "test.org")
   (show-all)
   (goto-char (point-max))
   (forward-line -1)
   (profiler-start 'cpu)
   (org-ctrl-c-ctrl-c)
   (profiler-report)
   (profiler-stop)

3. test.org: the org-mode file to reproduce the issue.

4. profiler-report: the "Profiler Report" file
   This file was created by pressing "C-x C-w" (as described in the Elisp Info)
   from a profile report buffer. However, I have no idea how to reproduce the
   "Profiler Report Buffer" from this file. I've tried M-x profiler-report-mode
   but it didn't work. If anyone knows how, please let me know and I would
   appreciate it.

In summery, the file "test.org" has more than 5000 org-table tables, each table
has a formula. If you move the point to the tables not too far away from the
beginning of the buffer and press "C-c C-c" on the formula line, you will notice
that the speed is very fast which is great. However, if you move the point far
away enough from the beginning of the buffer (move to the end for example), you
will find that "C-c C-c" on the formula line becomes much slower.

Many other operations are also very slow. However, because I haven't been able
to create the ECMs for those operations yet, lets focus on this one first.

Please let me know what else I can do to help addressing this issue.


Thanks in advance,

York

[-- Attachment #2: Test.zip --]
[-- Type: application/zip, Size: 29532 bytes --]

Re: Extremely slow org-table operations

[Nicolas Goaziou]

Hello,

York Zhao <gtdplatform@gmail.com> writes:

> A few months ago I tried Emacs 24.4 and was disappointed because almost all of
> the org-table operations became AT LEAST ten times slower. Because recently
> Emacs 24.4 gets officially released, I finally switched over. However, the
> problems I had several months ago were still there, i.e., in a big file,
> org-table operations are at least ten times slower. I tried profiling it but
> didn't help much. So I decided to report the issue here.

[...]

> In summery, the file "test.org" has more than 5000 org-table tables, each table
> has a formula. If you move the point to the tables not too far away from the
> beginning of the buffer and press "C-c C-c" on the formula line, you will notice
> that the speed is very fast which is great. However, if you move the point far
> away enough from the beginning of the buffer (move to the end for example), you
> will find that "C-c C-c" on the formula line becomes much slower.
>
> Many other operations are also very slow. However, because I haven't been able
> to create the ECMs for those operations yet, lets focus on this one
> first.

I think Org 8.3 should improve the situation. Please test it if you can.
Meanwhile, I suggest to insert some subsections in your file.


Regards,

-- 
Nicolas Goaziou

Re: Extremely slow org-table operations

[York Zhao]

> I think Org 8.3 should improve the situation. Please test it if you can.

I was aware that my "emacs -Q" test was done against org-mode version 8.2.x.
However, I use the latest org-mode from Git everyday. For example, the version
I'm currently using is (from M-x org-version): "Org-mode version 8.3beta
(release_8.3beta-509-g86ec0a.dirty @ /usr/local/share/emacs/site-lisp/org/)". So
apparently the problem had never been addressed.

> Meanwhile, I suggest to insert some subsections in your file.

What do you mean by "insert some subsections"?


Thanks


On Wed, Oct 29, 2014 at 5:43 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Hello,
>
> York Zhao <gtdplatform@gmail.com> writes:
>
>> A few months ago I tried Emacs 24.4 and was disappointed because almost all of
>> the org-table operations became AT LEAST ten times slower. Because recently
>> Emacs 24.4 gets officially released, I finally switched over. However, the
>> problems I had several months ago were still there, i.e., in a big file,
>> org-table operations are at least ten times slower. I tried profiling it but
>> didn't help much. So I decided to report the issue here.
>
> [...]
>
>> In summery, the file "test.org" has more than 5000 org-table tables, each table
>> has a formula. If you move the point to the tables not too far away from the
>> beginning of the buffer and press "C-c C-c" on the formula line, you will notice
>> that the speed is very fast which is great. However, if you move the point far
>> away enough from the beginning of the buffer (move to the end for example), you
>> will find that "C-c C-c" on the formula line becomes much slower.
>>
>> Many other operations are also very slow. However, because I haven't been able
>> to create the ECMs for those operations yet, lets focus on this one
>> first.
>
> I think Org 8.3 should improve the situation. Please test it if you can.
> Meanwhile, I suggest to insert some subsections in your file.
>
>
> Regards,
>
> --
> Nicolas Goaziou

Re: Extremely slow org-table operations

[Charles Millar]

Hi,

York Zhao wrote:
>> I think Org 8.3 should improve the situation. Please test it if you can.
> I was aware that my "emacs -Q" test was done against org-mode version 8.2.x.
> However, I use the latest org-mode from Git everyday. For example, the version
> I'm currently using is (from M-x org-version): "Org-mode version 8.3beta
> (release_8.3beta-509-g86ec0a.dirty @ /usr/local/share/emacs/site-lisp/org/)". So
> apparently the problem had never been addressed.
>
>> Meanwhile, I suggest to insert some subsections in your file.
> What do you mean by "insert some subsections"?
>
>
> Thanks
>
>
> On Wed, Oct 29, 2014 at 5:43 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>> Hello,
>>
>> York Zhao <gtdplatform@gmail.com> writes:
>>
>>> A few months ago I tried Emacs 24.4 and was disappointed because almost all of
>>> the org-table operations became AT LEAST ten times slower. Because recently
>>> Emacs 24.4 gets officially released, I finally switched over. However, the
>>> problems I had several months ago were still there, i.e., in a big file,
>>> org-table operations are at least ten times slower. I tried profiling it but
>>> didn't help much. So I decided to report the issue here.
>> [...]
>>
>>> In summery, the file "test.org" has more than 5000 org-table tables, each table
>>> has a formula. If you move the point to the tables not too far away from the
>>> beginning of the buffer and press "C-c C-c" on the formula line, you will notice
>>> that the speed is very fast which is great. However, if you move the point far
>>> away enough from the beginning of the buffer (move to the end for example), you
>>> will find that "C-c C-c" on the formula line becomes much slower.
>>>
>>> Many other operations are also very slow. However, because I haven't been able
>>> to create the ECMs for those operations yet, lets focus on this one
>>> first.
>> I think Org 8.3 should improve the situation. Please test it if you can.
>> Meanwhile, I suggest to insert some subsections in your file.
>>
>>
>> Regards,
>>
>> --
>> Nicolas Goaziou
>
IIRC Carsten Dominik made the following observation: org tables are 
extremely slow if they are used as workbooks/spreadsheets and there are 
many entries (many is undefined). He recommended that a spreadsheet 
program in such situations.

I learned that the hard way when I had one table - four columns, three 
simple addition formulas with about 1,000 entries. It seemed an eternity 
before the addition was completed.

Charlie Millar

Re: Extremely slow org-table operations

[Michael Brand]

Hi Charles

First I would like to point out that to my understanding there are now
two separate issues in discussion in this thread which are easy to mix
up:

1) The same table calculates fast or slow depending on the position
   within the same entry, see OP.

2) An entry with nothing but a single table, as you describe it:

On Thu, Oct 30, 2014 at 12:40 AM, Charles Millar <millarc@verizon.net> wrote:
> I learned that the hard way when I had one table - four columns, three
> simple addition formulas with about 1,000 entries. It seemed an eternity
> before the addition was completed.

Only out of curiosity I would like to see the TBLFM. And do you see
still the same slowness with 8.3beta?

Michael

Re: Extremely slow org-table operations

[Charles Millar]

Hi Michael,

Michael Brand wrote:
> Hi Charles
>
> First I would like to point out that to my understanding there are now
> two separate issues in discussion in this thread which are easy to mix
> up:
>
> 1) The same table calculates fast or slow depending on the position
>     within the same entry, see OP.
I did not understand; I thought that the speed slowed down as the OP 
progressed from table to table in the file.
> 2) An entry with nothing but a single table, as you describe it:
True; I did not bother to mention other files in which I had multiple 
tables, some with references to other tables in the file. At times "M-x 
org-table-iterate-buffer-tables" seemed to take eternity, squared. 
(Imagine a smiley face placed here.)
> On Thu, Oct 30, 2014 at 12:40 AM, Charles Millar <millarc@verizon.net> wrote:
>> I learned that the hard way when I had one table - four columns, three
>> simple addition formulas with about 1,000 entries. It seemed an eternity
>> before the addition was completed.
> Only out of curiosity I would like to see the TBLFM. And do you see
> still the same slowness with 8.3beta?
It has been quite a while and I suspect that my experience may have been 
with org <8.0. I try to find the older files, give them a run and let 
you know.  I will also give org another shot using multiple tables, as 
spreadsheets, in one file. The worst that can happen is that I will have 
to export the tables.

Charlie

Re: Extremely slow org-table operations

[York Zhao]

[-- Attachment #1: Type: text/plain, Size: 3329 bytes --]

@Charlie Millar:

> IIRC Carsten Dominik made the following observation: org tables are
extremely
> slow if they are used as workbooks/spreadsheets and there are many entries
> (many is undefined).

Thanks for the information, could you please clarify what "entry" means?
Does it
mean org headline, or a row in an org-table?

> I learned that the hard way when I had one table - four columns, three
simple
> addition formulas with about 1,000 entries. It seemed an eternity before
the
> addition was completed.

I guess the "entries" here mean the table rows right? Please confirm.


York


On Wed, Oct 29, 2014 at 7:40 PM, Charles Millar <millarc@verizon.net> wrote:

> Hi,
>
>
> York Zhao wrote:
>
>> I think Org 8.3 should improve the situation. Please test it if you can.
>>>
>> I was aware that my "emacs -Q" test was done against org-mode version
>> 8.2.x.
>> However, I use the latest org-mode from Git everyday. For example, the
>> version
>> I'm currently using is (from M-x org-version): "Org-mode version 8.3beta
>> (release_8.3beta-509-g86ec0a.dirty @ /usr/local/share/emacs/site-lisp/org/)".
>> So
>> apparently the problem had never been addressed.
>>
>>  Meanwhile, I suggest to insert some subsections in your file.
>>>
>> What do you mean by "insert some subsections"?
>>
>>
>> Thanks
>>
>>
>> On Wed, Oct 29, 2014 at 5:43 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr>
>> wrote:
>>
>>> Hello,
>>>
>>> York Zhao <gtdplatform@gmail.com> writes:
>>>
>>>  A few months ago I tried Emacs 24.4 and was disappointed because almost
>>>> all of
>>>> the org-table operations became AT LEAST ten times slower. Because
>>>> recently
>>>> Emacs 24.4 gets officially released, I finally switched over. However,
>>>> the
>>>> problems I had several months ago were still there, i.e., in a big file,
>>>> org-table operations are at least ten times slower. I tried profiling
>>>> it but
>>>> didn't help much. So I decided to report the issue here.
>>>>
>>> [...]
>>>
>>>  In summery, the file "test.org" has more than 5000 org-table tables,
>>>> each table
>>>> has a formula. If you move the point to the tables not too far away
>>>> from the
>>>> beginning of the buffer and press "C-c C-c" on the formula line, you
>>>> will notice
>>>> that the speed is very fast which is great. However, if you move the
>>>> point far
>>>> away enough from the beginning of the buffer (move to the end for
>>>> example), you
>>>> will find that "C-c C-c" on the formula line becomes much slower.
>>>>
>>>> Many other operations are also very slow. However, because I haven't
>>>> been able
>>>> to create the ECMs for those operations yet, lets focus on this one
>>>> first.
>>>>
>>> I think Org 8.3 should improve the situation. Please test it if you can.
>>> Meanwhile, I suggest to insert some subsections in your file.
>>>
>>>
>>> Regards,
>>>
>>> --
>>> Nicolas Goaziou
>>>
>>
>>  IIRC Carsten Dominik made the following observation: org tables are
> extremely slow if they are used as workbooks/spreadsheets and there are
> many entries (many is undefined). He recommended that a spreadsheet program
> in such situations.
>
> I learned that the hard way when I had one table - four columns, three
> simple addition formulas with about 1,000 entries. It seemed an eternity
> before the addition was completed.
>
> Charlie Millar
>
>

[-- Attachment #2: Type: text/html, Size: 4708 bytes --]

Fwd: Re: Extremely slow org-table operations

[Charles Millar]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: Re: [O] Extremely slow org-table operations.eml --]
[-- Type: message/rfc822, Size: 2287 bytes --]

From: Charles Millar <millarc@verizon.net>
To: Michael Brand <michael.ch.brand@gmail.com>, Org Mode <emacs-orgmode@gnu.org>
Subject: Re: [O] Extremely slow org-table operations
Date: Thu, 30 Oct 2014 08:39:11 -0400
Message-ID: <5452316F.5060203@verizon.net>

Hi Michael,

Michael Brand wrote:
> Hi Charles
>
> First I would like to point out that to my understanding there are now
> two separate issues in discussion in this thread which are easy to mix
> up:
>
> 1) The same table calculates fast or slow depending on the position
>     within the same entry, see OP.
I did not understand; I thought that the speed slowed down as the OP 
progressed from table to table in the file.
> 2) An entry with nothing but a single table, as you describe it:
True; I did not bother to mention other files in which I had multiple 
tables, some with references to other tables in the file. At times "M-x 
org-table-iterate-buffer-tables" seemed to take eternity, squared. 
(Imagine a smiley face placed here.)
> On Thu, Oct 30, 2014 at 12:40 AM, Charles Millar <millarc@verizon.net> wrote:
>> I learned that the hard way when I had one table - four columns, three
>> simple addition formulas with about 1,000 entries. It seemed an eternity
>> before the addition was completed.
> Only out of curiosity I would like to see the TBLFM. And do you see
> still the same slowness with 8.3beta?
It has been quite a while and I suspect that my experience may have been 
with org <8.0. I try to find the older files, give them a run and let 
you know.  I will also give org another shot using multiple tables, as 
spreadsheets, in one file. The worst that can happen is that I will have 
to export the tables.

Charlie

Re: Extremely slow org-table operations

[Nicolas Goaziou]

Hello,

Michael Brand <michael.ch.brand@gmail.com> writes:

> 1) The same table calculates fast or slow depending on the position
>    within the same entry, see OP.

Note that this is due to org-table.el overusing `org-current-line'. It
should be factored out of the library at some point.


Regards,

-- 
Nicolas Goaziou

Re: Extremely slow org-table operations

[Charles Millar]

Earlier today I attempted to send the following.
Charles Millar wrote:
> Hi Michael,
>
> Michael Brand wrote:
>> Hi Charles
>>
>> First I would like to point out that to my understanding there are now
>> two separate issues in discussion in this thread which are easy to mix
>> up:
>>
>> 1) The same table calculates fast or slow depending on the position
>>     within the same entry, see OP.
> I did not understand; I thought that the speed slowed down as the OP 
> progressed from table to table in the file.
>> 2) An entry with nothing but a single table, as you describe it:
> True; I did not bother to mention other files in which I had multiple 
> tables, some with references to other tables in the file. At times 
> "M-x org-table-iterate-buffer-tables" seemed to take eternity, 
> squared. (Imagine a smiley face placed here.)
>> On Thu, Oct 30, 2014 at 12:40 AM, Charles Millar 
>> <millarc@verizon.net> wrote:
>>> I learned that the hard way when I had one table - four columns, three
>>> simple addition formulas with about 1,000 entries. It seemed an 
>>> eternity
>>> before the addition was completed.
>> Only out of curiosity I would like to see the TBLFM. And do you see
>> still the same slowness with 8.3beta?
> It has been quite a while and I suspect that my experience may have 
> been with org <8.0. I try to find the older files, give them a run and 
> let you know.  I will also give org another shot using multiple 
> tables, as spreadsheets, in one file. The worst that can happen is 
> that I will have to export the tables.
>
> Charlie

Re: Extremely slow org-table operations

[Charles Millar]

Hi Michael

 > Michael Brand wrote:
 >> Hi Charles
 >>
 >> First I would like to point out that to my understanding there are now
 >> two separate issues in discussion in this thread which are easy to mix
 >> up:
 >>
 >> 1) The same table calculates fast or slow depending on the position
 >>     within the same entry, see OP.
 > I did not understand; I thought that the speed slowed down as the OP 
progressed from table to table in the file.
 >> 2) An entry with nothing but a single table, as you describe it:
 > True; I did not bother to mention other files in which I had multiple 
tables, some with references to other tables in the file. At times "M-x 
org-table-iterate-buffer-tables" seemed to take eternity, squared. 
(Imagine a smiley face placed here.)
 >> On Thu, Oct 30, 2014 at 12:40 AM, Charles Millar 
<millarc@verizon.net> wrote:
 >>> I learned that the hard way when I had one table - four columns, three
 >>> simple addition formulas with about 1,000 entries. It seemed an 
eternity
 >>> before the addition was completed.
 >> Only out of curiosity I would like to see the TBLFM. And do you see
 >> still the same slowness with 8.3beta?
 > It has been quite a while and I suspect that my experience may have 
been with org <8.0. I try to find the older files, give them a run and 
let you know.  I will also give org another shot using multiple tables, 
as spreadsheets, in one file. The worst that can happen is that I will 
have to export the tables.
 >
 > Charlie

Re: Extremely slow org-table operations

[Charles Millar]

Hi York,

York Zhao wrote:
> @Charlie Millar:
>
> > IIRC Carsten Dominik made the following observation: org tables are 
> extremely
> > slow if they are used as workbooks/spreadsheets and there are many 
> entries
> > (many is undefined).
>
> Thanks for the information, could you please clarify what "entry" 
> means? Does it
> mean org headline, or a row in an org-table?
>
> > I learned that the hard way when I had one table - four columns, 
> three simple
> > addition formulas with about 1,000 entries. It seemed an eternity 
> before the
> > addition was completed.
>
> I guess the "entries" here mean the table rows right? Please confirm.
>
You are correct; I should have said rows. In my file there were 1000 
(+/-) rows and each row had up to three "entries", not including the 
description in the first row.
For instance (without any formulas) in the following row I entered each 
amount in columns 2,3 & 4.

     | this was a transaction | 100.00 | 200.00 | 300.00|

So I considered this three entries. So actually there were 3,000 (+/-) 
entries.

Charlie

Re: Extremely slow org-table operations

[York Zhao]

>> > I learned that the hard way when I had one table - four columns, three
>> > simple
>> > addition formulas with about 1,000 entries. It seemed an eternity before
>> > the
>> > addition was completed.
>>
>> I guess the "entries" here mean the table rows right? Please confirm.
>>
> You are correct; I should have said rows. In my file there were 1000 (+/-)
> rows and each row had up to three "entries", not including the description
> in the first row.
> For instance (without any formulas) in the following row I entered each
> amount in columns 2,3 & 4.
>
>     | this was a transaction | 100.00 | 200.00 | 300.00|
>
> So I considered this three entries. So actually there were 3,000 (+/-)
> entries.

OK, now I see. Charlie's problem is actually completed different from this
problem. So what Carsten Dominik mentioned doesn't apply. This is because in
Charlie's case there are 1000 rows in a table, whereas mine has only two rows.
Since my formula only calculates two rows, and calculating the table doesn't
involve data input from anywhere else, it really doesn't make sens that it has
to be slow.

Needlessly to say that `org-mode' is fantastic, but with this issue, I have to
say that `org-mode' is unhealthy. Therefore, I really hope this issue gets
addressed. Please let me know what I can do to help.


Thanks in advance,

York


On Fri, Oct 31, 2014 at 3:12 PM, Charles Millar <millarc@verizon.net> wrote:
> Hi York,
>
> York Zhao wrote:
>>
>> @Charlie Millar:
>>
>> > IIRC Carsten Dominik made the following observation: org tables are
>> > extremely
>> > slow if they are used as workbooks/spreadsheets and there are many
>> > entries
>> > (many is undefined).
>>
>> Thanks for the information, could you please clarify what "entry" means?
>> Does it
>> mean org headline, or a row in an org-table?
>>
>> > I learned that the hard way when I had one table - four columns, three
>> > simple
>> > addition formulas with about 1,000 entries. It seemed an eternity before
>> > the
>> > addition was completed.
>>
>> I guess the "entries" here mean the table rows right? Please confirm.
>>
> You are correct; I should have said rows. In my file there were 1000 (+/-)
> rows and each row had up to three "entries", not including the description
> in the first row.
> For instance (without any formulas) in the following row I entered each
> amount in columns 2,3 & 4.
>
>     | this was a transaction | 100.00 | 200.00 | 300.00|
>
> So I considered this three entries. So actually there were 3,000 (+/-)
> entries.
>
> Charlie

Re: Extremely slow org-table operations

[Nicolas Goaziou]

York Zhao <gtdplatform@gmail.com> writes:

> Needlessly to say that `org-mode' is fantastic, but with this issue, I have to
> say that `org-mode' is unhealthy. Therefore, I really hope this issue gets
> addressed. Please let me know what I can do to help.

With your initial file, benchmarking a C-c C-c on the last table, I get:

  Elapsed time: 2.129800s

This is not fast, but nothing unbearable either. The same on the first
table gives

  Elapsed time: 1.195026s

As I said in this thread, the additional cost comes from the use of
`org-current-line' in "org-table.el".


Regards,