* working with tables can be quite painful...
@ 2016-09-16 14:53 Eric S Fraga
2016-09-17 8:48 ` Nicolas Goaziou
` (5 more replies)
0 siblings, 6 replies; 12+ messages in thread
From: Eric S Fraga @ 2016-09-16 14:53 UTC (permalink / raw)
To: Emacs Org mode mailing list
Hello,
I am working with a table. It has approximately 130 rows and 20 columns
so it's not huge but also not small. Three columns are text but the
rest are all numbers with some degree of sparsity. Instrumenting org
while working on this table, manipulating the entries in just one row, I
get the following:
| Function | calls | elapsed time | average time |
|--------------------------+-------+--------------+--------------|
| org-cycle | 20 | 41.550963140 | 2.0775481570 |
| org-table-next-field | 20 | 41.544266727 | 2.0772133363 |
| org-table-align | 5 | 41.470595702 | 8.2941191404 |
| org-mode-flyspell-verify | 52 | 1.0647362189 | 0.0204756965 |
| org-do-latex-and-related | 21 | 0.6656267140 | 0.0316965101 |
| org-element-at-point | 125 | 0.6356939890 | 0.0050855519 |
| org-element--parse-to | 125 | 0.6086256940 | 0.0048690055 |
| org-element--cache-put | 1399 | 0.4963533770 | 0.0003547915 |
From this, it would seem that the table align function is killing the
performance. 8 seconds per call? On an 8 core Intel(R) Core(TM)
i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
slowness!
This is with a not quite up to date org. I'm avoiding upgrading as I am
preparing material for teaching which starts soon and I don't want to
run into issues due to changes in org... so I apologise if things have
changed recently. This performance issue has existed for quite some time
now, however.
Any suggestions on speeding things up?
Thanks,
eric
--
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: working with tables can be quite painful...
2016-09-16 14:53 working with tables can be quite painful Eric S Fraga
@ 2016-09-17 8:48 ` Nicolas Goaziou
2016-09-17 15:22 ` Adam Porter
2016-09-17 15:37 ` Michael Brand
` (4 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Nicolas Goaziou @ 2016-09-17 8:48 UTC (permalink / raw)
To: Emacs Org mode mailing list
Hello,
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> I am working with a table. It has approximately 130 rows and 20 columns
> so it's not huge but also not small. Three columns are text but the
> rest are all numbers with some degree of sparsity. Instrumenting org
> while working on this table, manipulating the entries in just one row, I
> get the following:
>
> | Function | calls | elapsed time | average time |
> |--------------------------+-------+--------------+--------------|
> | org-cycle | 20 | 41.550963140 | 2.0775481570 |
> | org-table-next-field | 20 | 41.544266727 | 2.0772133363 |
> | org-table-align | 5 | 41.470595702 | 8.2941191404 |
> | org-mode-flyspell-verify | 52 | 1.0647362189 | 0.0204756965 |
> | org-do-latex-and-related | 21 | 0.6656267140 | 0.0316965101 |
> | org-element-at-point | 125 | 0.6356939890 | 0.0050855519 |
> | org-element--parse-to | 125 | 0.6086256940 | 0.0048690055 |
> | org-element--cache-put | 1399 | 0.4963533770 | 0.0003547915 |
>
> From this, it would seem that the table align function is killing the
> performance. 8 seconds per call? On an 8 core Intel(R) Core(TM)
> i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
> slowness!
>
> This is with a not quite up to date org. I'm avoiding upgrading as I am
> preparing material for teaching which starts soon and I don't want to
> run into issues due to changes in org... so I apologise if things have
> changed recently. This performance issue has existed for quite some time
> now, however.
>
> Any suggestions on speeding things up?
Could you send a profiler report so that I can get a better glimpse on
what part of `org-table-align' is lagging?
An ECM would help too.
Regards,
--
Nicolas Goaziou
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: working with tables can be quite painful...
2016-09-17 8:48 ` Nicolas Goaziou
@ 2016-09-17 15:22 ` Adam Porter
0 siblings, 0 replies; 12+ messages in thread
From: Adam Porter @ 2016-09-17 15:22 UTC (permalink / raw)
To: emacs-orgmode
In case this helps anyone, I've found this code makes profiling a lot
easier. It automatically instruments the desired functions, runs the
code you want to test, removes the instrumentation, and presents the
results.
#+BEGIN_SRC elisp
(defmacro profile-org (times &rest body)
`(let (output)
(dolist (p '("org-")) ; symbol prefixes to instrument
(elp-instrument-package p))
(dotimes (x ,times)
,@body)
(elp-results)
(elp-restore-all)
(point-min)
(forward-line 20)
(delete-region (point) (point-max))
(setq output (buffer-substring-no-properties (point-min) (point-max)))
(kill-buffer)
(delete-window)
output))
;; Used like this:
(profile-org 10
(org-table-next-field)
(org-table-align))
#+END_SRC
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: working with tables can be quite painful...
2016-09-16 14:53 working with tables can be quite painful Eric S Fraga
2016-09-17 8:48 ` Nicolas Goaziou
@ 2016-09-17 15:37 ` Michael Brand
[not found] ` <9ae6591e2754413885a96e19c2455472@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
` (3 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: Michael Brand @ 2016-09-17 15:37 UTC (permalink / raw)
To: Emacs Org mode mailing list
Hi Eric
Question, out of curiosity: Is there a difference when you delete all lines
above and below the table, with and without adding a headline above?
One of my tables fluctuates around 150 rows and around 20 to 40
columns, overall a few hundred characters wide (columns with some
history window for each row). Almost all fields are just text although
I think this doesn't matter at all. The table starts at about line 50
and is followed by about 7600 lines. Since years I manipulate it at
least every few days without any slowness, often on much weaker
hardware than yours.
Maybe you can find a temporary workaround.
Michael
^ permalink raw reply [flat|nested] 12+ messages in thread[parent not found: <9ae6591e2754413885a96e19c2455472@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>]
* Re: working with tables can be quite painful...
[not found] ` <9ae6591e2754413885a96e19c2455472@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-19 8:33 ` Eric S Fraga
2016-09-19 12:58 ` Michael Welle
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Eric S Fraga @ 2016-09-19 8:33 UTC (permalink / raw)
To: Emacs Org mode mailing list
On Saturday, 17 Sep 2016 at 08:48, Nicolas Goaziou wrote:
[...]
> Could you send a profiler report so that I can get a better glimpse on
> what part of `org-table-align' is lagging?
Hi Nicolas,
this morning, working on the same table is much less painful. I've run
the profile and did some movements and changes to the table and here is
the output from the profiler (I can send you the full report if you
wish).
I know that if I narrow the buffer to just the section (headline and
contents) that contains the table, the performance is
better. Anecdotally, it would seem that the performance degrades over
time so I wonder if there is a cache issue?
My gut feeling is that it is related to fontification as well.
--8<---------------cut here---------------start------------->8---
- command-execute 2420 53%
- call-interactively 2420 53%
- funcall-interactively 2325 51%
- pabbrev-expand-maybe 1244 27%
- if 1244 27%
- pabbrev-expand-maybe-full 1244 27%
- cond 1244 27%
- pabbrev-call-previous-tab-binding 1205 26%
- let 1205 26%
- if 1205 26%
- let 1205 26%
- if 1205 26%
- funcall 1205 26%
- org-cycle 1205 26%
- call-interactively 1198 26%
- funcall-interactively 1198 26%
- org-table-next-field 1198 26%
- org-table-align 1122 24%
+ font-lock-fontify-region 258 5%
+ mapcar 219 4%
+ org-indent-refresh-maybe 152 3%
apply 45 0%
org-element--cache-before-change 32 0%
org-element--cache-after-change 32 0%
+ jit-lock-after-change 16 0%
org-table-begin 9 0%
+ org-table-maybe-eval-formula 4 0%
org-table-end 4 0%
--8<---------------cut here---------------end--------------->8---
> An ECM would help too.
I cannot send the actual table I am working on as it relates to grades
for students. I will try to create a similar table with random entries.
thanks,
eric
--
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: working with tables can be quite painful...
2016-09-19 8:33 ` Eric S Fraga
@ 2016-09-19 12:58 ` Michael Welle
[not found] ` <df2f3c4ad6a34890ad82cd1d0d557003@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
2016-09-22 19:52 ` Nicolas Goaziou
2 siblings, 0 replies; 12+ messages in thread
From: Michael Welle @ 2016-09-19 12:58 UTC (permalink / raw)
To: emacs-orgmode
Hello,
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> On Saturday, 17 Sep 2016 at 08:48, Nicolas Goaziou wrote:
>
> [...]
>
>> Could you send a profiler report so that I can get a better glimpse on
>> what part of `org-table-align' is lagging?
>
> Hi Nicolas,
>
> this morning, working on the same table is much less painful. I've run
> the profile and did some movements and changes to the table and here is
> the output from the profiler (I can send you the full report if you
> wish).
>
> I know that if I narrow the buffer to just the section (headline and
> contents) that contains the table, the performance is
> better. Anecdotally, it would seem that the performance degrades over
> time so I wonder if there is a cache issue?
the output of your profiler run doesn't ring a bell, but a few years ago
I had the problem that after some uptime (my Emacs uptime is typical
several days, up to a few weeks) Org operations like building the agenda
became sloooow. Restarting Emacs fixed that, until the problem occurred
again after some time.
If I remember correctly I asked the list for advise. But it looked like
one of these problems only I have ;). Maybe some other lisp package had
its hands in the game, I don't know. Eventually, after some Emacs, Org
and other updates the problem was gone.
As I said, I don't think, you suffer from the same problem. But you said
you don't use the newest software versions, so just in case.
Regards
hmw
^ permalink raw reply [flat|nested] 12+ messages in thread[parent not found: <df2f3c4ad6a34890ad82cd1d0d557003@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>]
* Re: working with tables can be quite painful...
[not found] ` <df2f3c4ad6a34890ad82cd1d0d557003@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-19 14:23 ` Eric S Fraga
2016-09-19 17:54 ` Michael Welle
0 siblings, 1 reply; 12+ messages in thread
From: Eric S Fraga @ 2016-09-19 14:23 UTC (permalink / raw)
To: Michael Welle; +Cc: emacs-orgmode@gnu.org
On Monday, 19 Sep 2016 at 12:58, Michael Welle wrote:
[...]
> the output of your profiler run doesn't ring a bell, but a few years ago
> I had the problem that after some uptime (my Emacs uptime is typical
> several days, up to a few weeks) Org operations like building the agenda
> became sloooow. Restarting Emacs fixed that, until the problem occurred
> again after some time.
Interesting. The performance I was referring to last week was indeed in
an emacs that had been running for quite some time. I did close down
that emacs instance on Friday for unrelated reasons and the instance
this morning had only been running for minutes. Maybe the problem is
not org alone but relates to emacs as well.
> If I remember correctly I asked the list for advise. But it looked like
> one of these problems only I have ;). Maybe some other lisp package had
> its hands in the game, I don't know. Eventually, after some Emacs, Org
> and other updates the problem was gone.
Well, maybe it's a problem that others do have!
> As I said, I don't think, you suffer from the same problem. But you said
> you don't use the newest software versions, so just in case.
To clarify, when I said I wasn't using the latest version of org, it's
that I hadn't updated from git in a few weeks but I am using a snapshot
of emacs tracking the development version (and now I have org up to date).
--
: Eric S Fraga (0xFFFCF67D), Emacs 25.1.50.1, Org release_8.3.6-1149-g582233
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: working with tables can be quite painful...
2016-09-19 14:23 ` Eric S Fraga
@ 2016-09-19 17:54 ` Michael Welle
0 siblings, 0 replies; 12+ messages in thread
From: Michael Welle @ 2016-09-19 17:54 UTC (permalink / raw)
To: emacs-orgmode
Hello,
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> On Monday, 19 Sep 2016 at 12:58, Michael Welle wrote:
>
> [...]
>
>> the output of your profiler run doesn't ring a bell, but a few years ago
>> I had the problem that after some uptime (my Emacs uptime is typical
>> several days, up to a few weeks) Org operations like building the agenda
>> became sloooow. Restarting Emacs fixed that, until the problem occurred
>> again after some time.
>
> Interesting. The performance I was referring to last week was indeed in
> an emacs that had been running for quite some time. I did close down
> that emacs instance on Friday for unrelated reasons and the instance
> this morning had only been running for minutes. Maybe the problem is
> not org alone but relates to emacs as well.
yepp, that's what I thought. Maybe a minor mode or some function that
got evaluated only once in a while. Or something, that depends on the
input data of that something.
>> If I remember correctly I asked the list for advise. But it looked like
>> one of these problems only I have ;). Maybe some other lisp package had
>> its hands in the game, I don't know. Eventually, after some Emacs, Org
>> and other updates the problem was gone.
>
> Well, maybe it's a problem that others do have!
;)
>> As I said, I don't think, you suffer from the same problem. But you said
>> you don't use the newest software versions, so just in case.
>
> To clarify, when I said I wasn't using the latest version of org, it's
> that I hadn't updated from git in a few weeks but I am using a snapshot
> of emacs tracking the development version (and now I have org up to date).
Hm, my problem was definitely way more in the past. The only performance
related problem I have from time to time is when the vc-mode hooks get
in the way. But that's nothing Org can change. But on the other hand, my
tables are much smaller than yours and contain mostly static data.
Regards
hmw
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: working with tables can be quite painful...
2016-09-19 8:33 ` Eric S Fraga
2016-09-19 12:58 ` Michael Welle
[not found] ` <df2f3c4ad6a34890ad82cd1d0d557003@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-22 19:52 ` Nicolas Goaziou
2 siblings, 0 replies; 12+ messages in thread
From: Nicolas Goaziou @ 2016-09-22 19:52 UTC (permalink / raw)
To: Emacs Org mode mailing list
Hello,
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> I cannot send the actual table I am working on as it relates to grades
> for students. I will try to create a similar table with random
> entries.
You could also try the imperfect (make sure to eyeball the result):
(defun scramble-contents ()
(interactive)
(let ((tree (org-element-parse-buffer)))
(org-element-map tree '(code comment comment-block example-block fixed-width
keyword link node-property plain-text verbatim)
(lambda (obj)
(cl-case (org-element-type obj)
((code comment comment-block example-block fixed-width keyword
node-property verbatim)
(let ((value (org-element-property :value obj)))
(org-element-put-property
obj :value (replace-regexp-in-string "[[:alnum:]]" "x" value))))
(link
(unless (string= (org-element-property :type obj) "radio")
(org-element-put-property obj :raw-link "http://orgmode.org")))
(plain-text
(org-element-set-element
obj (replace-regexp-in-string "[[:alnum:]]" "x" obj)))))
nil nil nil t)
(let ((buffer (get-buffer-create "*Scrambled text*")))
(with-current-buffer buffer
(insert (org-element-interpret-data tree))
(goto-char (point-min)))
(switch-to-buffer buffer))))
Regards,
--
Nicolas Goaziou
^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <61c781eee3b64613a40a39b95bc54e63@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>]
* Re: working with tables can be quite painful...
[not found] ` <61c781eee3b64613a40a39b95bc54e63@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-19 8:36 ` Eric S Fraga
0 siblings, 0 replies; 12+ messages in thread
From: Eric S Fraga @ 2016-09-19 8:36 UTC (permalink / raw)
To: Michael Brand; +Cc: Emacs Org mode mailing list
On Saturday, 17 Sep 2016 at 15:37, Michael Brand wrote:
> Hi Eric
>
> Question, out of curiosity: Is there a difference when you delete all lines
> above and below the table, with and without adding a headline above?
Hi Michael,
Maybe. I know that org's performance improves when I narrow the buffer
to just the section I am working on, when I work on long documents so
the lines above and below the table may affect things.
> One of my tables fluctuates around 150 rows and around 20 to 40
> columns, overall a few hundred characters wide (columns with some
> history window for each row). Almost all fields are just text although
> I think this doesn't matter at all. The table starts at about line 50
> and is followed by about 7600 lines. Since years I manipulate it at
> least every few days without any slowness, often on much weaker
> hardware than yours.
How long do you spend manipulating the table at any one time? I have an
impression that performance degrades over time in a single session. But
I have no evidence for this. It's just an impression.
thanks,
eric
--
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: working with tables can be quite painful...
2016-09-16 14:53 working with tables can be quite painful Eric S Fraga
` (3 preceding siblings ...)
[not found] ` <61c781eee3b64613a40a39b95bc54e63@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-20 8:54 ` Jacob Nielsen
[not found] ` <b05decb709ba4f64932ba34835b08056@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
5 siblings, 0 replies; 12+ messages in thread
From: Jacob Nielsen @ 2016-09-20 8:54 UTC (permalink / raw)
To: emacs-orgmode
Eric S Fraga <e.fraga@ucl.ac.uk> writes:
Hi, I've had these lines in my org files for a long time. Perhaps they
help ?
# -*- cache-long-scans: nil; -*-
# This makes forward-line much faster and thus org-goto-line
# and thus org-table-sum (C-c +)
Best regards,
Jacob
> Hello,
>
> I am working with a table. It has approximately 130 rows and 20 columns
> so it's not huge but also not small. Three columns are text but the
> rest are all numbers with some degree of sparsity. Instrumenting org
> while working on this table, manipulating the entries in just one row, I
> get the following:
>
> | Function | calls | elapsed time | average time |
> |--------------------------+-------+--------------+--------------|
> | org-cycle | 20 | 41.550963140 | 2.0775481570 |
> | org-table-next-field | 20 | 41.544266727 | 2.0772133363 |
> | org-table-align | 5 | 41.470595702 | 8.2941191404 |
> | org-mode-flyspell-verify | 52 | 1.0647362189 | 0.0204756965 |
> | org-do-latex-and-related | 21 | 0.6656267140 | 0.0316965101 |
> | org-element-at-point | 125 | 0.6356939890 | 0.0050855519 |
> | org-element--parse-to | 125 | 0.6086256940 | 0.0048690055 |
> | org-element--cache-put | 1399 | 0.4963533770 | 0.0003547915 |
>
> From this, it would seem that the table align function is killing the
> performance. 8 seconds per call? On an 8 core Intel(R) Core(TM)
> i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
> slowness!
>
> This is with a not quite up to date org. I'm avoiding upgrading as I am
> preparing material for teaching which starts soon and I don't want to
> run into issues due to changes in org... so I apologise if things have
> changed recently. This performance issue has existed for quite some time
> now, however.
>
> Any suggestions on speeding things up?
>
> Thanks,
> eric
^ permalink raw reply [flat|nested] 12+ messages in thread[parent not found: <b05decb709ba4f64932ba34835b08056@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>]
* Re: working with tables can be quite painful...
[not found] ` <b05decb709ba4f64932ba34835b08056@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
@ 2016-09-20 9:03 ` Eric S Fraga
0 siblings, 0 replies; 12+ messages in thread
From: Eric S Fraga @ 2016-09-20 9:03 UTC (permalink / raw)
To: Jacob Nielsen; +Cc: emacs-orgmode@gnu.org
On Tuesday, 20 Sep 2016 at 08:54, Jacob Nielsen wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
> Hi, I've had these lines in my org files for a long time. Perhaps they
> help ?
>
> # -*- cache-long-scans: nil; -*-
> # This makes forward-line much faster and thus org-goto-line
> # and thus org-table-sum (C-c +)
Interesting. Reading the documentation for this variable notes that
having this set to t is most useful for buffers with very long
lines. As I use visual-line-mode, most of my lines are whole paragraphs
so frequently longer than the 500 characters that the documentation
mentions as "long lines". So it would seem that I should keep the
default setting of t... but it may be worthwhile playing around with
this variable.
There are obviously quite a few interacting elements which can affect
the performance of emacs.
Thanks!
--
: Eric S Fraga (0xFFFCF67D), Emacs 25.1.50.1, Org release_8.3.6-1149-g582233
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-09-22 19:53 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-16 14:53 working with tables can be quite painful Eric S Fraga
2016-09-17 8:48 ` Nicolas Goaziou
2016-09-17 15:22 ` Adam Porter
2016-09-17 15:37 ` Michael Brand
[not found] ` <9ae6591e2754413885a96e19c2455472@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
2016-09-19 8:33 ` Eric S Fraga
2016-09-19 12:58 ` Michael Welle
[not found] ` <df2f3c4ad6a34890ad82cd1d0d557003@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
2016-09-19 14:23 ` Eric S Fraga
2016-09-19 17:54 ` Michael Welle
2016-09-22 19:52 ` Nicolas Goaziou
[not found] ` <61c781eee3b64613a40a39b95bc54e63@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
2016-09-19 8:36 ` Eric S Fraga
2016-09-20 8:54 ` Jacob Nielsen
[not found] ` <b05decb709ba4f64932ba34835b08056@HE1PR01MB1898.eurprd01.prod.exchangelabs.com>
2016-09-20 9:03 ` Eric S Fraga
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).