From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tim Baumgard Subject: Bug: Org Table Performance Issue/Regression Date: Sun, 10 Jun 2018 14:04:08 -0500 Message-ID: Mime-Version: 1.0 (Mac OS X Mail 11.4 \(3445.8.2\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:36648) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fS5dS-0005ts-Nz for emacs-orgmode@gnu.org; Sun, 10 Jun 2018 15:04:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fS5dM-000888-Vv for emacs-orgmode@gnu.org; Sun, 10 Jun 2018 15:04:16 -0400 Received: from mr90p54im-ztdg04142001.me.com ([17.120.66.146]:61341) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fS5dM-000878-PL for emacs-orgmode@gnu.org; Sun, 10 Jun 2018 15:04:12 -0400 Received: from process-dkim-sign-daemon.mr90p54im-ztdg04142001.me.com by mr90p54im-ztdg04142001.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0PA400400FCK3500@mr90p54im-ztdg04142001.me.com> for emacs-orgmode@gnu.org; Sun, 10 Jun 2018 19:04:10 +0000 (GMT) Received: from icloud.com ([127.0.0.1]) by mr90p54im-ztdg04142001.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0PA4004DKFMW2X00@mr90p54im-ztdg04142001.me.com> for emacs-orgmode@gnu.org; Sun, 10 Jun 2018 19:04:10 +0000 (GMT) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: emacs-orgmode@gnu.org I recently updated Emacs from 25.3 (Org mode v. 8.2.10) to 26.1 (Org = mode v. 9.1.9) and noticed a performance issue/regression in one of my Org = files that contains a table that currently has nine columns, around 1,200 rows, and = uses formulas. In 25.3, I could create a new row or use org-table-copy-down = without issues. In 26.1, there=E2=80=99s a multi-second delay each time I try to = do either of these things. As mentioned below, this issue also occurs in the latest = commits to both Emacs and Org mode. I did some profiling and found that garbage-collect was being called = quite a bit. I set gc-cons-threshold in both versions to a large value to = effectively disable garbage collection and noticed two things: 1) it largely fixed = the performance issues=E2=80=94though performance still wasn=E2=80=99t as = good as it was in 25.3=E2=80=94and 2) Emacs/Org mode was using about four times as much memory as it did = after creating a new row and doing calling org-table-copy-down three times. Specifically, Emacs 25.3 was using a little under 200 MB and = Emacs 26.3 was using a around 800 MB. I also noticed that Org mode is recalculating the table formulas more = often than in past versions. It used to be that it would only recalculate the = formulas for the whole table when I called org-table-recalculate, but after testing = some things and then diff=E2=80=99ing the files, I noticed that other rows = were being updated based on the table formulas I=E2=80=99m using. I=E2=80=99m unsure if = this is related. For what it=E2=80=99s worth, using the Org mode files from Emacs 25.3 in = 26.1 doesn=E2=80=99t display the performance issue, but there are other issues that make it a = poor workaround. What I=E2=80=99m currently doing to work around this for my = use case is, only for this file, setting gc-cons-threshold to a large value and = adding a hook to after-save-hook to call garbage-collect since calling it once when = saving doesn=E2=80=99t take too long. I=E2=80=99ve inlined a test file and a test init.el file below. The test = file includes a reduced version of the table I=E2=80=99m currently using, including the = table TBLFM line. The init.el file includes the functions required for my table = formulas and how I was using gc-cons-threshold and my workaround. I was still seeing = the issue even without using the formulas, but I thought I'd include them in = case it's helpful. Some notes: - All of my testing was done using emacs -Q - I=E2=80=99m running Emacs on macOS compiled from the sources available = on gnu.org.=20 - I also tested everything using the latest commits of Emacs (efa750e68466fb1bab03028701350a745e0504b6) and Org mode (842002f9f7e3246cc285cc9bca1adb715120438d) from the = repositories. - Emacs: GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin17.6.0, NS appkit-1561.40 Version 10.13.5 (Build 17F77)) of 2018-06-10 - Package: Org mode version 9.1.13 (release_9.1.13-791-g842002) test.org =3D=3D=3D=3D=3D=3D | | A | B | C | D | E = | F | G | H | = |---+---------+-----------+--------+---------------------------+----------= -----------------+----------+-----+----------------------------| | # | ABCDEFG | HIJKLMNOP | QRSTUV | [2018-06-10 Sun 00:00:00] | = [2018-06-10 Sun 02:00:00] | 02:00:00 | 2.0 | ABCDEFGHIJKLMNOPQRSTUVWXYZ = | #+TBLFM: $7=3D'(time-stamp-difference $5 = $6)::$8=3D'(time-stamp-difference-hours $5 $6) init.el =3D=3D=3D=3D (defun time-stamp-difference (earlier-date later-date) "Get the difference from two time stamps. This only works when the = difference is less than 24 hours." (let* ((time-difference (round (- (time-to-seconds = (date-to-time later-date)) = (time-to-seconds (date-to-time earlier-date))))) (absolute-time-difference (max = time-difference 0))) (format-time-string "%T" (encode-time = absolute-time-difference 0 0 1 1 1)))) (defun time-stamp-difference-hours (earlier-date later-date) "Get the difference from two time stamps in the number of hours." (let* ((seconds (- (time-to-seconds (date-to-time = later-date)) (time-to-seconds = (date-to-time earlier-date)))) (absolute-seconds (max seconds 0)) (hours (/ (/ absolute-seconds 60) = 60.0))) (/ (round (* hours 1000.0)) 1000.0))) ;; This sets the garbage-collection threshold to a large value, = effectively ;; disabling it for testing purposes. ;; (setq-default gc-cons-threshold 80000000000) ;; (add-hook 'after-save-hook 'garbage-collect)