From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id sFhMIwIevmH8gQAAgWs5BA (envelope-from ) for ; Sat, 18 Dec 2021 18:44:34 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id eK0LHwIevmGlOgAAB5/wlQ (envelope-from ) for ; Sat, 18 Dec 2021 17:44:34 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3B9DE8AC6 for ; Sat, 18 Dec 2021 18:44:34 +0100 (CET) Received: from localhost ([::1]:33344 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mydl3-0004DT-9U for larch@yhetil.org; Sat, 18 Dec 2021 12:44:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44926) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mydkc-0004DG-8N for emacs-orgmode@gnu.org; Sat, 18 Dec 2021 12:44:06 -0500 Received: from [2a00:1450:4864:20::433] (port=38443 helo=mail-wr1-x433.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mydka-0000ey-BM for emacs-orgmode@gnu.org; Sat, 18 Dec 2021 12:44:05 -0500 Received: by mail-wr1-x433.google.com with SMTP id e5so10518897wrc.5 for ; Sat, 18 Dec 2021 09:44:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=q2P2zHR2HvXuxLp1oiDcFYzWng6Ftxn8ZtvHK7kOnzg=; b=T2FtS0Xm4sx+HFgtfn7HzFaedCAFUSVl/EIGcnTmP8mIg87yUU6vXRP+IUx42fyYAI 9ERsXYRSFj/Bf7Gr/EBFRec8xWttT0+J1s+UX3+cyaX9i9r3zZxQJEaaHpIgz26EE+uv rAtjZT14/vbBzW9XxjNmOQB7GRnT9YE6CCZFrz+Rvj7w28iNld0yk7b54qjdXKwNPV8L 064MTjmuyt0+cusRD0qRJ69TlD5pdGJoCaeVG82AEMEx0XZa/cOekRQ77Yjxd7rmfqQQ IWeMa9ZLlggbTR95PEsdV5E+fU1wbJI5igJMEbV9NcGWpYBNLxM2X5iPVypeuXhE1bn+ TSBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=q2P2zHR2HvXuxLp1oiDcFYzWng6Ftxn8ZtvHK7kOnzg=; b=Z9y4KsCKpK1vMs+SofGGfgB/qIn+4TEuMDmRraoLm3nx4BlSN67AvJCDhcSst6jeB9 mYOG/kqHeos90zPwqLxxJgoA9x7CKk58Ag2hoM2g0Vpsuecy7W/6eQeXNIIhvpszRvrQ 2YUcbhlwn3/JWU6JpCaJcdEQCxwm5MUBnc7SUVl0SONW293aI5FNscTnqGS4MokHTQFx 2N1LFUrhJlVBlD9bT6F+dVIhTf6E9Pcp4ByRbKo2DMKfYQCRT7xfKPVFfMrHz5EcZdGy KussGge/zPXb742jwbvXf/731cU6DHlTG7zZtAMCba3m7vyduBtbAi0XI166x70Bq/Vt LEJA== X-Gm-Message-State: AOAM533thFuE2y6/xTOJG8OMa5Mz+6/Tv7j/ArfQv3Kup0VanNl1cgQ4 MVwLyhs29YSg0jqlvgPWAZPRTFGUn23q4AJbDQM= X-Google-Smtp-Source: ABdhPJzanVXozaVa32YuAJ00/Ql2p0VE7FZJjSJ+a5KcNctoOpu9IUny08seLe32cnjiUdcGgVl4cgaFPDhAtauMytM= X-Received: by 2002:a5d:68ce:: with SMTP id p14mr7100689wrw.116.1639849442387; Sat, 18 Dec 2021 09:44:02 -0800 (PST) MIME-Version: 1.0 References: <87y25ztcgr.fsf@localhost> <87bl1hdvuj.fsf@localhost> <87mtl037ez.fsf@localhost> <878rwiil7c.fsf@localhost> In-Reply-To: <878rwiil7c.fsf@localhost> From: Kaushal Modi Date: Sat, 18 Dec 2021 12:43:26 -0500 Message-ID: Subject: Re: [BUG] org-element--cache gives "Unregistered buffer modifications" warning [9.5 (9.5-g859984 @ /home/john/.emacs.d/straight/build/org/)] To: Ihor Radchenko Content-Type: multipart/alternative; boundary="0000000000007fb39f05d36f33e4" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::433 (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=kaushal.modi@gmail.com; helo=mail-wr1-x433.google.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-org list , John Mathena Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639849474; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=q2P2zHR2HvXuxLp1oiDcFYzWng6Ftxn8ZtvHK7kOnzg=; b=rP43E5VyH6jpNCpQHCDhYBhLxcytgGRbBF7g4k5L3qPKJ9IOi1sjCFBdzbAd7IN0OCXxPy sIGu2e5C+4Z7VqW7KnEzsEN5D6hW9EMk1dcjkm0q07o022q2BPtAZKsMrtUDYDbseJ15/2 +MOwPAHqV6g4Jr1iEgWqLI9Lxx/GTQW6+A0U8lLMuj6kyqAnedWLDVZoQ5DVo/wN5ZTiaS 2LFFdaNd5iF+H8CeX/q2tYnVBYa01Nzv7yd8JhtG04D3NEeDcRORe/zjPSlzO9oOZbJyeC obacYhIJxKXvFJEZl/+A4V5xo/ZUQRO7ibvzVSZ2Zp1P2ypxPpnw2rgLYXIRxA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639849474; a=rsa-sha256; cv=none; b=GY+xSs7l97wLd+iXPqWfuIdUCj4nhEwz3Qztc7DWbJJcjauDbho0T+MknnSShohwmrEJEu jTXdLg9lcrKmM0O32f/D44vIdeqjP6q80Kp5NggIbIBQMLEOWEsqBX1qvx/xdJs60ZYPyl Lf8HU67vu+pgwPJ4eT0n/loKv8E4H1yDtjcJHsDOrsCIchf2AmkF8me0+Dhm34PM2DUwf+ gj8hFNADoQtDCbywK3ATZiniZ28AciZ2LHIyeIL9GFNRgdAFWZGESAyk9JOW4HyKfQ5aMy yTz1Q1rfGBtQy9osbLhl3Lj6gZC+UGWQAmhU6V+K3DVRpAs9+gN2bSfYeoijRQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=T2FtS0Xm; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.21 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=T2FtS0Xm; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3B9DE8AC6 X-Spam-Score: -2.21 X-Migadu-Scanner: scn0.migadu.com X-TUID: XxaSy3isMvlE --0000000000007fb39f05d36f33e4 Content-Type: text/plain; charset="UTF-8" On Fri, Dec 17, 2021 at 11:34 PM Ihor Radchenko wrote: > > There's a setup included in there; I forgot about that. If you don't mind > > trying it again, this would download everything needed: > > Thanks! I was able to follow your steps this time. > I really appreciate making this extra effort to help me out. Thanks! > > Sorry, but I am getting similar backtrace in another function now: > > Sigh... Should be fixed now. I did not expect that arguments to cache > functions could ever be markers. > > After the fix, I was able to see the warning you were talking about in > the earlier email. Your test case revealed an important omission in the > cache logic when a huge number of edits are stacked together into > asynchronous cache queue. Thanks you for providing the excellent > reproducer! > No problem! The ox-hugo tests end up indirectly testing a lot of Org features :D > On latest main, your example file is exported without any warnings. > I just verified that. Also, I did not even need to add `(org-element-cache-reset)' in `org-hugo--get-pre-processed-buffer' after the `(insert ..)' in the temp buffer. Is that expected? Also, I noticed that now the exports are much more snappier than before! Also, I noticed that the large number of edits makes the cache use a lot > of memory adding load onto the Emacs garbage collector. May I know what > exactly ox-hugo does that triggers that many edits? Exporting your > example files took approximately 30sec on my system (both with and > without cache). I believe that the speed can be improved if cache is > used more optimally. > That command exports 81 subtrees (as of now) from ox-hugo-manual.org to independent Markdown files. That said, I think that `org-hugo--get-pre-processed-buffer' [1] is the performance bottleneck. It was added so that Org subtree exports can work even if one subtree has an internal link to a different subtree in the same file. The pre processing logic slurps up the AST of the whole Org buffer, does org-element-copy on all the link org-elements to point to a place outside the exported subtree, fixes that link so that the export works, and copies the modified link element back on the original element. And then the whole Org temp buffer is overwritten with the modified AST. And then.. the subtree export happens from there. This is the only way possible as far as I know to support cross-subtree links. Credit for this implementation goes to Folkert van der Beek[2]. Let me know if you have any suggestions to improve the performance. I think that the org-element-copy and writing the huge AST to temp Org buffer is keeping the GC busy (the ox-hugo-manual.org is about 3700 lines right now). [1]: https://github.com/kaushalmodi/ox-hugo/blob/1ea466d33f76eaf2567593fa612b82c22ef86d3d/ox-hugo.el#L4033-L4151 [2]: https://github.com/kaushalmodi/ox-hugo/pull/280 --0000000000007fb39f05d36f33e4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, Dec 17, 2021 at 11:34 PM Ihor Radchenko <yantar92@gmail.com> wrote:
> There's a setup included i= n there; I forgot about that. If you don't mind
> trying it again, this would download everything needed:

Thanks! I was able to follow your steps this time.
I really appreciate making this extra effort to help me out. Th= anks!
=C2=A0
> Sorry, but I am getting similar backtrace in another function now:

Sigh... Should be fixed now. I did not expect that arguments to cache
functions could ever be markers.

After the fix, I was able to see the warning you were talking about in
the earlier email. Your test case revealed an important omission in the
cache logic when a huge number of edits are stacked together into
asynchronous cache queue. Thanks you for providing the excellent
reproducer!

No problem! The ox-hugo tes= ts end up indirectly testing a lot of Org features :D
=C2=A0=
On latest main, your example file is exported without any warnings.

I just verified that. Also, I did not even nee= d to add=20 `(org-element-cache-reset)' in=20 `org-hugo--get-pre-processed-buffer' after the `(insert ..)' in the temp buffer.

Is that expected?

Also, I noticed that now the = exports are much more snappier than before!

Also, I noticed that the large number of edits makes the cache use a lot of memory adding load onto the Emacs garbage collector. May I know what
exactly ox-hugo does that triggers that many edits? Exporting your
example files took approximately 30sec on my system (both with and
without cache). I believe that the speed can be improved if cache is
used more optimally.

That command expor= ts 81 subtrees (as of now) from ox-hu= go-manual.org to independent Markdown files.=C2=A0

<= /div>
That said, I think that `org-hugo--get= -pre-processed-buffer' [1] is the performance bottleneck. It was= added so that Org subtree exports can work even if one subtree has an inte= rnal link to a different subtree in the same file.

The pre processing logic slurps up the AST of the whole Org buffer, does o= rg-element-copy on all the link org-elements to point to a place outside th= e exported subtree, fixes that link so that the export works, and copies th= e modified link element back on the original element. And then the whole Or= g temp buffer is overwritten with the modified AST.
And then.. th= e subtree export happens from there.

This is the o= nly way possible as far as I know to support cross-subtree links. Credit fo= r this implementation goes to Folkert van der Beek[2].

Let me know if you have any suggestions to improve the performance= . I think that the org-element-copy and writing the huge AST to temp Org bu= ffer is keeping the GC busy (the ox-h= ugo-manual.org is about 3700 lines right now).

=

--0000000000007fb39f05d36f33e4--