On Fri, Dec 17, 2021 at 11:34 PM Ihor Radchenko <yantar92@gmail.com> wrote:

> > There's a setup included in there; I forgot about that. If you don't mind
> > trying it again, this would download everything needed:
>
> Thanks! I was able to follow your steps this time.
>

I really appreciate making this extra effort to help me out. Thanks!


> > Sorry, but I am getting similar backtrace in another function now:
>
> Sigh... Should be fixed now. I did not expect that arguments to cache
> functions could ever be markers.
>
> After the fix, I was able to see the warning you were talking about in
> the earlier email. Your test case revealed an important omission in the
> cache logic when a huge number of edits are stacked together into
> asynchronous cache queue. Thanks you for providing the excellent
> reproducer!
>

No problem! The ox-hugo tests end up indirectly testing a lot of Org
features :D


> On latest main, your example file is exported without any warnings.
>

I just verified that. Also, I did not even need to add
`(org-element-cache-reset)' in `org-hugo--get-pre-processed-buffer' after
the `(insert ..)' in the temp buffer.

Is that expected?

Also, I noticed that now the exports are much more snappier than before!

Also, I noticed that the large number of edits makes the cache use a lot
> of memory adding load onto the Emacs garbage collector. May I know what
> exactly ox-hugo does that triggers that many edits? Exporting your
> example files took approximately 30sec on my system (both with and
> without cache). I believe that the speed can be improved if cache is
> used more optimally.
>

That command exports 81 subtrees (as of now) from ox-hugo-manual.org to
independent Markdown files.

That said, I think that `org-hugo--get-pre-processed-buffer' [1] is the
performance bottleneck. It was added so that Org subtree exports can work
even if one subtree has an internal link to a different subtree in the same
file.

The pre processing logic slurps up the AST of the whole Org buffer, does
org-element-copy on all the link org-elements to point to a place outside
the exported subtree, fixes that link so that the export works, and copies
the modified link element back on the original element. And then the whole
Org temp buffer is overwritten with the modified AST.
And then.. the subtree export happens from there.

This is the only way possible as far as I know to support cross-subtree
links. Credit for this implementation goes to Folkert van der Beek[2].

Let me know if you have any suggestions to improve the performance. I think
that the org-element-copy and writing the huge AST to temp Org buffer is
keeping the GC busy (the ox-hugo-manual.org is about 3700 lines right now).

[1]:
https://github.com/kaushalmodi/ox-hugo/blob/1ea466d33f76eaf2567593fa612b82c22ef86d3d/ox-hugo.el#L4033-L4151

[2]: https://github.com/kaushalmodi/ox-hugo/pull/280