From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id +IujMm/c72HU1AAAgWs5BA (envelope-from ) for ; Tue, 25 Jan 2022 12:18:07 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id iJdHL2/c72F71gAAauVa8A (envelope-from ) for ; Tue, 25 Jan 2022 12:18:07 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 09EF32D6AD for ; Tue, 25 Jan 2022 12:18:07 +0100 (CET) Received: from localhost ([::1]:43122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nCJpu-0000t8-8A for larch@yhetil.org; Tue, 25 Jan 2022 06:18:06 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38420) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nCEEc-0004mV-Hw for emacs-orgmode@gnu.org; Tue, 25 Jan 2022 00:19:14 -0500 Received: from wilbur.contactoffice.com ([212.3.242.68]:53994) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nCEEY-0000p6-Ft for emacs-orgmode@gnu.org; Tue, 25 Jan 2022 00:19:13 -0500 Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by wilbur.contactoffice.com (Postfix) with ESMTP id 81241A0A for ; Tue, 25 Jan 2022 06:19:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1643087946; s=20210208-e7xh; d=mailfence.com; i=pareto.optimal@mailfence.com; h=Date:Message-ID:MIME-Version:Content-Type:From; l=19941; bh=W108OUG48F1K5MCthOvXWwPfENaNqQgwNgiDrsiqHjo=; b=u+X8BzFrdtsMqOR41TPsGxLtVt+XCSYNm+4UAA9rcbDEQXBOePW3xzv/rTUHrpw4 BBEcOZ1uVAdxMhVbv0vFpmSf+oyTmcVMNRWiGIa6TCvGaWkok5Qca7E6gzLZ3MDu1A/ +qDL5DWqPFUm1fIhin0SuWEGA2S5hUbyB0U59ivFkrEWOQeK2vx+P1cEmqsIFT5xbp2 0PDxwAmWniBx9FlmMxXN95duOTVqdLUGvQIcxufNF4tPDQsItrH7cWObxdAx5EXdhbR 09jlSc/ZGgMr4gzoL4leuEWXFxjr7sjdmFp6vbS3CRLzcKMKDiIc/nv4DdLV8P4zJ5B uOblNhzQgw== Date: Tue, 25 Jan 2022 06:19:04 +0100 (CET) Message-ID: <147178804.982080.1643087944384@ichabod.co-bxl> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_982077_1635451825.1643087944381" Subject: Poor org-babel-tangle-file performance with more than 100 trivial noweb-references From: pareto optimal To: emacs-orgmode@gnu.org X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:311504211 Received-SPF: pass client-ip=212.3.242.68; envelope-from=pareto.optimal@mailfence.com; helo=wilbur.contactoffice.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 25 Jan 2022 06:16:47 -0500 X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1643109487; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=W108OUG48F1K5MCthOvXWwPfENaNqQgwNgiDrsiqHjo=; b=Cilkv2p0YHAf0X/qDzqSiNWsz0qcr/OD21jCG1ElukhXJt7Cbw/lf75jsoID7FEk/0HiYo nnF/6QhV9owmLk+UEs8lJ9rQ1jL9MzmEdmuLY+A4493NS4l4yFd2nGPO9grf+hTRZ2aAz/ ndgdkQJIUtsZSrG2M4u4jyDY0Io6+VZtTVMgyJtwnvMZkkdWWrnVuqZKUYDTs0A14Q2BF3 62ZDkTDahuSdlIgtUhDWarjDwYE7BtVdH2ImKHNxhnCb/DnXu+GshZY6v1bDrZudq/mPjt J8U572eKwRkTiQeWFme2C91puIN4N/AdvrO4JwnZlTe6meCgw1YZtlFEwLe7QQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1643109487; a=rsa-sha256; cv=none; b=cmqpBWHQGcQFyd7Y6mPd9gqFhjICoAR67viZJTGjb/2NynBK1Kb8nGSAb+M0Da4IJP0irp /MUK9FHFqynOuNFkoheq/P+xijSfq/IOv4sHlMpADko25zTEKp7e+nSyiRNaeAo/yaAJZu vd3D8vrp7sf2hkadxNZkA52yvoGL542ffASiJN3dzSIRCd1bU9rX5wYBuhCVtwjmevKGuJ vZ4OM8jiDdYlESwbxmsCpULMH0L0EXIJHCVrFMTUlCL8Vhmh//iPgse/0Wow8XdPul506u P/DZYMprvCw3kORY6elgdS2LRRFO5YN0SZyPeRp/hssbZgUbfiZRbdIzvBaafw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=u+X8BzFr; dmarc=pass (policy=quarantine) header.from=mailfence.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -9.32 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=u+X8BzFr; dmarc=pass (policy=quarantine) header.from=mailfence.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 09EF32D6AD X-Spam-Score: -9.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: E57FqRnI9m4D ------=_Part_982077_1635451825.1643087944381 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Using =3Demacs -Q=3D to tangle org files with more than over 100 noweb-refs= gets slow fast. Given this org code for N=3D2: #+begin_src org :tangle 2.org =C2=A0 ,#+begin_src elisp :noweb yes :tangle 2.el =C2=A0 <> =C2=A0 ,#+end_src =C2=A0 ,#+begin_src elisp :noweb-ref some-reference :tangle no =C2=A0 ;; comment =C2=A0 ,#+end_src =C2=A0 ,#+begin_src elisp :noweb-ref some-reference :tangle no =C2=A0 ;; comment =C2=A0 ,#+end_src #+end_src and this macro for benchmarking: #+begin_src emacs-lisp (cl-defmacro bench (&optional (times 100000) &rest body) =C2=A0 (declare (indent defun)) =C2=A0 `(progn =C2=A0=C2=A0=C2=A0=C2=A0 (garbage-collect) =C2=A0=C2=A0=C2=A0=C2=A0 (list '("Total runtime" "# of GCs" "Total GC runti= me") =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'hline =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (benchmark-run= -compiled ,times =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (p= rogn =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 ,@body))))) #+end_src and the org-babel-tangle-file call with gc-cons-threshold at most-positive-= fixnum: #+begin_src emacs-lisp =C2=A0 (let ((gc-cons-threshold most-positive-fixnum)) =C2=A0=C2=A0=C2=A0 (bench (org-babel-tangle-file "2.org"))) #+end_src Using Gcc Emacs 28.0.91 (which I typicall use) I get these results: | N blocks | runtime | # of GCs | |----------+---------+----------| |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 10 |=C2=A0=C2=A0 0.027 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20 |=C2=A0=C2=A0 0.049 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40 |=C2=A0=C2=A0 0.121 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 80 |=C2=A0=C2=A0 0.391 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 160 |=C2=A0=C2=A0 1.426 |=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 320 |=C2=A0=C2=A0 6.765 |=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 640 |=C2=A0 23.972 |=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0 | so roughly it scales like: #+begin_example 0.8x (10-20) 2.5x=C2=A0 (20-40) 3x=C2=A0 (40-80) 3.5x=C2=A0 (160-320) 4x=C2=A0 (320-640) #+end_example Though I'm not sure how much that can tell us... my guess is noweb-ref's do= n't use a dictionary with =3DO(1)=3D lookups? Out of curiousity I also benchmarked emacs 27.2 which came with org 9.4.4 a= nd it gave roughly the same results: * emacs -Q (emacs 27.2/org-9.4.4) ** =3Dgc-cons-threshold=3D most-positive-fixnum | N blocks | runtime | # of GCs | |----------+---------+----------| |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 10 |=C2=A0=C2=A0 0.031 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20 |=C2=A0=C2=A0 0.054 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40 |=C2=A0=C2=A0 0.142 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 80 |=C2=A0=C2=A0 0.445 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 160 |=C2=A0=C2=A0 1.641 |=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 320 |=C2=A0=C2=A0 6.478 |=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 | |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 640 |=C2=A0 25.729 |=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0 | I have the full benchmarking experiment here: https://www.paretooptimal.dev= /poor-org-babel-performance-with-lots-of-noweb-references/ Emacs=C2=A0 : GNU Emacs 28.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version= 3.24.30, cairo version 1.16.0) Package: Org mode version 9.5.2 (9.5.2-gfbff08 @ /nix/store/3ff9visj5fl1a3d= 4krpv9wgmd078f5l1-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.5.2/= ) current state: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D (setq =C2=A0org-link-elisp-confirm-function 'yes-or-no-p =C2=A0org-bibtex-headline-format-function #[257 "\300=01\236A\207" [:title]= 3 "\n\n(fn ENTRY)"] =C2=A0org-export-before-parsing-hook '(org-attach-expand-links) =C2=A0org-archive-hook '(org-attach-archive-delete-maybe) =C2=A0org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-draw= ers org-cycle-show-empty-lines =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 org-optimize-window-after-visi= bility-change) =C2=A0org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook change-majo= r-mode-hook org-show-all append local] 5] =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0#[0 "\300\301\302\303\304$\207" =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0 [add-hook change-major-m= ode-hook org-babel-show-result-all append local] 5] =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0org-babel-result-hide-spec org-= babel-hide-all-hashes) =C2=A0org-confirm-shell-link-function 'yes-or-no-p =C2=A0outline-isearch-open-invisible-function 'outline-isearch-open-invisib= le =C2=A0org-agenda-before-write-hook '(org-agenda-add-entry-text) =C2=A0org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-= configure-edit-buffer) =C2=A0org-confirm-elisp-link-function 'yes-or-no-p =C2=A0org-speed-command-hook '(org-speed-command-activate org-babel-speed-c= ommand-activate) =C2=A0org-tab-first-hook '(org-babel-hide-result-toggle-maybe org-babel-hea= der-arg-expand) =C2=A0org-link-shell-confirm-function 'yes-or-no-p =C2=A0org-babel-pre-tangle-hook '(save-buffer) =C2=A0org-agenda-loop-over-headlines-in-active-region nil =C2=A0org-occur-hook '(org-first-headline-recenter) =C2=A0org-metadown-hook '(org-babel-pop-to-session-maybe) =C2=A0org-link-parameters '(("attachment" :follow org-attach-follow :comple= te org-attach-complete-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("id" :follow org-id-open) ("eww" :follow org-eww-open :store org-eww-store= -link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("rmail" :follow org-rmail-open :store org-rmail-store-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("mhe" :follow org-mhe-open :store org-mhe-store-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("irc" :follow org-irc-visit :store org-irc-store-link :export org-irc-expo= rt) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("info" :follow org-info-open :export org-info-export :store org-info-store= -link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("gnus" :follow org-gnus-open :store org-gnus-store-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("docview" :follow org-docview-open :export org-docview-export :store =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 org-docview-store-= link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("bibtex" :follow org-bibtex-open :store org-bibtex-store-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("bbdb" :follow org-bbdb-open :export org-bbdb-export :complete org-bbdb-co= mplete-link =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 :store org-bbdb-st= ore-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("w3m" :store org-w3m-store-link) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("doi" :follow org-link-doi-open :export org-link-doi-export) ("file+sys") = ("file+emacs") =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("shell" :follow org-link--open-shell) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("news" :follow =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #[514 "\301\300\30= 2=04Q=02\"\207" ["news" browse-url ":"] 6 "\n\n(fn URL ARG)"]) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("mailto" :follow =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #[514 "\301\300\30= 2=04Q=02\"\207" ["mailto" browse-url ":"] 6 "\n\n(fn URL ARG)"]) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("https" :follow =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #[514 "\301\300\30= 2=04Q=02\"\207" ["https" browse-url ":"] 6 "\n\n(fn URL ARG)"]) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("http" :follow =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #[514 "\301\300\30= 2=04Q=02\"\207" ["http" browse-url ":"] 6 "\n\n(fn URL ARG)"]) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("ftp" :follow =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #[514 "\301\300\30= 2=04Q=02\"\207" ["ftp" browse-url ":"] 6 "\n\n(fn URL ARG)"]) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("help" :follow org-link--open-help :store org-link--store-help) =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ("file" :complete org-link-complete-file) ("elisp" :follow org-link--open-e= lisp)) =C2=A0org-metaup-hook '(org-babel-load-in-session-maybe) =C2=A0) -- Sent with https://mailfence.com Secure and private email ------=_Part_982077_1635451825.1643087944381 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Using =3Demacs -Q=3D to t= angle org files with more than over 100 noweb-refs gets slow fast.

G= iven this org code for N=3D2:

#+begin_src org :tangle 2.org
 = ; ,#+begin_src elisp :noweb yes :tangle 2.el
  <<some-referen= ce>>
  ,#+end_src

  ,#+begin_src elisp :noweb-ref= some-reference :tangle no
  ;; comment
  ,#+end_src
  ,#+begin_src elisp :noweb-ref some-reference :tangle no
  ;= ; comment
  ,#+end_src
#+end_src

and this macro for bench= marking:

#+begin_src emacs-lisp
(cl-defmacro bench (&optional= (times 100000) &rest body)
  (declare (indent defun))
 = ; `(progn
     (garbage-collect)
  &nbs= p;  (list '("Total runtime" "# of GCs" "Total GC runtime")
 &n= bsp;         'hline
  =          (benchmark-run-compiled ,t= imes
           &= nbsp; (progn
          = ;     ,@body)))))
#+end_src

and the org-babel= -tangle-file call with gc-cons-threshold at most-positive-fixnum:

#+= begin_src emacs-lisp
  (let ((gc-cons-threshold most-positive-fixnu= m))
    (bench (org-babel-tangle-file "2.org")))
#+end= _src

Using Gcc Emacs 28.0.91 (which I typicall use) I get these resu= lts:


| N blocks | runtime | # of GCs |
|----------+---------+= ----------|
|       10 |   0.027= |        0 |
|   &nbs= p;   20 |   0.049 |      =   0 |
|       40 |   0.121 = |        0 |
|    = ;   80 |   0.391 |      &= nbsp; 0 |
|      160 |   1.426 | = ;       0 |
|     = ; 320 |   6.765 |        0 ||      640 |  23.972 |   &nb= sp;    0 |

so roughly it scales like:

#+begin_= example
0.8x (10-20)
2.5x  (20-40)
3x  (40-80)
3.5x&n= bsp; (160-320)
4x  (320-640)
#+end_example

Though I'm not= sure how much that can tell us... my guess is noweb-ref's don't use a dict= ionary with =3DO(1)=3D lookups?

Out of curiousity I also benchmarked= emacs 27.2 which came with org 9.4.4 and it gave roughly the same results:=


* emacs -Q (emacs 27.2/org-9.4.4)
** =3Dgc-cons-threshold=3D= most-positive-fixnum

| N blocks | runtime | # of GCs |
|--------= --+---------+----------|
|       10 | = ;  0.031 |        0 |
| &nb= sp;     20 |   0.054 |    = ;    0 |
|       40 | =   0.142 |        0 |
| &nbs= p;     80 |   0.445 |    =     0 |
|      160 |  = 1.641 |        0 |
|  &nbs= p;   320 |   6.478 |      = ;  0 |
|      640 |  25.729 | &n= bsp;      0 |

I have the full benchmarking = experiment here: https://www.paretooptimal.dev/poor-org-babel-performance-w= ith-lots-of-noweb-references/


Emacs  : GNU Emacs 28.0.91 (b= uild 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)Package: Org mode version 9.5.2 (9.5.2-gfbff08 @ /nix/store/3ff9visj5fl1a3= d4krpv9wgmd078f5l1-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.5.2= /)

current state:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
(= setq
 org-link-elisp-confirm-function 'yes-or-no-p
 org-bib= tex-headline-format-function #[257 "\300=01\236A\207" [:title] 3 "\n\n(fn E= NTRY)"]
 org-export-before-parsing-hook '(org-attach-expand-links)<= br> org-archive-hook '(org-attach-archive-delete-maybe)
 org-c= ycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers org-cyc= le-show-empty-lines
          org-opt= imize-window-after-visibility-change)
 org-mode-hook '(#[0 "\300\30= 1\302\303\304$\207" [add-hook change-major-mode-hook org-show-all append lo= cal] 5]
         #[0 "\300\301\302\30= 3\304$\207"
           [add-hook= change-major-mode-hook org-babel-show-result-all append local] 5]
 = ;        org-babel-result-hide-spec org-babel= -hide-all-hashes)
 org-confirm-shell-link-function 'yes-or-no-p
=  outline-isearch-open-invisible-function 'outline-isearch-open-invisib= le
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
&n= bsp;org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-co= nfigure-edit-buffer)
 org-confirm-elisp-link-function 'yes-or-no-p<= br> org-speed-command-hook '(org-speed-command-activate org-babel-spee= d-command-activate)
 org-tab-first-hook '(org-babel-hide-result-tog= gle-maybe org-babel-header-arg-expand)
 org-link-shell-confirm-func= tion 'yes-or-no-p
 org-babel-pre-tangle-hook '(save-buffer)
&nbs= p;org-agenda-loop-over-headlines-in-active-region nil
 org-occur-ho= ok '(org-first-headline-recenter)
 org-metadown-hook '(org-babel-po= p-to-session-maybe)
 org-link-parameters '(("attachment" :follow or= g-attach-follow :complete org-attach-complete-link)
    &= nbsp;          ("id" :follow org-id= -open) ("eww" :follow org-eww-open :store org-eww-store-link)
 &nbs= p;             ("rmail" := follow org-rmail-open :store org-rmail-store-link)
    &n= bsp;          ("mhe" :follow org-mh= e-open :store org-mhe-store-link)
        =        ("irc" :follow org-irc-visit :store or= g-irc-store-link :export org-irc-export)
      =          ("info" :follow org-info-open := export org-info-export :store org-info-store-link)
    &n= bsp;          ("gnus" :follow org-g= nus-open :store org-gnus-store-link)
      &nbs= p;        ("docview" :follow org-docview-open= :export org-docview-export :store
       =     org-docview-store-link)
     &nb= sp;         ("bibtex" :follow org-bibtex= -open :store org-bibtex-store-link)
       = ;        ("bbdb" :follow org-bbdb-open :expor= t org-bbdb-export :complete org-bbdb-complete-link
    &n= bsp;       :store org-bbdb-store-link)
 &n= bsp;             ("w3m" := store org-w3m-store-link)
         &n= bsp;     ("doi" :follow org-link-doi-open :export org-l= ink-doi-export) ("file+sys") ("file+emacs")
     &nb= sp;         ("shell" :follow org-link--o= pen-shell)
            = ;   ("news" :follow
        &nbs= p;   #[514 "\301\300\302=04Q=02\"\207" ["news" browse-url ":"] 6 = "\n\n(fn URL ARG)"])
          &= nbsp;    ("mailto" :follow
      = ;      #[514 "\301\300\302=04Q=02\"\207" ["mailto" brow= se-url ":"] 6 "\n\n(fn URL ARG)"])
       =        ("https" :follow
   &nbs= p;        #[514 "\301\300\302=04Q=02\"\207" [= "https" browse-url ":"] 6 "\n\n(fn URL ARG)"])
     =           ("http" :follow
 =            #[514 "\301\300\302=04Q= =02\"\207" ["http" browse-url ":"] 6 "\n\n(fn URL ARG)"])
   &= nbsp;           ("ftp" :follow=
            #[514 "\301\30= 0\302=04Q=02\"\207" ["ftp" browse-url ":"] 6 "\n\n(fn URL ARG)"])
 =               ("help= " :follow org-link--open-help :store org-link--store-help)
   =             ("file" :comp= lete org-link-complete-file) ("elisp" :follow org-link--open-elisp))
&nb= sp;org-metaup-hook '(org-babel-load-in-session-maybe)
 )

--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_982077_1635451825.1643087944381--