From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 5EaBIIDHf2B+7gAAgWs5BA (envelope-from ) for ; Wed, 21 Apr 2021 08:34:40 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 0OTpFYDHf2CRCAAAbx9fmQ (envelope-from ) for ; Wed, 21 Apr 2021 06:34:40 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 28CD91446B for ; Wed, 21 Apr 2021 08:34:38 +0200 (CEST) Received: from localhost ([::1]:45904 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZ6RY-0001ML-6s for larch@yhetil.org; Wed, 21 Apr 2021 02:34:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34266) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZ6QX-0001LJ-O8 for emacs-orgmode@gnu.org; Wed, 21 Apr 2021 02:33:37 -0400 Received: from mout01.posteo.de ([185.67.36.65]:46651) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZ6QU-0004NX-6Q for emacs-orgmode@gnu.org; Wed, 21 Apr 2021 02:33:33 -0400 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 03D5F24002B for ; Wed, 21 Apr 2021 08:33:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.eu; s=2017; t=1618986804; bh=rjoJyNgpCEVuSDK/3+m+8OLXrhz8zuDYqSVqXr1UR1Y=; h=Subject:To:Cc:From:Date:From; b=CVAEIEFpT0PrJeebpgyY4czqvwWrOYIEjpDJFVEqq/p5HIyaaNXLStDez+5Glf7+1 aqFgNJuIN/V2IBkogl8Mi+E6LwuxXD/x80CmX4ABFpv48LXXS0NZ7Qq2pRn5xlh1+Z y9NNpnJ5AxdnosugIJjMuKhkaR8JdFIvIxYoKfHmkFveoJGIyJnkzpXe+VhqGQzc0N 6XY1SMZzv8Mqtf5+TGvI0T5/fILmpc3euBMzuI6x8q3PCWB1fffb9gjJIkoxCpfRrj SzclGJ8lQTnEgUw/ByCqT28PECsnbg8aXrxOT62UBlQ5xcPRSeLHcVQE4NFJGnO5Rs w08nxx1wwYjXw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4FQ9jY5YVHz6tmX; Wed, 21 Apr 2021 08:33:21 +0200 (CEST) Subject: Re: [PATCH] ob-tangle.el: Speed up tangling To: Tom Gillespie References: <57480e77-024a-adcc-ec9a-c20b84ac762a@posteo.eu> From: =?UTF-8?Q?S=c3=a9bastien_Miquel?= Message-ID: <9dfb44d3-0c4a-6a66-0efa-c70e9434ee5e@posteo.eu> Date: Wed, 21 Apr 2021 06:33:21 +0000 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------37867A7BD3EBDE03D8B83D52" Content-Language: fr Received-SPF: pass client-ip=185.67.36.65; envelope-from=sebastien.miquel@posteo.eu; helo=mout01.posteo.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: sebastien.miquel@posteo.eu Cc: emacs-orgmode Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618986880; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Zv6pop+Z+kNeN95Cx7ia8zkNXyC839GiXjZdR0DIJ3c=; b=qtIwmilKhxcmXXwh9eh27Q9wvpVP2pTgTUqCzsdN1lDI+qOQkkrIWHedxLUgN54/bHAINZ rAWFmt4+gQesvb4JnMFc7sfn89ydHeOXqQ//KrNYbVWBOgq42fikq9aEMwA7ENKLqxC3YG h7+aVljwTLpn+iis9jQQmwe4Ox7gzz/9p0bG1c7IGzFCuta2VXmeRAUgTRCaFiZfQ24bN+ VEAtAAGF2iOpGAGeJINTwmBEaFkgITAJAABF2TeEHGVbt1Kd3D6yc/VLMxXdBt6A+0ssmr wbMQrsIUNuurSzx2M+q6jBCoHwn+A+/rYhdPoUIC1MyPpjo8SmTViCM5pzChWw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618986880; a=rsa-sha256; cv=none; b=YUGZp2K6gbBBmiEFumoJEeCC19rR480UhDbVawV23h0JddThzSritIT3HlAX7qqFBcajZe YMgXIXU8k1V1WOBUqP4tb9FkjoHB3KJz07Mq5fKcSqw8hPryCjQpcP08ryVUzt6e655bfY KEikTPEKf4oZ9CjD9/CAdYKtANrSOsO7R2t4R3TqgYZn3rNKmkzyu8d0rQrbPDETGLBt0c oxmExA511asBZnnEpPoyjTjT/X6gOb9g9ZPwfuanAcNJJIdJXsU3Lg3b6UNh0yvD1KTf1t Q4oEfYaEMTDfGdoLggnjhLrB+rZoy5rFXY7Wz86t/oybZ6ha6OLxyS9MPeu8JA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.eu header.s=2017 header.b=CVAEIEFp; dmarc=pass (policy=none) header.from=posteo.eu; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -3.14 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.eu header.s=2017 header.b=CVAEIEFp; dmarc=pass (policy=none) header.from=posteo.eu; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 28CD91446B X-Spam-Score: -3.14 X-Migadu-Scanner: scn0.migadu.com X-TUID: 2ErBPuhoAnpo This is a multi-part message in MIME format. --------------37867A7BD3EBDE03D8B83D52 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi Tom, Thank you again for your comments. Tom Gillespie writes: > I think that the location of condition-case is ok, but I wonder what > would happen if something were to fail before entering that? I think > that only a subset of the files would be tangled, but they would all > have their correct modes, so I think that that is ok. On second thought, I'm uneasy about my approach. If tangling fails, the user might miss the error message since it is quickly replaced by the tangling info. Ideally we should backup all the tangled files and restore them all if a single one fails to ensure we're back to a consistent state. I'm unsure what would be best practices here. In case of a remote tangled files, I don't know if temporary files should be remote or not, and what guarantees do emacs primitives such as ~rename-file~ offer. Although a robust tangling system that deals with errors and guarantees that the state ends up consistent would be nice to have, I'll take the failure considerations off this patch to keep it simple. It'll make a better starting point for future work at least. As is currently the case, if tangling fails, an error with be thrown, the user will certainly notice and should assume that everything is broken until another tangling succeeds. I've kept the modes improvements. Regards, -- Sébastien Miquel --------------37867A7BD3EBDE03D8B83D52 Content-Type: text/x-patch; charset=UTF-8; name="0001-ob-tangle.el-Improve-tangling.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-ob-tangle.el-Improve-tangling.patch" >From 6b123c956ac7abe0210cf7b1145ebe0a68f04713 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= Date: Sat, 17 Apr 2021 21:48:30 +0200 Subject: [PATCH] ob-tangle.el: Improve tangling ,* lisp/ob-tangle.el (org-babel-tangle-collect-blocks): Group collected blocks by tangled file name. (org-babel-tangle): Avoid quadratic behavior in number of blocks and set modes before writing to file. --- lisp/ob-tangle.el | 151 ++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 78 deletions(-) diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el index 4c0c3132d..8ca6b66fe 100644 --- a/lisp/ob-tangle.el +++ b/lisp/ob-tangle.el @@ -225,67 +225,55 @@ matching a regular expression." (or (cdr (assq :tangle (nth 2 (org-babel-get-src-block-info 'light)))) (user-error "Point is not in a source code block")))) path-collector) - (mapc ;; map over all languages - (lambda (by-lang) - (let* ((lang (car by-lang)) - (specs (cdr by-lang)) - (ext (or (cdr (assoc lang org-babel-tangle-lang-exts)) lang)) - (lang-f (org-src-get-lang-mode lang)) - she-banged) - (mapc - (lambda (spec) - (let ((get-spec (lambda (name) (cdr (assoc name (nth 4 spec)))))) - (let* ((tangle (funcall get-spec :tangle)) - (she-bang (let ((sheb (funcall get-spec :shebang))) - (when (> (length sheb) 0) sheb))) - (tangle-mode (funcall get-spec :tangle-mode)) - (base-name (cond - ((string= "yes" tangle) - (file-name-sans-extension - (nth 1 spec))) - ((string= "no" tangle) nil) - ((> (length tangle) 0) tangle))) - (file-name (when base-name - ;; decide if we want to add ext to base-name - (if (and ext (string= "yes" tangle)) - (concat base-name "." ext) base-name)))) - (when file-name - ;; Possibly create the parent directories for file. - (let ((m (funcall get-spec :mkdirp)) - (fnd (file-name-directory file-name))) - (and m fnd (not (string= m "no")) - (make-directory fnd 'parents))) - ;; delete any old versions of file - (and (file-exists-p file-name) - (not (member file-name (mapcar #'car path-collector))) - (delete-file file-name)) - ;; drop source-block to file - (with-temp-buffer - (when (fboundp lang-f) (ignore-errors (funcall lang-f))) - (when (and she-bang (not (member file-name she-banged))) + (mapc ;; map over file-names + (lambda (by-fn) + (let ((file-name (car by-fn))) + (when file-name + (let ((lspecs (cdr by-fn)) + (fnd (file-name-directory file-name)) + modes make-dir she-banged lang) + ;; drop source-blocks to file + ;; We avoid append-to-file as it does not work with tramp. + (with-temp-buffer + (mapc + (lambda (lspec) + (let* ((block-lang (car lspec)) + (spec (cdr lspec)) + (get-spec (lambda (name) (cdr (assq name (nth 4 spec))))) + (she-bang (let ((sheb (funcall get-spec :shebang))) + (when (> (length sheb) 0) sheb))) + (tangle-mode (funcall get-spec :tangle-mode))) + (unless (string-equal block-lang lang) + (setq lang block-lang) + (let ((lang-f (org-src-get-lang-mode lang))) + (when (fboundp lang-f) (ignore-errors (funcall lang-f))))) + ;; if file contains she-bangs, then make it executable + (when she-bang + (unless tangle-mode (setq tangle-mode #o755))) + (when tangle-mode + (add-to-list modes tangle-mode)) + ;; Possibly create the parent directories for file. + (let ((m (funcall get-spec :mkdirp))) + (and m fnd (not (string= m "no")) + (setq make-dir t))) + ;; Handle :padlines unless first line in file + (unless (or (string= "no" (funcall get-spec :padline)) + (= (point) (point-min))) + (insert "\n")) + (when (and she-bang (not she-banged)) (insert (concat she-bang "\n")) - (setq she-banged (cons file-name she-banged))) - (org-babel-spec-to-string spec) - ;; We avoid append-to-file as it does not work with tramp. - (let ((content (buffer-string))) - (with-temp-buffer - (when (file-exists-p file-name) - (insert-file-contents file-name)) - (goto-char (point-max)) - ;; Handle :padlines unless first line in file - (unless (or (string= "no" (cdr (assq :padline (nth 4 spec)))) - (= (point) (point-min))) - (insert "\n")) - (insert content) - (write-region nil nil file-name)))) - ;; if files contain she-bangs, then make the executable - (when she-bang - (unless tangle-mode (setq tangle-mode #o755))) - ;; update counter - (setq block-counter (+ 1 block-counter)) - (unless (assoc file-name path-collector) - (push (cons file-name tangle-mode) path-collector)))))) - specs))) + (setq she-banged t)) + (org-babel-spec-to-string spec) + (setq block-counter (+ 1 block-counter)))) + lspecs) + (when make-dir + (make-directory fnd 'parents)) + ;; erase previous file and set permissions on empty + ;; file before writing + (write-region "" nil file-name nil 0) + (mapc (lambda (mode) (set-file-modes file-name mode)) modes) + (write-region nil nil file-name) + (push file-name path-collector)))))) (if (equal arg '(4)) (org-babel-tangle-single-block 1 t) (org-babel-tangle-collect-blocks lang-re tangle-file))) @@ -300,12 +288,8 @@ matching a regular expression." (lambda (file) (org-babel-with-temp-filebuffer file (run-hooks 'org-babel-post-tangle-hook))) - (mapcar #'car path-collector))) - ;; set permissions on tangled files - (mapc (lambda (pair) - (when (cdr pair) (set-file-modes (car pair) (cdr pair)))) - path-collector) - (mapcar #'car path-collector))))) + path-collector)) + path-collector)))) (defun org-babel-tangle-clean () "Remove comments inserted by `org-babel-tangle'. @@ -368,12 +352,12 @@ that the appropriate major-mode is set. SPEC has the form: (defun org-babel-tangle-collect-blocks (&optional lang-re tangle-file) "Collect source blocks in the current Org file. -Return an association list of source-code block specifications of -the form used by `org-babel-spec-to-string' grouped by language. -Optional argument LANG-RE can be used to limit the collected -source code blocks by languages matching a regular expression. -Optional argument TANGLE-FILE can be used to limit the collected -code blocks by target file." +Return an association list of language and source-code block +specifications of the form used by `org-babel-spec-to-string' +grouped by tangled file name. Optional argument LANG-RE can be +used to limit the collected source code blocks by languages +matching a regular expression. Optional argument TANGLE-FILE can +be used to limit the collected code blocks by target file." (let ((counter 0) last-heading-pos blocks) (org-babel-map-src-blocks (buffer-file-name) (let ((current-heading-pos @@ -390,12 +374,23 @@ code blocks by target file." (unless (or (string= src-tfile "no") (and tangle-file (not (equal tangle-file src-tfile))) (and lang-re (not (string-match-p lang-re src-lang)))) - ;; Add the spec for this block to blocks under its - ;; language. - (let ((by-lang (assoc src-lang blocks)) - (block (org-babel-tangle-single-block counter))) - (if by-lang (setcdr by-lang (cons block (cdr by-lang))) - (push (cons src-lang (list block)) blocks))))))) + ;; Add the spec for this block to blocks under its tangled + ;; file name. + (let* ((block (org-babel-tangle-single-block counter)) + (base-name (cond + ((string= "yes" src-tfile) + ;; buffer name + (file-name-sans-extension + (nth 1 block))) + ((> (length src-tfile) 0) src-tfile))) + (ext (or (cdr (assoc src-lang org-babel-tangle-lang-exts)) src-lang)) + (file-name (when base-name + ;; decide if we want to add ext to base-name + (if (and ext (string= "yes" src-tfile)) + (concat base-name "." ext) base-name))) + (by-fn (assoc file-name blocks))) + (if by-fn (setcdr by-fn (cons (cons src-lang block) (cdr by-fn))) + (push (cons file-name (list (cons src-lang block))) blocks))))))) ;; Ensure blocks are in the correct order. (mapcar (lambda (b) (cons (car b) (nreverse (cdr b)))) (nreverse blocks)))) -- 2.31.1 --------------37867A7BD3EBDE03D8B83D52--