emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
From: "Sébastien Miquel" <sebastien.miquel@posteo.eu>
To: Tom Gillespie <tgbugs@gmail.com>
Cc: emacs-orgmode <emacs-orgmode@gnu.org>
Subject: Re: [PATCH] ob-tangle.el: Speed up tangling
Date: Mon, 19 Apr 2021 08:05:40 +0000	[thread overview]
Message-ID: <b65831ff-0790-8e9e-cf4e-f7d13dbf8005@posteo.eu> (raw)
In-Reply-To: <CA+G3_PMBOOY+jLNuR5C131Gjv2WHXcZsRu8SHz_p9B0p22R1QA@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

Hi Tom,

Thank you for the comments.

Tom Gillespie writes:
 > All of the issues that I'm aware of are related to what
 > happens if tangling fails part way through the process.

That's not something I had considered. I wrote a new version of the
patch (attached) which addresses the insecure behaviour and the
possibility of failure. Please tell me what you think.

I've also
  + silenced the ~write-region~ messages, since I'm now writing to
    temporary files.
  + added the list of tangled files to the message to the user at the
    end of the tangling process.
  + replaced the use of ~when-let~.


Sébastien Miquel

[-- Attachment #2: 0001-ob-tangle.el-Improve-tangling.patch --]
[-- Type: text/x-patch, Size: 9854 bytes --]

From 82e4c1beade71194c90d377cdff7ef23532f4aa2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Sat, 17 Apr 2021 21:48:30 +0200
Subject: [PATCH] ob-tangle.el: Improve tangling

,* lisp/ob-tangle.el (org-babel-tangle-collect-blocks): Group
collected blocks by tangled file name.
(org-babel-tangle): Avoid quadratic behavior in number of blocks.
Preserve original file in case of failure. Display the list of tangled
files at the end.
 lisp/ob-tangle.el | 167 ++++++++++++++++++++++++----------------------
 1 file changed, 87 insertions(+), 80 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 4c0c3132d..efafef5b8 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -225,87 +225,83 @@ matching a regular expression."
 	       (or (cdr (assq :tangle (nth 2 (org-babel-get-src-block-info 'light))))
 		   (user-error "Point is not in a source code block"))))
-	(mapc ;; map over all languages
-	 (lambda (by-lang)
-	   (let* ((lang (car by-lang))
-		  (specs (cdr by-lang))
-		  (ext (or (cdr (assoc lang org-babel-tangle-lang-exts)) lang))
-		  (lang-f (org-src-get-lang-mode lang))
-		  she-banged)
-	     (mapc
-	      (lambda (spec)
-		(let ((get-spec (lambda (name) (cdr (assoc name (nth 4 spec))))))
-		  (let* ((tangle (funcall get-spec :tangle))
-			 (she-bang (let ((sheb (funcall get-spec :shebang)))
-                                     (when (> (length sheb) 0) sheb)))
-			 (tangle-mode (funcall get-spec :tangle-mode))
-			 (base-name (cond
-				     ((string= "yes" tangle)
-				      (file-name-sans-extension
-				       (nth 1 spec)))
-				     ((string= "no" tangle) nil)
-				     ((> (length tangle) 0) tangle)))
-			 (file-name (when base-name
-				      ;; decide if we want to add ext to base-name
-				      (if (and ext (string= "yes" tangle))
-					  (concat base-name "." ext) base-name))))
-		    (when file-name
-		      ;; Possibly create the parent directories for file.
-		      (let ((m (funcall get-spec :mkdirp))
-			    (fnd (file-name-directory file-name)))
-			(and m fnd (not (string= m "no"))
-			     (make-directory fnd 'parents)))
-		      ;; delete any old versions of file
-		      (and (file-exists-p file-name)
-			   (not (member file-name (mapcar #'car path-collector)))
-			   (delete-file file-name))
-		      ;; drop source-block to file
-		      (with-temp-buffer
-			(when (fboundp lang-f) (ignore-errors (funcall lang-f)))
-			(when (and she-bang (not (member file-name she-banged)))
+	(mapc ;; map over file-names
+	 (lambda (by-fn)
+	   (let ((file-name (car by-fn)))
+	     (when file-name
+               (let ((lspecs (cdr by-fn))
+		     (fnd (file-name-directory file-name))
+		     modes make-dir she-banged lang)
+	         ;; drop source-blocks to file
+	         ;; We avoid append-to-file as it does not work with tramp.
+	         (with-temp-buffer
+		   (mapc
+		    (lambda (lspec)
+		      (let* ((block-lang (car lspec))
+			     (spec (cdr lspec))
+			     (get-spec (lambda (name) (cdr (assq name (nth 4 spec)))))
+			     (she-bang (let ((sheb (funcall get-spec :shebang)))
+				         (when (> (length sheb) 0) sheb)))
+			     (tangle-mode (funcall get-spec :tangle-mode)))
+		        (unless (string-equal block-lang lang)
+			  (setq lang block-lang)
+			  (let ((lang-f (org-src-get-lang-mode lang)))
+			    (when (fboundp lang-f) (ignore-errors (funcall lang-f)))))
+		        ;; if file contains she-bangs, then make it executable
+		        (when she-bang
+			  (unless tangle-mode (setq tangle-mode #o755)))
+		        (when tangle-mode
+			  (add-to-list modes tangle-mode))
+		        ;; Possibly create the parent directories for file.
+		        (let ((m (funcall get-spec :mkdirp)))
+			  (and m fnd (not (string= m "no"))
+			       (setq make-dir t)))
+		        ;; Handle :padlines unless first line in file
+		        (unless (or (string= "no" (funcall get-spec :padline))
+				    (= (point) (point-min)))
+			  (insert "\n"))
+		        (when (and she-bang (not she-banged))
 			  (insert (concat she-bang "\n"))
-			  (setq she-banged (cons file-name she-banged)))
-			(org-babel-spec-to-string spec)
-			;; We avoid append-to-file as it does not work with tramp.
-			(let ((content (buffer-string)))
-			  (with-temp-buffer
-			    (when (file-exists-p file-name)
-			      (insert-file-contents file-name))
-			    (goto-char (point-max))
-			    ;; Handle :padlines unless first line in file
-			    (unless (or (string= "no" (cdr (assq :padline (nth 4 spec))))
-					(= (point) (point-min)))
-			      (insert "\n"))
-			    (insert content)
-			    (write-region nil nil file-name))))
-		      ;; if files contain she-bangs, then make the executable
-		      (when she-bang
-			(unless tangle-mode (setq tangle-mode #o755)))
-		      ;; update counter
-		      (setq block-counter (+ 1 block-counter))
-		      (unless (assoc file-name path-collector)
-			(push (cons file-name tangle-mode) path-collector))))))
-	      specs)))
+			  (setq she-banged t))
+		        (org-babel-spec-to-string spec)
+		        (setq block-counter (+ 1 block-counter))))
+		    lspecs)
+		   (when make-dir
+		     (make-directory fnd 'parents))
+                   ;; write buffer to file-name using a tmp-file to try to preserve
+                   ;; the original file in case of failure
+                   (let ((tmp-file (concat file-name ".tangling")))
+                     (condition-case ex
+                         (progn
+                           ;; set permissions on empty file before writing
+                           (write-region "" nil tmp-file nil 0)
+		           (mapc (lambda (mode) (set-file-modes tmp-file mode)) modes)
+		           (write-region nil nil tmp-file nil 0)
+                           (rename-file tmp-file file-name t)
+                           (push file-name path-collector))
+                       ('error
+                        (message (format "Failed to tangle to %s: [%s]" file-name ex))
+                        ;; try to cleanup
+                        (when (file-exists-p tmp-file)
+                          (ignore-errors (delete-file tmp-file)))))))))))
 	 (if (equal arg '(4))
 	     (org-babel-tangle-single-block 1 t)
 	   (org-babel-tangle-collect-blocks lang-re tangle-file)))
-	(message "Tangled %d code block%s from %s" block-counter
+	(message "Tangled %d code block%s from %s to %s" block-counter
 		 (if (= block-counter 1) "" "s")
-		   (or (buffer-base-buffer) (current-buffer)))))
+		   (or (buffer-base-buffer) (current-buffer))))
+                 (if (= (length path-collector) 1)
+                     (car path-collector) path-collector))
 	;; run `org-babel-post-tangle-hook' in all tangled files
 	(when org-babel-post-tangle-hook
 	   (lambda (file)
 	     (org-babel-with-temp-filebuffer file
 	       (run-hooks 'org-babel-post-tangle-hook)))
-	   (mapcar #'car path-collector)))
-	;; set permissions on tangled files
-	(mapc (lambda (pair)
-		(when (cdr pair) (set-file-modes (car pair) (cdr pair))))
-	      path-collector)
-	(mapcar #'car path-collector)))))
+	   path-collector))
+	path-collector))))
 (defun org-babel-tangle-clean ()
   "Remove comments inserted by `org-babel-tangle'.
@@ -368,12 +364,12 @@ that the appropriate major-mode is set.  SPEC has the form:
 (defun org-babel-tangle-collect-blocks (&optional lang-re tangle-file)
   "Collect source blocks in the current Org file.
-Return an association list of source-code block specifications of
-the form used by `org-babel-spec-to-string' grouped by language.
-Optional argument LANG-RE can be used to limit the collected
-source code blocks by languages matching a regular expression.
-Optional argument TANGLE-FILE can be used to limit the collected
-code blocks by target file."
+Return an association list of language and source-code block
+specifications of the form used by `org-babel-spec-to-string'
+grouped by tangled file name. Optional argument LANG-RE can be
+used to limit the collected source code blocks by languages
+matching a regular expression. Optional argument TANGLE-FILE can
+be used to limit the collected code blocks by target file."
   (let ((counter 0) last-heading-pos blocks)
     (org-babel-map-src-blocks (buffer-file-name)
       (let ((current-heading-pos
@@ -390,12 +386,23 @@ code blocks by target file."
 	  (unless (or (string= src-tfile "no")
 		      (and tangle-file (not (equal tangle-file src-tfile)))
 		      (and lang-re (not (string-match-p lang-re src-lang))))
-	    ;; Add the spec for this block to blocks under its
-	    ;; language.
-	    (let ((by-lang (assoc src-lang blocks))
-		  (block (org-babel-tangle-single-block counter)))
-	      (if by-lang (setcdr by-lang (cons block (cdr by-lang)))
-		(push (cons src-lang (list block)) blocks)))))))
+	    ;; Add the spec for this block to blocks under its tangled
+	    ;; file name.
+	    (let* ((block (org-babel-tangle-single-block counter))
+		   (base-name (cond
+			       ((string= "yes" src-tfile)
+                                ;; buffer name
+				(file-name-sans-extension
+				 (nth 1 block)))
+			       ((> (length src-tfile) 0) src-tfile)))
+		   (ext (or (cdr (assoc src-lang org-babel-tangle-lang-exts)) src-lang))
+		   (file-name (when base-name
+				;; decide if we want to add ext to base-name
+				(if (and ext (string= "yes" src-tfile))
+				    (concat base-name "." ext) base-name)))
+		   (by-fn (assoc file-name blocks)))
+	      (if by-fn (setcdr by-fn (cons (cons src-lang block) (cdr by-fn)))
+		(push (cons file-name (list (cons src-lang block))) blocks)))))))
     ;; Ensure blocks are in the correct order.
     (mapcar (lambda (b) (cons (car b) (nreverse (cdr b))))
 	    (nreverse blocks))))

  reply	other threads:[~2021-04-19  8:06 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-18  7:22 Sébastien Miquel
2021-04-18 18:47 ` Tom Gillespie
2021-04-19  8:05   ` Sébastien Miquel [this message]
2021-04-20  8:33     ` Tom Gillespie
2021-04-21  6:33       ` Sébastien Miquel
2021-04-21  8:02         ` Timothy
2021-05-01 15:09         ` Bastien
2021-05-01 20:13           ` Sébastien Miquel
2021-05-01 20:32             ` Bastien

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://www.orgmode.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b65831ff-0790-8e9e-cf4e-f7d13dbf8005@posteo.eu \
    --to=sebastien.miquel@posteo.eu \
    --cc=emacs-orgmode@gnu.org \
    --cc=tgbugs@gmail.com \
    --subject='Re: [PATCH] ob-tangle.el: Speed up tangling' \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).