From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id UP1bCyN/fGBfVgEAgWs5BA (envelope-from ) for ; Sun, 18 Apr 2021 20:49:07 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id KF7nBSN/fGCuMAAAbx9fmQ (envelope-from ) for ; Sun, 18 Apr 2021 18:49:07 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9469119F1B for ; Sun, 18 Apr 2021 20:49:06 +0200 (CEST) Received: from localhost ([::1]:46624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lYCTh-0008DS-Hz for larch@yhetil.org; Sun, 18 Apr 2021 14:49:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33066) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYCSg-0008CR-TH for emacs-orgmode@gnu.org; Sun, 18 Apr 2021 14:48:02 -0400 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]:39601) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lYCSd-0004oC-Vg for emacs-orgmode@gnu.org; Sun, 18 Apr 2021 14:48:02 -0400 Received: by mail-wr1-x42d.google.com with SMTP id s7so31615314wru.6 for ; Sun, 18 Apr 2021 11:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jSDYgpX9DiA0Ne0oI1go4sSckiNIjw39iU5kVWn6yjk=; b=qRYUKyRouNs3G/3Z4hArxVNqOTpYpnK1sgRm86YiHXrLs3QUCAZgwwV0EVjUkqU27p Ab75sIvCVf6kEUJG/C4q2HiA46UQv6r1w7//jjZA/VHVIQ6FBR73lcCfqwHXmzqHv4mh Ig3nQPbevLzMSSUOYRlWvVzuI3Wd4PFJN2Jfw8COkpAKSz5MF0PIL/qABzunC0K6B/mX x8Ps+JpE9Qzxdm04UoT20O/UxE+5OcxzyH2ijxJjPWxwWfAZEEsWT6C+lk9GYVUZBw66 owf8JxnK1djdqE/D7LdrZekJzBwLvptBISafsdtkXYhg+s3La2DZY+IaUIkD4e6n2Xzl 45Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jSDYgpX9DiA0Ne0oI1go4sSckiNIjw39iU5kVWn6yjk=; b=rpThjZG8ZO5tvmMMkROU/tno166uwLPQR9++pPUps0kxYIE/Fb822JcR/5vHJsQU8D qhlnkgatuP0LpgMvSLcMBPjkbOWpS871ghukMW/X9/3jajN3AzhgCnLjLW8wX+MlfCs2 S9q+sBdYmeHZBB4zuSjbGJzuz/S+3Lb1hUuW8DMpteBMiVHHqEiNnzXJEBAwZd+YDRNI PWKODq5cozrqRTUD1xLSRobV9MgL6SdCEyN1/+5pmIB3Gcx+o/yna5bxtBTQ1TNyap6o rmdtHlAt3ug7Ap9mmQlalRpFbO93oJ+hvjwq1Je369pOFJd1te1CE4mQrAuLjGsABGOL Tn7A== X-Gm-Message-State: AOAM531zB4AjFbMMvl1Ez9ku9Hy8wFRQ6YITlG7ycP3S7C1mEPef6izc RgWaddmpjlK5ZoW52+K+jJf25yAtygYhJaE2AT0V8ebs7+g= X-Google-Smtp-Source: ABdhPJy/cBt+SnOe4Jl9TS7qtDbNAQtegFZUTVI01hLA1+fP+UMEs6S00QcupaTapyUyXEiymsmUALCXkarygA8cLR4= X-Received: by 2002:adf:fd90:: with SMTP id d16mr10170343wrr.96.1618771678043; Sun, 18 Apr 2021 11:47:58 -0700 (PDT) MIME-Version: 1.0 References: <57480e77-024a-adcc-ec9a-c20b84ac762a@posteo.eu> In-Reply-To: <57480e77-024a-adcc-ec9a-c20b84ac762a@posteo.eu> From: Tom Gillespie Date: Sun, 18 Apr 2021 11:47:46 -0700 Message-ID: Subject: Re: [PATCH] ob-tangle.el: Speed up tangling To: sebastien.miquel@posteo.eu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=tgbugs@gmail.com; helo=mail-wr1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: emacs-orgmode Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618771746; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=jSDYgpX9DiA0Ne0oI1go4sSckiNIjw39iU5kVWn6yjk=; b=kDrFsKLdTfxU3p528FKks2le14gToMBZqby560iNUB2MDquMEk679x8d0DPY1OSnLcTcf5 XNtcdXF3FdsZVZcBtBTWDrD3KfscakjQehgYJd32WMSKsbY/uFJEu8Fq9HYIjgEinWn3oo Tjjjtd7d6GsUD9opa5JkdMqPNVwEnd8+KQQ7ElFIXP0INL3Fv8exeoKZdYFG+iXiBlv0a2 +gXWkwnWNHf/5sBM/M3nA/1BZQO4JjI43N2ULMA6aVUnXCZUF7ptEPpk3yUmOrx60Bj6Yk OhYgcWH1svE0ndz3xx4zqbNHb4G+1PRklb67fvYocjxvOWSYZC9M6IdnQzt9wQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618771746; a=rsa-sha256; cv=none; b=oLvCnxfUupEtHNAXfjLZcCGemp8cZZn7avYobCfTkdCS7kSKkpD6iU49aEo2GD9N6wIhWE uvqjyGShulYdWmmeQG3sV5cpZOu7bzGZhVRo6jVrjmCIL9S9KIwTVnhj7VMqOQXSvGcnzZ 4C8AOn6LOIhJr/5GGMx5VNO4KeyDzCbE6yD+1yy2o9fDP2ybNWYbH2Gsj5hagK6DGS1c9q fFkNEXEX1i4OOMZGaEK8+9K6bdn3rIaRznHklLS0jk9mTtjxmGmn+fxbJdGh7KkZ4VTyEO ZbMx3p7WGpre6y0huz7f91yI5Col3BDNV+UDiGc3ErTNrO5mT66VX0iiLA1UPg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=qRYUKyRo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -3.14 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=qRYUKyRo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 9469119F1B X-Spam-Score: -3.14 X-Migadu-Scanner: scn0.migadu.com X-TUID: AZu4j6wJtxhG Hi S=C3=A9bastien, Some comments while looking over this (will report back when I have tested it out as well). This is a section of the ob export functionality that I have been looking for on and off for quite a while because it is responsible for some bad and insecure behavior. I think that some of your changes may have fixed/improved this as a side effect. I don't know whether it is worth doing anything about the issues in this patch, but since we are here, I think they are worth mentioning. All of the issues that I'm aware of are related to what happens if tangling fails part way through the process. First, your patch already fixes a major issue which is that the modes of all files would not be set if any one of them failed to tangle. Next, during the process the existing file is deleted prior to tangling, which means that it cannot be restored if tangling fails, it would be better if the old file was moved to a temporary location and then deleted on success or replaced on failure. This likely requires wrapping the bits that can fail in unwind-protect and restoring on failure or fully deleting at the end of success. The next issue is that setting the tangle mode should happen before the file is written, an empty file should be created, the mode should then be set, the contents of the file should be written only after the mode has been set. This involves a bit of reordering of operations in lines 124-126 of your patch. This ordering of opertions prevents security issues related to race conditions and potential errors being evoked during write-region (though again, your changes already make the tangling code much more secure by setting the modes on each file immediately after writing instead of how it works currently where if any other block encounters an error then no modes were set). Best! Tom On Sun, Apr 18, 2021 at 12:23 AM S=C3=A9bastien Miquel wrote: > > Hi, > > The attached patch modifies the ~org-babel-tangle~ function to avoid a > quadratic behavior in the number of blocks tangled to a single file. > > Tangling an org buffer with 200 blocks to 5 different files yields a > 25 % speedup. > > > * lisp/ob-tangle.el (org-babel-tangle-collect-blocks): Group > collected blocks by tangled file name. > (org-babel-tangle): Avoid quadratic behavior in number of blocks. > > -- > S=C3=A9bastien Miquel