From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thierry Banel Subject: Re: speeding up Babel Gnuplot Date: Wed, 04 Jan 2017 00:06:40 +0100 Message-ID: <586C2E80.4050805@free.fr> References: <5864217C.7060001@free.fr> <586963CB.1000006@free.fr> <87d1g6qrqh.fsf@nicolasgoaziou.fr> <586AB3DB.6070702@free.fr> <586C1A53.90601@free.fr> <87wpebbygr.fsf@nicolasgoaziou.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:39934) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cOYAK-0006AS-6a for emacs-orgmode@gnu.org; Tue, 03 Jan 2017 18:06:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cOYAG-0004U5-5o for emacs-orgmode@gnu.org; Tue, 03 Jan 2017 18:06:48 -0500 Received: from smtp1-g21.free.fr ([212.27.42.1]:62478) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cOYAF-0004TY-VD for emacs-orgmode@gnu.org; Tue, 03 Jan 2017 18:06:44 -0500 In-Reply-To: <87wpebbygr.fsf@nicolasgoaziou.fr> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Nicolas Goaziou Cc: emacs-orgmode@gnu.org Le 03/01/2017 22:55, Nicolas Goaziou a =E9crit : > Hello, > > Thierry Banel writes: > >> Here is a patch to avoid generating temporary files multiple times. >> >> There is no way to ensure a single call to >> (org-babel-gnuplot-process-vars) without modifying ob-core.el. I don't >> want to do that because I would have to change a lot of babel backends= . >> Thus, I come back to my first light patch. >> >> A 'param' list is passed around. It reflects the #+BEGIN_SRC header. M= y >> patch changes it in-place from: >> (((:var data (3000) (2999) (2998) (2997) ... >> to: >> (((:var data . "/tmp/babel-16991kSr/gnuplot-16991YBq") ... >> >> The 'param' list behaves as a cache. There is nothing wrong with that. >> The worst thing that can happen is the caching no longer working in ca= se >> 'param' would be copied some day. Results would stay correct. > Thank you. > > What is the benefit of this patch? I mean, > `org-babel-gnuplot-process-vars' is already quite fast here. Do you hav= e > some benchmark for that? The benefit is Babel Gnuplot running twice as fast on large Org tables (thousands of rows). On small tables there is no real benefit. Two temporary files are left over in /tmp. They have identical content: data suitably formatted for Gnuplot. Creating such large temp files out-weights any other Babel processing. Granted, you have already speeded-up `org-babel-gnuplot-process-vars' quite a lot by reworking `org-export-table-row-number'. Now, going down from 4 seconds to 2 seconds on a 5000 rows table (on my computer) is quite pleasant. My patch is very light: `org-babel-gnuplot-process-vars' is the only modified function, and the change involves only adding a (setcdr) instruction to cache the result of a heavy processing. >> (car pair) ;; variable name >> - (let* ((val (cdr pair)) ;; variable value >> - (lp (listp val))) >> - (if lp >> + (let ((val (cdr pair))) ;; variable value >> + (if (not (listp val)) >> + val >> + (let ((temp-file (org-babel-temp-file "gnuplot-")) >> + (first (car val))) >> + (setcdr pair temp-file) ;; <------ caching here > It would be nice to expunge the comment a bit. Yes sure. If the patch is accepted, I'll clean it. > Another option would be to generate a file according to the hash of > contents so `org-babel-gnuplot-process-vars' knows when to create a new > file. Your proposal provides an additional benefit: caching file generation between several invocations of Babel. (The cache in my patch is intended to be used within a single Babel invocation, and is then garbage collected.). The drawback is that we need to go through all rows of the table, compute the hash, just to discover that the hash was already known. The purpose of the cache was precisely to avoid going through the table again. Your proposal involves substantial work. We may also want to extend it to all other Babel backends (R, shell, C, etc.). I may help if enough users need it. Regards Thierry