emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* when does :cache not cache?
@ 2021-10-22  7:55 Eric S Fraga
  2021-12-12  7:45 ` Ihor Radchenko
  0 siblings, 1 reply; 8+ messages in thread
From: Eric S Fraga @ 2021-10-22  7:55 UTC (permalink / raw)
  To: Emacs Org mode mailing list

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

Hello all (again!),

in the paper I am writing, I have a number of gnuplot src blocks, some
of which process a significant amount of data so take some time to
generate the actual plots.  The data are static so caching the results
make sense.  However, even though I have ":cache yes" on each of these
named src blocks, and I have (for good measure), the property
"header-args:gnuplot" set to ":cache yes" as well, the plots are being
regenerated each time I export to PDF via LaTeX.

In trying to create a minimal example (see attached), it seems that
caching stops working as soon as I add a ":var data=..." header argument
to generate a plot from a table of data.

Why does specifying a variable to a src block violate the caching
directive?  Is this intended behaviour (assuming I've understood what is
happening correctly)?

Thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096

[-- Attachment #2: gnuplot.org --]
[-- Type: application/vnd.lotus-organizer, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: when does :cache not cache?
  2021-10-22  7:55 when does :cache not cache? Eric S Fraga
@ 2021-12-12  7:45 ` Ihor Radchenko
  2021-12-13  8:13   ` Eric S Fraga
  0 siblings, 1 reply; 8+ messages in thread
From: Ihor Radchenko @ 2021-12-12  7:45 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: Emacs Org mode mailing list

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> in the paper I am writing, I have a number of gnuplot src blocks, some
> of which process a significant amount of data so take some time to
> generate the actual plots.  The data are static so caching the results
> make sense.  However, even though I have ":cache yes" on each of these
> named src blocks, and I have (for good measure), the property
> "header-args:gnuplot" set to ":cache yes" as well, the plots are being
> regenerated each time I export to PDF via LaTeX.

I was able to reproduce. The reason why caching does not work is related
to the way :var assignments work in ob-gnuplot. We dump the table data
into temporary files and refer to those files in generated gnuplot
script body. The temporary files names change on every execution and the
gnuplot script hash is never going to be the same.

I am attaching tentative patch to fix the issue.
The patch introduces a new functionality to ob-core.el allowing more
stable temporary file names. I am not sure if my implementation is the
best way to solve the problem, so comments are welcome.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-gnuplot.el-Make-cache-argument-work-with-var-assi.patch --]
[-- Type: text/x-diff, Size: 5773 bytes --]

From ac11b4d08edd577b29a398296364b4340096a6ae Mon Sep 17 00:00:00 2001
Message-Id: <ac11b4d08edd577b29a398296364b4340096a6ae.1639294626.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Dec 2021 15:31:35 +0800
Subject: [PATCH] ob-gnuplot.el: Make :cache argument work with :var
 assignments

* lisp/ob-core.el (org-babel-temporary-stable-directory): New variable
holding a temporary directory name that does not change between Emacs
sessions.
(org-babel-remove-temporary-stable-directory): New function removing
`org-babel-temporary-stable-directory' on Emacs shutdown.
(org-babel-temp-stable-file): Generate stable temporary file name for
object storage.  The file name is constant with for equal objects.
(org-babel-execute-src-block): Explicitly identify that if the result
is cached.
* lisp/ob-gnuplot.el (org-babel-gnuplot-process-vars): Make use of
`org-babel-stable-file' to make expanded body stable with respect to
:var assignments.

Fixes https://orgmode.org/list/87mtn1o5mn.fsf@ucl.ac.uk
---
 lisp/ob-core.el    | 52 +++++++++++++++++++++++++++++++++++++++++++++-
 lisp/ob-gnuplot.el |  5 ++++-
 2 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 7a9467b0e..d572423b7 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -735,7 +735,8 @@ (defun org-babel-execute-src-block (&optional arg info params)
 	    (forward-line)
 	    (skip-chars-forward " \t")
 	    (let ((result (org-babel-read-result)))
-	      (message (replace-regexp-in-string "%" "%%" (format "%S" result)))
+	      (message (format "Cached: %s"
+                               (replace-regexp-in-string "%" "%%" (format "%S" result))))
 	      result)))
 	 ((org-babel-confirm-evaluate info)
 	  (let* ((lang (nth 0 info))
@@ -3112,6 +3113,22 @@   (defvar org-babel-temporary-directory
 Used by `org-babel-temp-file'.  This directory will be removed on
 Emacs shutdown."))
 
+(defvar org-babel-temporary-stable-directory)
+(unless (or noninteractive (boundp 'org-babel-temporary-stable-directory))
+  (defvar org-babel-temporary-stable-directory
+    (or (and (boundp 'org-babel-temporary-stable-directory)
+	     (file-exists-p org-babel-temporary-stable-directory)
+	     org-babel-temporary-stable-directory)
+        (condition-case nil
+            (make-directory
+	     (expand-file-name
+              "babel-stable"
+              (temporary-file-directory)))
+          (t nil)))
+    "Directory to hold temporary files created to execute code blocks.
+Used by `org-babel-temp-file'.  This directory will be removed on
+Emacs shutdown."))
+
 (defcustom org-babel-remote-temporary-directory "/tmp/"
   "Directory to hold temporary files on remote hosts."
   :group 'org-babel
@@ -3155,6 +3172,30 @@ (defun org-babel-temp-file (prefix &optional suffix)
 	       temporary-file-directory)))
       (make-temp-file prefix nil suffix))))
 
+(defun org-babel-temp-stable-file (data prefix &optional suffix)
+  "Create a temporary file in the `org-babel-remove-temporary-stable-directory'.
+The file name is stable with respect to DATA.  The file name is
+constructed like the following: PREFIXDATAhashSUFFIX."
+  (if (file-remote-p default-directory)
+      (let* ((prefix
+              (concat (file-remote-p default-directory)
+                      (expand-file-name
+		       prefix org-babel-temporary-stable-directory)))
+             (path (concat prefix (format "%s" (sxhash data)) (or suffix ""))))
+        (with-temp-file path)
+        path)
+    (let* ((temporary-file-directory
+	    (or (and (boundp 'org-babel-temporary-stable-directory)
+		     (file-exists-p org-babel-temporary-stable-directory)
+		     org-babel-temporary-stable-directory)
+	        temporary-file-directory))
+           (path (concat
+                  (expand-file-name
+		   prefix org-babel-temporary-stable-directory)
+                  (format "%s" (sxhash data)) (or suffix ""))))
+      (with-temp-file path)
+      path)))
+
 (defun org-babel-remove-temporary-directory ()
   "Remove `org-babel-temporary-directory' on Emacs shutdown."
   (when (and (boundp 'org-babel-temporary-directory)
@@ -3178,7 +3219,16 @@ (defun org-babel-remove-temporary-directory ()
 		    org-babel-temporary-directory
 		  "[directory not defined]"))))))
 
+(defun org-babel-remove-temporary-stable-directory ()
+  "Remove `org-babel-temporary-stable-directory' and on Emacs shutdown."
+  (when (and (boundp 'org-babel-temporary-stable-directory)
+	     (file-exists-p org-babel-temporary-stable-directory))
+    (let ((org-babel-temporary-directory
+           org-babel-temporary-stable-directory))
+      (org-babel-remove-temporary-directory))))
+
 (add-hook 'kill-emacs-hook #'org-babel-remove-temporary-directory)
+(add-hook 'kill-emacs-hook #'org-babel-remove-temporary-stable-directory)
 
 (defun org-babel-one-header-arg-safe-p (pair safe-list)
   "Determine if the PAIR is a safe babel header arg according to SAFE-LIST.
diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index 8c4a5957b..f831b4996 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -94,7 +94,10 @@ (defun org-babel-gnuplot-process-vars (params)
 	       (let* ((first  (car val))
 		      (tablep (or (listp first) (symbolp first))))
 		 (if tablep val (mapcar 'list val)))
-	       (org-babel-temp-file "gnuplot-") params)
+               ;; Make temporary file name stable with respect to data.
+               ;; If we do not do it, :cache argument becomes useless.
+               (org-babel-temp-stable-file params "gnuplot-")
+               params)
 	    (if (and (stringp val)
 		     (file-remote-p val)  ;; check if val is a remote file
 		     (file-exists-p val)) ;; call to file-exists-p is slow, maybe remove it
-- 
2.32.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: when does :cache not cache?
  2021-12-12  7:45 ` Ihor Radchenko
@ 2021-12-13  8:13   ` Eric S Fraga
  0 siblings, 0 replies; 8+ messages in thread
From: Eric S Fraga @ 2021-12-13  8:13 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Emacs Org mode mailing list

Dear Ihor,

> The reason why caching does not work is related to the way :var
> assignments work in ob-gnuplot. We dump the table data into temporary
> files and refer to those files in generated gnuplot script body. The
> temporary files names change on every execution and the gnuplot script
> hash is never going to be the same.

Ah, this makes perfect sense!  Thank you for looking into this.

I will try your patch later this week (busy with end of term teaching
aspects at the moment).

-- 
: Eric S Fraga, with org release_9.5.1-243-gad53c5 in Emacs 29.0.50
: Latest paper written in org: https://arxiv.org/abs/2106.05096


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: when does :cache not cache?
  2021-10-24 11:49 Emmanuel Charpentier
@ 2021-10-25 10:58 ` Eric S Fraga
  0 siblings, 0 replies; 8+ messages in thread
From: Eric S Fraga @ 2021-10-25 10:58 UTC (permalink / raw)
  To: Emmanuel Charpentier; +Cc: emacs-orgmode

On Sunday, 24 Oct 2021 at 13:49, Emmanuel Charpentier wrote:
> Workaround : cache the computations,not the plotting itself (which
> should be fast,and must be made on every table, anyway...) :

actually, the problem in my case is that the plotting *is* the expensive
part!  In case, problem solved/avoided by using ":eval never-export".

thank you,
eric
-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096


^ permalink raw reply	[flat|nested] 8+ messages in thread

* when does :cache not cache?
@ 2021-10-24 11:49 Emmanuel Charpentier
  2021-10-25 10:58 ` Eric S Fraga
  0 siblings, 1 reply; 8+ messages in thread
From: Emmanuel Charpentier @ 2021-10-24 11:49 UTC (permalink / raw)
  To: e.fraga, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

Workaround : cache the computations,not the plotting itself (which
should be fast,and must be made on every table, anyway...) :

#+options: toc:nil author:nil

* Main text

Use of a static table:

#+call: timeit()
#+call: testplot[:file test0.pdf](data=table0)
#+call: timeit()

Use of a slow function:

#+call: timeit()
#+call: testplot[:file test1.pdf](data=table1)
#+call: timeit()

Use of a /cached/ slow function:

#+call: timeit()
#+call: testplot[:file test2.pdf](data=table2)
#+call: timeit()

* Annexes :noexport:

This is not exported, but computes results.

#+name: timeit
#+begin_src emacs-lisp
  (format-time-string "%Hh %Mm %Ss")
#+end_src

#+name: table0
| Val | Square |
|-----+--------|
|   0 |      0 |
|   1 |      1 |
|   2 |      4 |
|   3 |      9 |
|   4 |     16 |
#+TBLFM: $2=$1^2

#+name: table1
#+begin_src emacs-lisp
  (sleep-for 5)
  (setq s ( list (list "x" "x^2") 'hline))
  (dotimes (i 5 s) (setq s (append s (list (cons i (list (* i i)))))))
  s
#+end_src

#+RESULTS[46320b31c46cef901580bad78aee7032d97ffe64]: table1
| x | x^2 |
|---+-----|
| 0 |   0 |
| 1 |   1 |
| 2 |   4 |
| 3 |   9 |
| 4 |  16 |


#+name: table2
#+begin_src emacs-lisp :cache yes
  (sleep-for 5)
  (setq s ( list (list "x" "x^2") 'hline))
  (dotimes (i 5 s) (setq s (append s (list (cons i (list (* i i)))))))
  s
#+end_src

#+name: tf
| festfile.pdf ]

#+RESULTS[46320b31c46cef901580bad78aee7032d97ffe64]: table2
| x | x^2 |
|---+-----|
| 0 |   0 |
| 1 |   1 |
| 2 |   4 |
| 3 |   9 |
| 4 |  16 |

#+name: testplot
#+begin_src gnuplot :var data=table0 :exports results
  reset
  plot data with linespoints
#+end_src

HTH,

--
Emmanuel Charpentier


[-- Attachment #2: Type: text/html, Size: 2727 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: when does :cache not cache?
  2021-10-23 10:09 ` Eric S Fraga
@ 2021-10-23 15:15   ` Eric S Fraga
  0 siblings, 0 replies; 8+ messages in thread
From: Eric S Fraga @ 2021-10-23 15:15 UTC (permalink / raw)
  To: Org Mode List

Hello all,

So, following the code: every time I ask to evaluate the gnuplot src
block in the minimal example I posted yesterday, the new-hash that is
calculated is different so the cache setting is ignored.

Digging deeper leads to me getting lost.  The hash does depend on the
incorporation of the variables set using :var but I do not see why the
hash should be different if the table has not changed.

Anyway, that's as far as I got.

For the record, I have simply put ":eval never-export" on all of my
gnuplot src blocks and the export happens much more quickly now.

eric
-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: when does :cache not cache?
  2021-10-23  9:13 Emmanuel Charpentier
@ 2021-10-23 10:09 ` Eric S Fraga
  2021-10-23 15:15   ` Eric S Fraga
  0 siblings, 1 reply; 8+ messages in thread
From: Eric S Fraga @ 2021-10-23 10:09 UTC (permalink / raw)
  To: Emmanuel Charpentier; +Cc: emacs-orgmode

On Saturday, 23 Oct 2021 at 11:13, Emmanuel Charpentier wrote:
> You may find this John Kitchin's blog
> post [...] illuminating.

Unfortunately not but thank you.  It is a very useful blog post but
doesn't cover my case, where I use a :var header to refer to a
table.  The table is static so I still do not understand why :cache
on the src block that references it doesn't work.

I may have to dive into the code, I guess.  I may be some time... ;-)

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096


^ permalink raw reply	[flat|nested] 8+ messages in thread

* when does :cache not cache?
@ 2021-10-23  9:13 Emmanuel Charpentier
  2021-10-23 10:09 ` Eric S Fraga
  0 siblings, 1 reply; 8+ messages in thread
From: Emmanuel Charpentier @ 2021-10-23  9:13 UTC (permalink / raw)
  To: e.fraga, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 91 bytes --]

You may find this John Kitchin's blog post illuminating.

HTH,

--
Emmanuel Charpentier


[-- Attachment #2: Type: text/html, Size: 366 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-12-13  8:35 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-22  7:55 when does :cache not cache? Eric S Fraga
2021-12-12  7:45 ` Ihor Radchenko
2021-12-13  8:13   ` Eric S Fraga
2021-10-23  9:13 Emmanuel Charpentier
2021-10-23 10:09 ` Eric S Fraga
2021-10-23 15:15   ` Eric S Fraga
2021-10-24 11:49 Emmanuel Charpentier
2021-10-25 10:58 ` Eric S Fraga

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).