[PATCH] sha1 hash of latex fragments to avoid regeneration

[PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

Hi,

The attached patch changes the latex fragment image generation so that
it saves images into files named by the sha1 hash of the latex source
code.  By checking for the existence of image files before image
generation the regeneration of identical images is avoided.

In practice I find that this greatly speeds up export to html and the
`org-preview-latex-fragment' command.

Cheers -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-latex-fragment-images-cached-using-sha1-hash-keys.patch --]
[-- Type: text/x-patch, Size: 2887 bytes --]

From 13e1c48fa6cac43b0c87ca0fbc8e349f7a9fa864 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Sun, 15 Nov 2009 17:00:09 -0700
Subject: [PATCH] latex fragment images cached using sha1 hash keys

  Latex fragment images are now saved in files named by the sha1 hash
  of the latex text used to create the image.  By checking if files
  exist before images generation the regeneration of identical latex
  images is avoided.
---
 lisp/ChangeLog |    6 ++++++
 lisp/org.el    |   18 +++++++-----------
 2 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 339f248..f18755c 100755
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,9 @@
+2009-11-16  Eric Schulte  <schulte.eric@gmail.com>
+
+	* org.el (org-format-latex): Latex images are now saved to files
+	named by the sha1 hash of the latex source text avoiding
+	regeneration of identical images.
+
 2009-11-15  Carsten Dominik  <carsten.dominik@gmail.com>
 
 	* org-wl.el (org-wl-store-link): Handle the case that
diff --git a/lisp/org.el b/lisp/org.el
index bf6573b..46348fc 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -14550,15 +14550,9 @@ Some of the options can be changed using the variable
 	 (opt org-format-latex-options)
 	 (matchers (plist-get opt :matchers))
 	 (re-list org-latex-regexps)
-	 (cnt 0) txt link beg end re e checkdir
+	 (cnt 0) txt hash link beg end re e checkdir
 	 executables-checked
 	 m n block linkfile movefile ov)
-    ;; Check if there are old images files with this prefix, and remove them
-    (when (file-directory-p todir)
-      (mapc 'delete-file
-	    (directory-files
-	     todir 'full
-	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
     ;; Check the different regular expressions
     (while (setq e (pop re-list))
       (setq m (car e) re (nth 1 e) n (nth 2 e)
@@ -14576,9 +14570,10 @@ Some of the options can be changed using the variable
 	    (setq txt (match-string n)
 		  beg (match-beginning n) end (match-end n)
 		  cnt (1+ cnt)
-		  linkfile (format "%s_%04d.png" prefix cnt)
-		  movefile (format "%s_%04d.png" absprefix cnt)
 		  link (concat block "[[file:" linkfile "]]" block))
+            (setq hash (sha1 txt)
+		  linkfile (format "%s_%s.png" prefix hash)
+		  movefile (format "%s_%s.png" absprefix hash))
 	    (if msg (message msg cnt))
 	    (goto-char beg)
 	    (unless checkdir ; make sure the directory exists
@@ -14592,8 +14587,9 @@ Some of the options can be changed using the variable
 	       "dvipng" "needed to convert LaTeX fragments to images")
 	      (setq executables-checked t))
 
-	    (org-create-formula-image
-	     txt movefile opt forbuffer)
+            (unless (file-exists-p movefile)
+              (org-create-formula-image
+               txt movefile opt forbuffer))
 	    (if overlays
 		(progn
 		  (mapc (lambda (o)
-- 
1.6.4.73.gc144


[-- Attachment #3: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Carsten Dominik]

Hi Eric,

this is fantastic, thank you for implementing it.  I have wanted some  
speedup
for this for a long time.

I think your implementation still suffers from one issue:

The produced image also depends on the variables org-format-latex- 
options,
org-format-latex-header, org-export-latex-package-alist,
and on the `forbuffer' flag (because images made for display in
the buffer and fo HTML export generaly need different resolution).

One way to deal with this would be to make a list containing the values
of these four variables and using prin1-to-string to convert this list
into a string, and then to prepend this string to TXT when creating
the hash.

Now, I am sure that you are already planning to do the same
for ditaa images etc?  That would be a treat, because ditaa
can be terribly slow for complex figures, and this would speed up
the cycle when writing document by quite a bit.

There is one further issue:  Cleaning up images that are no
longer used.

With the LaTeX fragments it is not a big problem, because there
live in a special directory.  This would be a bigger concern for
ditaa images etc which tend to live in the same directory as the
source.  Maybe that could be solved by

1. Making sure that each image still have a name like "blue", so
    that the name now would be "blus_loooooonghashvalue.png" or so.
2. Maybe creating a command that will look for orphaned images
    and remove them, by looking for the hash in the name and
    checking access times.  I am not sure if this is needed,
    and not sure what would be the best way to implement it.

After looking at these things, I would be *very* happy to accept
this patch.

- Carsten

On Nov 16, 2009, at 1:07 AM, Eric Schulte wrote:

> Hi,
>
> The attached patch changes the latex fragment image generation so that
> it saves images into files named by the sha1 hash of the latex source
> code.  By checking for the existence of image files before image
> generation the regeneration of identical images is avoided.
>
> In practice I find that this greatly speeds up export to html and the
> `org-preview-latex-fragment' command.
>
> Cheers -- Eric
>
> From 13e1c48fa6cac43b0c87ca0fbc8e349f7a9fa864 Mon Sep 17 00:00:00 2001
> From: Eric Schulte <schulte.eric@gmail.com>
> Date: Sun, 15 Nov 2009 17:00:09 -0700
> Subject: [PATCH] latex fragment images cached using sha1 hash keys
>
>  Latex fragment images are now saved in files named by the sha1 hash
>  of the latex text used to create the image.  By checking if files
>  exist before images generation the regeneration of identical latex
>  images is avoided.
> ---
> lisp/ChangeLog |    6 ++++++
> lisp/org.el    |   18 +++++++-----------
> 2 files changed, 13 insertions(+), 11 deletions(-)
>
> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
> index 339f248..f18755c 100755
> --- a/lisp/ChangeLog
> +++ b/lisp/ChangeLog
> @@ -1,3 +1,9 @@
> +2009-11-16  Eric Schulte  <schulte.eric@gmail.com>
> +
> +	* org.el (org-format-latex): Latex images are now saved to files
> +	named by the sha1 hash of the latex source text avoiding
> +	regeneration of identical images.
> +
> 2009-11-15  Carsten Dominik  <carsten.dominik@gmail.com>
>
> 	* org-wl.el (org-wl-store-link): Handle the case that
> diff --git a/lisp/org.el b/lisp/org.el
> index bf6573b..46348fc 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -14550,15 +14550,9 @@ Some of the options can be changed using  
> the variable
> 	 (opt org-format-latex-options)
> 	 (matchers (plist-get opt :matchers))
> 	 (re-list org-latex-regexps)
> -	 (cnt 0) txt link beg end re e checkdir
> +	 (cnt 0) txt hash link beg end re e checkdir
> 	 executables-checked
> 	 m n block linkfile movefile ov)
> -    ;; Check if there are old images files with this prefix, and  
> remove them
> -    (when (file-directory-p todir)
> -      (mapc 'delete-file
> -	    (directory-files
> -	     todir 'full
> -	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
>     ;; Check the different regular expressions
>     (while (setq e (pop re-list))
>       (setq m (car e) re (nth 1 e) n (nth 2 e)
> @@ -14576,9 +14570,10 @@ Some of the options can be changed using  
> the variable
> 	    (setq txt (match-string n)
> 		  beg (match-beginning n) end (match-end n)
> 		  cnt (1+ cnt)
> -		  linkfile (format "%s_%04d.png" prefix cnt)
> -		  movefile (format "%s_%04d.png" absprefix cnt)
> 		  link (concat block "[[file:" linkfile "]]" block))
> +            (setq hash (sha1 txt)
> +		  linkfile (format "%s_%s.png" prefix hash)
> +		  movefile (format "%s_%s.png" absprefix hash))
> 	    (if msg (message msg cnt))
> 	    (goto-char beg)
> 	    (unless checkdir ; make sure the directory exists
> @@ -14592,8 +14587,9 @@ Some of the options can be changed using the  
> variable
> 	       "dvipng" "needed to convert LaTeX fragments to images")
> 	      (setq executables-checked t))
>
> -	    (org-create-formula-image
> -	     txt movefile opt forbuffer)
> +            (unless (file-exists-p movefile)
> +              (org-create-formula-image
> +               txt movefile opt forbuffer))
> 	    (if overlays
> 		(progn
> 		  (mapc (lambda (o)
> -- 
> 1.6.4.73.gc144
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

Hi Carsten,

Thanks for the feedback, I have comments inline below

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Hi Eric,
>
> this is fantastic, thank you for implementing it.  I have wanted some
> speedup
> for this for a long time.
>
> I think your implementation still suffers from one issue:
>
> The produced image also depends on the variables org-format-latex-
> options,
> org-format-latex-header, org-export-latex-package-alist,
> and on the `forbuffer' flag (because images made for display in
> the buffer and fo HTML export generaly need different resolution).
>
> One way to deal with this would be to make a list containing the values
> of these four variables and using prin1-to-string to convert this list
> into a string, and then to prepend this string to TXT when creating
> the hash.
>

That sounds like the best solution.  I have made this change in the
newly attached patch.

>
> Now, I am sure that you are already planning to do the same
> for ditaa images etc?

of course :)

> That would be a treat, because ditaa can be terribly slow for complex
> figures, and this would speed up the cycle when writing document by
> quite a bit.
>

Dan and I have been working on general caching solution for org-babel.
Once we get that sorted out it should provide for the caching of all
org-babel results which would include ditaa, dot, gnuplot, etc...

I am currently more interested in making these changes in org-babel than
in org-exp-blocks, but in this case it may be worth implementing caching
in both cases.

>
> There is one further issue:  Cleaning up images that are no
> longer used.
>
> With the LaTeX fragments it is not a big problem, because there
> live in a special directory.  This would be a bigger concern for
> ditaa images etc which tend to live in the same directory as the
> source.  Maybe that could be solved by
>
> 1. Making sure that each image still have a name like "blue", so
>    that the name now would be "blus_loooooonghashvalue.png" or so.
> 2. Maybe creating a command that will look for orphaned images
>    and remove them, by looking for the hash in the name and
>    checking access times.  I am not sure if this is needed,
>    and not sure what would be the best way to implement it.
>

Yes, this will not be an issue in the org-babel implementation as the
hash key is stored separate from the file name, but I can see how this
would need to be considered for any org-exp-blocks hash-based image
caching.  The first option you propose above sounds very doable, as long
as we are comfortable removing any files that match regular expressions
like

  blue_[[:alnum:]]+\.png

which seems safe enough...

>
> After looking at these things, I would be *very* happy to accept
> this patch.
>

I'll give this some more thought, but perhaps the latex image fragment
patch is now viable.

Best -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-latex-fragment-images-cached-using-sha1-hash-keys.patch --]
[-- Type: text/x-patch, Size: 3184 bytes --]

From 0e9a359c1d5e8f67c20066533171fb1edc11ba61 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Mon, 16 Nov 2009 16:53:34 -0700
Subject: [PATCH] latex fragment images cached using sha1 hash keys

  Latex fragment images are now saved in files named by the sha1 hash
  of the latex text used to create the image.  By checking if files
  exist before images generation the regeneration of identical latex
  images is avoided.
---
 lisp/ChangeLog |    6 ++++++
 lisp/org.el    |   22 +++++++++++-----------
 2 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 5f83aaa..b581931 100755
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,9 @@
+2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
+
+	* org.el (org-format-latex): Latex images are now saved to files
+	named by the sha1 hash of the latex source text avoiding
+	regeneration of identical images.
+
 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
 
 	* org-html.el (org-export-html-home/up-format): Add an ID to the
diff --git a/lisp/org.el b/lisp/org.el
index bf6573b..15a8f9e 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -14550,15 +14550,9 @@ Some of the options can be changed using the variable
 	 (opt org-format-latex-options)
 	 (matchers (plist-get opt :matchers))
 	 (re-list org-latex-regexps)
-	 (cnt 0) txt link beg end re e checkdir
+	 (cnt 0) txt hash link beg end re e checkdir
 	 executables-checked
 	 m n block linkfile movefile ov)
-    ;; Check if there are old images files with this prefix, and remove them
-    (when (file-directory-p todir)
-      (mapc 'delete-file
-	    (directory-files
-	     todir 'full
-	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
     ;; Check the different regular expressions
     (while (setq e (pop re-list))
       (setq m (car e) re (nth 1 e) n (nth 2 e)
@@ -14576,9 +14570,14 @@ Some of the options can be changed using the variable
 	    (setq txt (match-string n)
 		  beg (match-beginning n) end (match-end n)
 		  cnt (1+ cnt)
-		  linkfile (format "%s_%04d.png" prefix cnt)
-		  movefile (format "%s_%04d.png" absprefix cnt)
 		  link (concat block "[[file:" linkfile "]]" block))
+            (setq hash (sha1 (prin1-to-string
+                              (list org-format-latex-header
+                                    (if (boundp 'org-export-latex-package-alist)
+                                        org-export-latex-package-alist)
+                                    forbuffer txt)))
+		  linkfile (format "%s_%s.png" prefix hash)
+		  movefile (format "%s_%s.png" absprefix hash))
 	    (if msg (message msg cnt))
 	    (goto-char beg)
 	    (unless checkdir ; make sure the directory exists
@@ -14592,8 +14591,9 @@ Some of the options can be changed using the variable
 	       "dvipng" "needed to convert LaTeX fragments to images")
 	      (setq executables-checked t))
 
-	    (org-create-formula-image
-	     txt movefile opt forbuffer)
+            (unless (file-exists-p movefile)
+              (org-create-formula-image
+               txt movefile opt forbuffer))
 	    (if overlays
 		(progn
 		  (mapc (lambda (o)
-- 
1.6.4.73.gc144


[-- Attachment #3: Type: text/plain, Size: 3825 bytes --]


>
> - Carsten
>
> On Nov 16, 2009, at 1:07 AM, Eric Schulte wrote:
>
>> Hi,
>>
>> The attached patch changes the latex fragment image generation so that
>> it saves images into files named by the sha1 hash of the latex source
>> code.  By checking for the existence of image files before image
>> generation the regeneration of identical images is avoided.
>>
>> In practice I find that this greatly speeds up export to html and the
>> `org-preview-latex-fragment' command.
>>
>> Cheers -- Eric
>>
>> From 13e1c48fa6cac43b0c87ca0fbc8e349f7a9fa864 Mon Sep 17 00:00:00 2001
>> From: Eric Schulte <schulte.eric@gmail.com>
>> Date: Sun, 15 Nov 2009 17:00:09 -0700
>> Subject: [PATCH] latex fragment images cached using sha1 hash keys
>>
>>  Latex fragment images are now saved in files named by the sha1 hash
>>  of the latex text used to create the image.  By checking if files
>>  exist before images generation the regeneration of identical latex
>>  images is avoided.
>> ---
>> lisp/ChangeLog |    6 ++++++
>> lisp/org.el    |   18 +++++++-----------
>> 2 files changed, 13 insertions(+), 11 deletions(-)
>>
>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>> index 339f248..f18755c 100755
>> --- a/lisp/ChangeLog
>> +++ b/lisp/ChangeLog
>> @@ -1,3 +1,9 @@
>> +2009-11-16  Eric Schulte  <schulte.eric@gmail.com>
>> +
>> +	* org.el (org-format-latex): Latex images are now saved to files
>> +	named by the sha1 hash of the latex source text avoiding
>> +	regeneration of identical images.
>> +
>> 2009-11-15  Carsten Dominik  <carsten.dominik@gmail.com>
>>
>> 	* org-wl.el (org-wl-store-link): Handle the case that
>> diff --git a/lisp/org.el b/lisp/org.el
>> index bf6573b..46348fc 100644
>> --- a/lisp/org.el
>> +++ b/lisp/org.el
>> @@ -14550,15 +14550,9 @@ Some of the options can be changed using
>> the variable
>> 	 (opt org-format-latex-options)
>> 	 (matchers (plist-get opt :matchers))
>> 	 (re-list org-latex-regexps)
>> -	 (cnt 0) txt link beg end re e checkdir
>> +	 (cnt 0) txt hash link beg end re e checkdir
>> 	 executables-checked
>> 	 m n block linkfile movefile ov)
>> -    ;; Check if there are old images files with this prefix, and
>> remove them
>> -    (when (file-directory-p todir)
>> -      (mapc 'delete-file
>> -	    (directory-files
>> -	     todir 'full
>> -	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
>>     ;; Check the different regular expressions
>>     (while (setq e (pop re-list))
>>       (setq m (car e) re (nth 1 e) n (nth 2 e)
>> @@ -14576,9 +14570,10 @@ Some of the options can be changed using
>> the variable
>> 	    (setq txt (match-string n)
>> 		  beg (match-beginning n) end (match-end n)
>> 		  cnt (1+ cnt)
>> -		  linkfile (format "%s_%04d.png" prefix cnt)
>> -		  movefile (format "%s_%04d.png" absprefix cnt)
>> 		  link (concat block "[[file:" linkfile "]]" block))
>> +            (setq hash (sha1 txt)
>> +		  linkfile (format "%s_%s.png" prefix hash)
>> +		  movefile (format "%s_%s.png" absprefix hash))
>> 	    (if msg (message msg cnt))
>> 	    (goto-char beg)
>> 	    (unless checkdir ; make sure the directory exists
>> @@ -14592,8 +14587,9 @@ Some of the options can be changed using the
>> variable
>> 	       "dvipng" "needed to convert LaTeX fragments to images")
>> 	      (setq executables-checked t))
>>
>> -	    (org-create-formula-image
>> -	     txt movefile opt forbuffer)
>> +            (unless (file-exists-p movefile)
>> +              (org-create-formula-image
>> +               txt movefile opt forbuffer))
>> 	    (if overlays
>> 		(progn
>> 		  (mapc (lambda (o)
>> --
>> 1.6.4.73.gc144
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
> - Carsten

[-- Attachment #4: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Hi Carsten,
>
> Thanks for the feedback, I have comments inline below
>
> Carsten Dominik <carsten.dominik@gmail.com> writes:

[...]

>> Now, I am sure that you are already planning to do the same
>> for ditaa images etc?
>
> of course :)

A first pass at a patch implementing caching of ditaa and dot images
generated by org-exp-blocks is attached.  It seems to work in all
initial tests, and it will only remove files which match the following
pattern

"beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"

such that the length of the part matched by \\([[:alnum:]]+\\) is 40
characters.

Best -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-use-sha1-hash-keys-to-cache-ditaa-and-dot-images-whe.patch --]
[-- Type: text/x-patch, Size: 5775 bytes --]

From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Mon, 16 Nov 2009 19:33:11 -0700
Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images when exporting through org-exp-blocks

---
 lisp/ChangeLog         |    6 ++++
 lisp/org-exp-blocks.el |   60 ++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 54 insertions(+), 12 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 5f83aaa..c2d44fa 100755
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,9 @@
+2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
+
+	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
+	hash keys to cache and re-use images generated by the
+	org-exp-blocks interface to ditaa and dot.
+
 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
 
 	* org-html.el (org-export-html-home/up-format): Add an ID to the
diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
index 71e1608..72fe6c4 100644
--- a/lisp/org-exp-blocks.el
+++ b/lisp/org-exp-blocks.el
@@ -217,9 +217,15 @@ Specify the path at which the image should be saved as the first
 element of headers, any additional elements of headers will be
 passed to the ditaa utility as command line arguments."
   (message "ditaa-formatting...")
-  (let ((out-file (if headers (car headers)))
-	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
-	(data-file (make-temp-file "org-ditaa")))
+  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
+         (data-file (make-temp-file "org-ditaa"))
+         (hash (sha1 (prin1-to-string (list body args))))
+         (raw-out-file (if headers (car headers)))
+         (out-file-parts (if (string-match "\\(.+\\)\\.\\([^\\.]+\\)$" raw-out-file)
+                             (cons (match-string 1 raw-out-file)
+                                   (match-string 2 raw-out-file))
+                           (cons raw-out-file "png")))
+         (out-file (concat (car out-file-parts) "_" hash "." (cdr out-file-parts))))
     (unless (file-exists-p org-ditaa-jar-path)
       (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-path)))
     (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
@@ -229,9 +235,21 @@ passed to the ditaa utility as command line arguments."
 			    "\n")))
     (cond
      ((or htmlp latexp docbookp)
-      (with-temp-file data-file (insert body))
-      (message (concat "java -jar " org-ditaa-jar-path " " args " " data-file " " out-file))
-      (shell-command (concat "java -jar " org-ditaa-jar-path " " args " " data-file " " out-file))
+      (unless (file-exists-p out-file)
+        (mapc ;; remove old hashed versions of this file
+         (lambda (file)
+           (when (and (string-match (concat (regexp-quote (car out-file-parts))
+                                            "_\\([[:alnum:]]+\\)\\."
+                                            (regexp-quote (cdr out-file-parts)))
+                                    file)
+                      (= (length (match-string 1 out-file)) 40))
+             (delete-file (expand-file-name file
+                                            (file-name-directory out-file)))))
+         (directory-files (or (file-name-directory out-file)
+                              default-directory)))
+        (with-temp-file data-file (insert body))
+        (message (concat "java -jar " org-ditaa-jar-path " " args " " data-file " " out-file))
+        (shell-command (concat "java -jar " org-ditaa-jar-path " " args " " data-file " " out-file)))
       (format "\n[[file:%s]]\n" out-file))
      (t (concat
 	 "\n#+BEGIN_EXAMPLE\n"
@@ -259,14 +277,32 @@ digraph data_relationships {
 }
 #+end_dot"
   (message "dot-formatting...")
-  (let ((out-file (if headers (car headers)))
-	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
-	(data-file (make-temp-file "org-ditaa")))
+  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
+         (data-file (make-temp-file "org-ditaa"))
+         (hash (sha1 (prin1-to-string (list body args))))
+         (raw-out-file (if headers (car headers)))
+         (out-file-parts (if (string-match "\\(.+\\)\\.\\([^\\.]+\\)$" raw-out-file)
+                             (cons (match-string 1 raw-out-file)
+                                   (match-string 2 raw-out-file))
+                           (cons raw-out-file "png")))
+         (out-file (concat (car out-file-parts) "_" hash "." (cdr out-file-parts))))
     (cond
      ((or htmlp latexp docbookp)
-      (with-temp-file data-file (insert body))
-      (message (concat "dot " data-file " " args " -o " out-file))
-      (shell-command (concat "dot " data-file " " args " -o " out-file))
+      (unless (file-exists-p out-file)
+        (mapc ;; remove old hashed versions of this file
+         (lambda (file)
+           (when (and (string-match (concat (regexp-quote (car out-file-parts))
+                                            "_\\([[:alnum:]]+\\)\\."
+                                            (regexp-quote (cdr out-file-parts)))
+                                    file)
+                      (= (length (match-string 1 out-file)) 40))
+             (delete-file (expand-file-name file
+                                            (file-name-directory out-file)))))
+         (directory-files (or (file-name-directory out-file)
+                              default-directory)))
+        (with-temp-file data-file (insert body))
+        (message (concat "dot " data-file " " args " -o " out-file))
+        (shell-command (concat "dot " data-file " " args " -o " out-file)))
       (format "\n[[file:%s]]\n" out-file))
      (t (concat
 	 "\n#+BEGIN_EXAMPLE\n"
-- 
1.6.4.73.gc144


[-- Attachment #3: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Carsten Dominik]

Wow, this is fantastic!

Do you think it is ready to be included (because you say first pass...)

- Carsten

On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Hi Carsten,
>>
>> Thanks for the feedback, I have comments inline below
>>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
> [...]
>
>>> Now, I am sure that you are already planning to do the same
>>> for ditaa images etc?
>>
>> of course :)
>
> A first pass at a patch implementing caching of ditaa and dot images
> generated by org-exp-blocks is attached.  It seems to work in all
> initial tests, and it will only remove files which match the following
> pattern
>
> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>
> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
> characters.
>
> Best -- Eric
>
> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00 2001
> From: Eric Schulte <schulte.eric@gmail.com>
> Date: Mon, 16 Nov 2009 19:33:11 -0700
> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images  
> when exporting through org-exp-blocks
>
> ---
> lisp/ChangeLog         |    6 ++++
> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++ 
> +---------
> 2 files changed, 54 insertions(+), 12 deletions(-)
>
> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
> index 5f83aaa..c2d44fa 100755
> --- a/lisp/ChangeLog
> +++ b/lisp/ChangeLog
> @@ -1,3 +1,9 @@
> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
> +
> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
> +	hash keys to cache and re-use images generated by the
> +	org-exp-blocks interface to ditaa and dot.
> +
> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>
> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
> index 71e1608..72fe6c4 100644
> --- a/lisp/org-exp-blocks.el
> +++ b/lisp/org-exp-blocks.el
> @@ -217,9 +217,15 @@ Specify the path at which the image should be  
> saved as the first
> element of headers, any additional elements of headers will be
> passed to the ditaa utility as command line arguments."
>   (message "ditaa-formatting...")
> -  (let ((out-file (if headers (car headers)))
> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
> -	(data-file (make-temp-file "org-ditaa")))
> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)  
> " ")))
> +         (data-file (make-temp-file "org-ditaa"))
> +         (hash (sha1 (prin1-to-string (list body args))))
> +         (raw-out-file (if headers (car headers)))
> +         (out-file-parts (if (string-match "\\(.+\\)\\.\\([^\\.]+\\) 
> $" raw-out-file)
> +                             (cons (match-string 1 raw-out-file)
> +                                   (match-string 2 raw-out-file))
> +                           (cons raw-out-file "png")))
> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr  
> out-file-parts))))
>     (unless (file-exists-p org-ditaa-jar-path)
>       (error (format "Could not find ditaa.jar at %s" org-ditaa-jar- 
> path)))
>     (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line  
> arguments."
> 			    "\n")))
>     (cond
>      ((or htmlp latexp docbookp)
> -      (with-temp-file data-file (insert body))
> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "  
> data-file " " out-file))
> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "  
> args " " data-file " " out-file))
> +      (unless (file-exists-p out-file)
> +        (mapc ;; remove old hashed versions of this file
> +         (lambda (file)
> +           (when (and (string-match (concat (regexp-quote (car out- 
> file-parts))
> +                                            "_\\([[:alnum:]]+\\)\\."
> +                                            (regexp-quote (cdr out- 
> file-parts)))
> +                                    file)
> +                      (= (length (match-string 1 out-file)) 40))
> +             (delete-file (expand-file-name file
> +                                            (file-name-directory  
> out-file)))))
> +         (directory-files (or (file-name-directory out-file)
> +                              default-directory)))
> +        (with-temp-file data-file (insert body))
> +        (message (concat "java -jar " org-ditaa-jar-path " " args "  
> " data-file " " out-file))
> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "  
> args " " data-file " " out-file)))
>       (format "\n[[file:%s]]\n" out-file))
>      (t (concat
> 	 "\n#+BEGIN_EXAMPLE\n"
> @@ -259,14 +277,32 @@ digraph data_relationships {
> }
> #+end_dot"
>   (message "dot-formatting...")
> -  (let ((out-file (if headers (car headers)))
> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
> -	(data-file (make-temp-file "org-ditaa")))
> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)  
> " ")))
> +         (data-file (make-temp-file "org-ditaa"))
> +         (hash (sha1 (prin1-to-string (list body args))))
> +         (raw-out-file (if headers (car headers)))
> +         (out-file-parts (if (string-match "\\(.+\\)\\.\\([^\\.]+\\) 
> $" raw-out-file)
> +                             (cons (match-string 1 raw-out-file)
> +                                   (match-string 2 raw-out-file))
> +                           (cons raw-out-file "png")))
> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr  
> out-file-parts))))
>     (cond
>      ((or htmlp latexp docbookp)
> -      (with-temp-file data-file (insert body))
> -      (message (concat "dot " data-file " " args " -o " out-file))
> -      (shell-command (concat "dot " data-file " " args " -o " out- 
> file))
> +      (unless (file-exists-p out-file)
> +        (mapc ;; remove old hashed versions of this file
> +         (lambda (file)
> +           (when (and (string-match (concat (regexp-quote (car out- 
> file-parts))
> +                                            "_\\([[:alnum:]]+\\)\\."
> +                                            (regexp-quote (cdr out- 
> file-parts)))
> +                                    file)
> +                      (= (length (match-string 1 out-file)) 40))
> +             (delete-file (expand-file-name file
> +                                            (file-name-directory  
> out-file)))))
> +         (directory-files (or (file-name-directory out-file)
> +                              default-directory)))
> +        (with-temp-file data-file (insert body))
> +        (message (concat "dot " data-file " " args " -o " out-file))
> +        (shell-command (concat "dot " data-file " " args " -o " out- 
> file)))
>       (format "\n[[file:%s]]\n" out-file))
>      (t (concat
> 	 "\n#+BEGIN_EXAMPLE\n"
> -- 
> 1.6.4.73.gc144
>

- Carsten

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Wow, this is fantastic!
>
> Do you think it is ready to be included (because you say first pass...)
>

Yes,

I said first pass because I had only done minimal testing.  However all
indications are that it works, and there are no further changes I would
like to make, so if it looks good to you I would say "yes", please apply
it.

Thanks -- Eric

>
> - Carsten
>
> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>
>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>
>>> Hi Carsten,
>>>
>>> Thanks for the feedback, I have comments inline below
>>>
>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>> [...]
>>
>>>> Now, I am sure that you are already planning to do the same
>>>> for ditaa images etc?
>>>
>>> of course :)
>>
>> A first pass at a patch implementing caching of ditaa and dot images
>> generated by org-exp-blocks is attached.  It seems to work in all
>> initial tests, and it will only remove files which match the following
>> pattern
>>
>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>
>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>> characters.
>>
>> Best -- Eric
>>
>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00 2001
>> From: Eric Schulte <schulte.eric@gmail.com>
>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>> when exporting through org-exp-blocks
>>
>> ---
>> lisp/ChangeLog         |    6 ++++
>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>> +---------
>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>
>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>> index 5f83aaa..c2d44fa 100755
>> --- a/lisp/ChangeLog
>> +++ b/lisp/ChangeLog
>> @@ -1,3 +1,9 @@
>> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
>> +
>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>> +	hash keys to cache and re-use images generated by the
>> +	org-exp-blocks interface to ditaa and dot.
>> +
>> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>>
>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>> index 71e1608..72fe6c4 100644
>> --- a/lisp/org-exp-blocks.el
>> +++ b/lisp/org-exp-blocks.el
>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>> saved as the first
>> element of headers, any additional elements of headers will be
>> passed to the ditaa utility as command line arguments."
>>   (message "ditaa-formatting...")
>> -  (let ((out-file (if headers (car headers)))
>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>> -	(data-file (make-temp-file "org-ditaa")))
>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>> " ")))
>> +         (data-file (make-temp-file "org-ditaa"))
>> +         (hash (sha1 (prin1-to-string (list body args))))
>> +         (raw-out-file (if headers (car headers)))
>> +         (out-file-parts (if (string-match
>> "\\(.+\\)\\.\\([^\\.]+\\)
>> $" raw-out-file)
>> +                             (cons (match-string 1 raw-out-file)
>> +                                   (match-string 2 raw-out-file))
>> +                           (cons raw-out-file "png")))
>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>> out-file-parts))))
>>     (unless (file-exists-p org-ditaa-jar-path)
>>       (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>> path)))
>>     (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>> arguments."
>> 			    "\n")))
>>     (cond
>>      ((or htmlp latexp docbookp)
>> -      (with-temp-file data-file (insert body))
>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>> data-file " " out-file))
>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>> args " " data-file " " out-file))
>> +      (unless (file-exists-p out-file)
>> +        (mapc ;; remove old hashed versions of this file
>> +         (lambda (file)
>> +           (when (and (string-match (concat (regexp-quote (car out-
>> file-parts))
>> +                                            "_\\([[:alnum:]]+\\)\\."
>> +                                            (regexp-quote (cdr out-
>> file-parts)))
>> +                                    file)
>> +                      (= (length (match-string 1 out-file)) 40))
>> +             (delete-file (expand-file-name file
>> +                                            (file-name-directory
>> out-file)))))
>> +         (directory-files (or (file-name-directory out-file)
>> +                              default-directory)))
>> +        (with-temp-file data-file (insert body))
>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>> " data-file " " out-file))
>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>> args " " data-file " " out-file)))
>>       (format "\n[[file:%s]]\n" out-file))
>>      (t (concat
>> 	 "\n#+BEGIN_EXAMPLE\n"
>> @@ -259,14 +277,32 @@ digraph data_relationships {
>> }
>> #+end_dot"
>>   (message "dot-formatting...")
>> -  (let ((out-file (if headers (car headers)))
>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>> -	(data-file (make-temp-file "org-ditaa")))
>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>> " ")))
>> +         (data-file (make-temp-file "org-ditaa"))
>> +         (hash (sha1 (prin1-to-string (list body args))))
>> +         (raw-out-file (if headers (car headers)))
>> +         (out-file-parts (if (string-match
>> "\\(.+\\)\\.\\([^\\.]+\\)
>> $" raw-out-file)
>> +                             (cons (match-string 1 raw-out-file)
>> +                                   (match-string 2 raw-out-file))
>> +                           (cons raw-out-file "png")))
>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>> out-file-parts))))
>>     (cond
>>      ((or htmlp latexp docbookp)
>> -      (with-temp-file data-file (insert body))
>> -      (message (concat "dot " data-file " " args " -o " out-file))
>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>> file))
>> +      (unless (file-exists-p out-file)
>> +        (mapc ;; remove old hashed versions of this file
>> +         (lambda (file)
>> +           (when (and (string-match (concat (regexp-quote (car out-
>> file-parts))
>> +                                            "_\\([[:alnum:]]+\\)\\."
>> +                                            (regexp-quote (cdr out-
>> file-parts)))
>> +                                    file)
>> +                      (= (length (match-string 1 out-file)) 40))
>> +             (delete-file (expand-file-name file
>> +                                            (file-name-directory
>> out-file)))))
>> +         (directory-files (or (file-name-directory out-file)
>> +                              default-directory)))
>> +        (with-temp-file data-file (insert body))
>> +        (message (concat "dot " data-file " " args " -o " out-file))
>> +        (shell-command (concat "dot " data-file " " args " -o "
>> out-
>> file)))
>>       (format "\n[[file:%s]]\n" out-file))
>>      (t (concat
>> 	 "\n#+BEGIN_EXAMPLE\n"
>> --
>> 1.6.4.73.gc144
>>
>
> - Carsten

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Carsten Dominik]

Hi Eric,

I had a problem while pushing, please verify that the patch got in  
correctly.  Thanks!

- Carsten

On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:

> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> Wow, this is fantastic!
>>
>> Do you think it is ready to be included (because you say first  
>> pass...)
>>
>
> Yes,
>
> I said first pass because I had only done minimal testing.  However  
> all
> indications are that it works, and there are no further changes I  
> would
> like to make, so if it looks good to you I would say "yes", please  
> apply
> it.
>
> Thanks -- Eric
>
>>
>> - Carsten
>>
>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>
>>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>>
>>>> Hi Carsten,
>>>>
>>>> Thanks for the feedback, I have comments inline below
>>>>
>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>
>>> [...]
>>>
>>>>> Now, I am sure that you are already planning to do the same
>>>>> for ditaa images etc?
>>>>
>>>> of course :)
>>>
>>> A first pass at a patch implementing caching of ditaa and dot images
>>> generated by org-exp-blocks is attached.  It seems to work in all
>>> initial tests, and it will only remove files which match the  
>>> following
>>> pattern
>>>
>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>
>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>> characters.
>>>
>>> Best -- Eric
>>>
>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00  
>>> 2001
>>> From: Eric Schulte <schulte.eric@gmail.com>
>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>> when exporting through org-exp-blocks
>>>
>>> ---
>>> lisp/ChangeLog         |    6 ++++
>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>> +---------
>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>> index 5f83aaa..c2d44fa 100755
>>> --- a/lisp/ChangeLog
>>> +++ b/lisp/ChangeLog
>>> @@ -1,3 +1,9 @@
>>> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
>>> +
>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>> +	hash keys to cache and re-use images generated by the
>>> +	org-exp-blocks interface to ditaa and dot.
>>> +
>>> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>>>
>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>> index 71e1608..72fe6c4 100644
>>> --- a/lisp/org-exp-blocks.el
>>> +++ b/lisp/org-exp-blocks.el
>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>> saved as the first
>>> element of headers, any additional elements of headers will be
>>> passed to the ditaa utility as command line arguments."
>>>  (message "ditaa-formatting...")
>>> -  (let ((out-file (if headers (car headers)))
>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>> -	(data-file (make-temp-file "org-ditaa")))
>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>> " ")))
>>> +         (data-file (make-temp-file "org-ditaa"))
>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>> +         (raw-out-file (if headers (car headers)))
>>> +         (out-file-parts (if (string-match
>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>> $" raw-out-file)
>>> +                             (cons (match-string 1 raw-out-file)
>>> +                                   (match-string 2 raw-out-file))
>>> +                           (cons raw-out-file "png")))
>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>> out-file-parts))))
>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>> path)))
>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>> arguments."
>>> 			    "\n")))
>>>    (cond
>>>     ((or htmlp latexp docbookp)
>>> -      (with-temp-file data-file (insert body))
>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>> data-file " " out-file))
>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>> args " " data-file " " out-file))
>>> +      (unless (file-exists-p out-file)
>>> +        (mapc ;; remove old hashed versions of this file
>>> +         (lambda (file)
>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>> file-parts))
>>> +                                            "_\\([[:alnum:]]+\\)\ 
>>> \."
>>> +                                            (regexp-quote (cdr out-
>>> file-parts)))
>>> +                                    file)
>>> +                      (= (length (match-string 1 out-file)) 40))
>>> +             (delete-file (expand-file-name file
>>> +                                            (file-name-directory
>>> out-file)))))
>>> +         (directory-files (or (file-name-directory out-file)
>>> +                              default-directory)))
>>> +        (with-temp-file data-file (insert body))
>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>> " data-file " " out-file))
>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>> args " " data-file " " out-file)))
>>>      (format "\n[[file:%s]]\n" out-file))
>>>     (t (concat
>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>> }
>>> #+end_dot"
>>>  (message "dot-formatting...")
>>> -  (let ((out-file (if headers (car headers)))
>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>> -	(data-file (make-temp-file "org-ditaa")))
>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>> " ")))
>>> +         (data-file (make-temp-file "org-ditaa"))
>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>> +         (raw-out-file (if headers (car headers)))
>>> +         (out-file-parts (if (string-match
>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>> $" raw-out-file)
>>> +                             (cons (match-string 1 raw-out-file)
>>> +                                   (match-string 2 raw-out-file))
>>> +                           (cons raw-out-file "png")))
>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>> out-file-parts))))
>>>    (cond
>>>     ((or htmlp latexp docbookp)
>>> -      (with-temp-file data-file (insert body))
>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>> file))
>>> +      (unless (file-exists-p out-file)
>>> +        (mapc ;; remove old hashed versions of this file
>>> +         (lambda (file)
>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>> file-parts))
>>> +                                            "_\\([[:alnum:]]+\\)\ 
>>> \."
>>> +                                            (regexp-quote (cdr out-
>>> file-parts)))
>>> +                                    file)
>>> +                      (= (length (match-string 1 out-file)) 40))
>>> +             (delete-file (expand-file-name file
>>> +                                            (file-name-directory
>>> out-file)))))
>>> +         (directory-files (or (file-name-directory out-file)
>>> +                              default-directory)))
>>> +        (with-temp-file data-file (insert body))
>>> +        (message (concat "dot " data-file " " args " -o " out- 
>>> file))
>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>> out-
>>> file)))
>>>      (format "\n[[file:%s]]\n" out-file))
>>>     (t (concat
>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>> --
>>> 1.6.4.73.gc144
>>>
>>
>> - Carsten

- Carsten

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

Hi Carsten,

I just pulled, reloaded, and re-ran my simple tests and the patch
appears to have been applied successfully.

Thanks -- Eric

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Hi Eric,
>
> I had a problem while pushing, please verify that the patch got in
> correctly.  Thanks!
>
> - Carsten
>
> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:
>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Wow, this is fantastic!
>>>
>>> Do you think it is ready to be included (because you say first
>>> pass...)
>>>
>>
>> Yes,
>>
>> I said first pass because I had only done minimal testing.  However
>> all
>> indications are that it works, and there are no further changes I
>> would
>> like to make, so if it looks good to you I would say "yes", please
>> apply
>> it.
>>
>> Thanks -- Eric
>>
>>>
>>> - Carsten
>>>
>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>>
>>>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>>>
>>>>> Hi Carsten,
>>>>>
>>>>> Thanks for the feedback, I have comments inline below
>>>>>
>>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>>
>>>> [...]
>>>>
>>>>>> Now, I am sure that you are already planning to do the same
>>>>>> for ditaa images etc?
>>>>>
>>>>> of course :)
>>>>
>>>> A first pass at a patch implementing caching of ditaa and dot images
>>>> generated by org-exp-blocks is attached.  It seems to work in all
>>>> initial tests, and it will only remove files which match the
>>>> following
>>>> pattern
>>>>
>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>>
>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>>> characters.
>>>>
>>>> Best -- Eric
>>>>
>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00
>>>> 2001
>>>> From: Eric Schulte <schulte.eric@gmail.com>
>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>>> when exporting through org-exp-blocks
>>>>
>>>> ---
>>>> lisp/ChangeLog         |    6 ++++
>>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>>> +---------
>>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>>
>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>>> index 5f83aaa..c2d44fa 100755
>>>> --- a/lisp/ChangeLog
>>>> +++ b/lisp/ChangeLog
>>>> @@ -1,3 +1,9 @@
>>>> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
>>>> +
>>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>>> +	hash keys to cache and re-use images generated by the
>>>> +	org-exp-blocks interface to ditaa and dot.
>>>> +
>>>> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>>>>
>>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>>> index 71e1608..72fe6c4 100644
>>>> --- a/lisp/org-exp-blocks.el
>>>> +++ b/lisp/org-exp-blocks.el
>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>>> saved as the first
>>>> element of headers, any additional elements of headers will be
>>>> passed to the ditaa utility as command line arguments."
>>>>  (message "ditaa-formatting...")
>>>> -  (let ((out-file (if headers (car headers)))
>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>> " ")))
>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>> +         (raw-out-file (if headers (car headers)))
>>>> +         (out-file-parts (if (string-match
>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>> $" raw-out-file)
>>>> +                             (cons (match-string 1 raw-out-file)
>>>> +                                   (match-string 2 raw-out-file))
>>>> +                           (cons raw-out-file "png")))
>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>> out-file-parts))))
>>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>>> path)))
>>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>>> arguments."
>>>> 			    "\n")))
>>>>    (cond
>>>>     ((or htmlp latexp docbookp)
>>>> -      (with-temp-file data-file (insert body))
>>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>>> data-file " " out-file))
>>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>> args " " data-file " " out-file))
>>>> +      (unless (file-exists-p out-file)
>>>> +        (mapc ;; remove old hashed versions of this file
>>>> +         (lambda (file)
>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>> file-parts))
>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>> \."
>>>> +                                            (regexp-quote (cdr out-
>>>> file-parts)))
>>>> +                                    file)
>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>> +             (delete-file (expand-file-name file
>>>> +                                            (file-name-directory
>>>> out-file)))))
>>>> +         (directory-files (or (file-name-directory out-file)
>>>> +                              default-directory)))
>>>> +        (with-temp-file data-file (insert body))
>>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>>> " data-file " " out-file))
>>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>> args " " data-file " " out-file)))
>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>     (t (concat
>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>>> }
>>>> #+end_dot"
>>>>  (message "dot-formatting...")
>>>> -  (let ((out-file (if headers (car headers)))
>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>> " ")))
>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>> +         (raw-out-file (if headers (car headers)))
>>>> +         (out-file-parts (if (string-match
>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>> $" raw-out-file)
>>>> +                             (cons (match-string 1 raw-out-file)
>>>> +                                   (match-string 2 raw-out-file))
>>>> +                           (cons raw-out-file "png")))
>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>> out-file-parts))))
>>>>    (cond
>>>>     ((or htmlp latexp docbookp)
>>>> -      (with-temp-file data-file (insert body))
>>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>>> file))
>>>> +      (unless (file-exists-p out-file)
>>>> +        (mapc ;; remove old hashed versions of this file
>>>> +         (lambda (file)
>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>> file-parts))
>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>> \."
>>>> +                                            (regexp-quote (cdr out-
>>>> file-parts)))
>>>> +                                    file)
>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>> +             (delete-file (expand-file-name file
>>>> +                                            (file-name-directory
>>>> out-file)))))
>>>> +         (directory-files (or (file-name-directory out-file)
>>>> +                              default-directory)))
>>>> +        (with-temp-file data-file (insert body))
>>>> +        (message (concat "dot " data-file " " args " -o " out-
>>>> file))
>>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>>> out-
>>>> file)))
>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>     (t (concat
>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>> --
>>>> 1.6.4.73.gc144
>>>>
>>>
>>> - Carsten
>
> - Carsten

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Francesco Pizzolante]

Hi,

> I just pulled, reloaded, and re-ran my simple tests and the patch
> appears to have been applied successfully.

This idea of caching images is really great and works well.

Nevertheless, I have a remark about the way it is implemented.

When I write the following code:

--8<---------------cut here---------------start------------->8---
#+BEGIN_ditaa procedure.png
--8<---------------cut here---------------end--------------->8---

I expect that:

- the system generates a `procedure.png' file in order for me to check it in
  the repository;

- that the `procedure.png' file is referenced everywhere (i.e. in the exported
  HTML code or LaTeX code).

The current way it works is:

1. The `procedure_XXX.png' is committed to the repository;

2. I alter my ditaa code, and export the new document;

3. The `procedure_XXX.png' file has been removed from my working copy and a
   new file `procedure_YYY.png' has been added in my working copy.

Thus, in order to commit the new files as they are, I need to:

1. revert the `procedure_XXX.png' file, in order to delete it afterward;

2. add the `procedure_YYY.png' file to the repository.

A nice solution would have been that all ditaa exports are done under the
filename `procedure.png', and that empty files such as
`procedure.png.ZZZ.cache' are created to "store" the fact that they have
already been generated.

That way, we even can ignore all the `*.cache' files from our working copy,
when checking for updates.

What do you think?

Francesco

>
> Thanks -- Eric
>
> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> Hi Eric,
>>
>> I had a problem while pushing, please verify that the patch got in
>> correctly.  Thanks!
>>
>> - Carsten
>>
>> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:
>>
>>> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>
>>>> Wow, this is fantastic!
>>>>
>>>> Do you think it is ready to be included (because you say first
>>>> pass...)
>>>>
>>>
>>> Yes,
>>>
>>> I said first pass because I had only done minimal testing.  However
>>> all
>>> indications are that it works, and there are no further changes I
>>> would
>>> like to make, so if it looks good to you I would say "yes", please
>>> apply
>>> it.
>>>
>>> Thanks -- Eric
>>>
>>>>
>>>> - Carsten
>>>>
>>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>>>
>>>>> "Eric Schulte" <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>>>
>>>>>> Hi Carsten,
>>>>>>
>>>>>> Thanks for the feedback, I have comments inline below
>>>>>>
>>>>>> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>>> Now, I am sure that you are already planning to do the same
>>>>>>> for ditaa images etc?
>>>>>>
>>>>>> of course :)
>>>>>
>>>>> A first pass at a patch implementing caching of ditaa and dot images
>>>>> generated by org-exp-blocks is attached.  It seems to work in all
>>>>> initial tests, and it will only remove files which match the
>>>>> following
>>>>> pattern
>>>>>
>>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>>>
>>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>>>> characters.
>>>>>
>>>>> Best -- Eric
>>>>>
>>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00
>>>>> 2001
>>>>> From: Eric Schulte <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>>>> when exporting through org-exp-blocks
>>>>>
>>>>> ---
>>>>> lisp/ChangeLog         |    6 ++++
>>>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>>>> +---------
>>>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>>>
>>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>>>> index 5f83aaa..c2d44fa 100755
>>>>> --- a/lisp/ChangeLog
>>>>> +++ b/lisp/ChangeLog
>>>>> @@ -1,3 +1,9 @@
>>>>> +2009-11-17  Eric Schulte  <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>> +
>>>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>>>> +	hash keys to cache and re-use images generated by the
>>>>> +	org-exp-blocks interface to ditaa and dot.
>>>>> +
>>>>> 2009-11-16  Carsten Dominik  <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>>
>>>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>>>> index 71e1608..72fe6c4 100644
>>>>> --- a/lisp/org-exp-blocks.el
>>>>> +++ b/lisp/org-exp-blocks.el
>>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>>>> saved as the first
>>>>> element of headers, any additional elements of headers will be
>>>>> passed to the ditaa utility as command line arguments."
>>>>>  (message "ditaa-formatting...")
>>>>> -  (let ((out-file (if headers (car headers)))
>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>> " ")))
>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>> +         (raw-out-file (if headers (car headers)))
>>>>> +         (out-file-parts (if (string-match
>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>> $" raw-out-file)
>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>> +                                   (match-string 2 raw-out-file))
>>>>> +                           (cons raw-out-file "png")))
>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>> out-file-parts))))
>>>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>>>> path)))
>>>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>>>> arguments."
>>>>> 			    "\n")))
>>>>>    (cond
>>>>>     ((or htmlp latexp docbookp)
>>>>> -      (with-temp-file data-file (insert body))
>>>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>>>> data-file " " out-file))
>>>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>> args " " data-file " " out-file))
>>>>> +      (unless (file-exists-p out-file)
>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>> +         (lambda (file)
>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>> file-parts))
>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>> \."
>>>>> +                                            (regexp-quote (cdr out-
>>>>> file-parts)))
>>>>> +                                    file)
>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>> +             (delete-file (expand-file-name file
>>>>> +                                            (file-name-directory
>>>>> out-file)))))
>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>> +                              default-directory)))
>>>>> +        (with-temp-file data-file (insert body))
>>>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>>>> " data-file " " out-file))
>>>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>> args " " data-file " " out-file)))
>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>     (t (concat
>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>>>> }
>>>>> #+end_dot"
>>>>>  (message "dot-formatting...")
>>>>> -  (let ((out-file (if headers (car headers)))
>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>> " ")))
>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>> +         (raw-out-file (if headers (car headers)))
>>>>> +         (out-file-parts (if (string-match
>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>> $" raw-out-file)
>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>> +                                   (match-string 2 raw-out-file))
>>>>> +                           (cons raw-out-file "png")))
>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>> out-file-parts))))
>>>>>    (cond
>>>>>     ((or htmlp latexp docbookp)
>>>>> -      (with-temp-file data-file (insert body))
>>>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>>>> file))
>>>>> +      (unless (file-exists-p out-file)
>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>> +         (lambda (file)
>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>> file-parts))
>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>> \."
>>>>> +                                            (regexp-quote (cdr out-
>>>>> file-parts)))
>>>>> +                                    file)
>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>> +             (delete-file (expand-file-name file
>>>>> +                                            (file-name-directory
>>>>> out-file)))))
>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>> +                              default-directory)))
>>>>> +        (with-temp-file data-file (insert body))
>>>>> +        (message (concat "dot " data-file " " args " -o " out-
>>>>> file))
>>>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>>>> out-
>>>>> file)))
>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>     (t (concat
>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>> --
>>>>> 1.6.4.73.gc144
>>>>>
>>>>
>>>> - Carsten
>>
>> - Carsten
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode-mXXj517/zsQ@public.gmane.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

-- 
________________________________________________
Francesco Pizzolante - fpz-djc/iPCCuDYQheJpep6IedvLeJWuRmrY@public.gmane.org
M i s s i o n   C r i t i c a l   I T
Phone ... +32 2-757.10.15
Fax ..... +32 2-759.27.60


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

Hi Francesco,

You raise good points below, and I am not sure how best to respond to
them.  My initial reaction is that you should not be checking
automatically generated files (e.g. the results of ditaa exports) into a
version control repository, however I understand that there are times
when such measures are required.

Would it be possible to switch from using org-exp-blocks to using
org-babel?  If so then you could use org-babel's caching which does
*not* affect the exported file name, but rather saves a sha1 key as
(mostly) hidden text in the org-mode buffer.

so for example

#+BEGIN_ditaa procedure.png
     +-----------------+
     |     Example     |
     |                 |
     |   +---------+   |
     |   |         |   |
     |   |         |   |
     |   +---------+   |
     +-----------------+
#+END_ditaa

would be written as

#+begin_src ditaa :file procedure.png :cache :exports results
     +-----------------+
     |     Example     |
     |                 |
     |   +---------+   |
     |   |         |   |
     |   |         |   |
     |   +---------+   |
     +-----------------+
#+end_src

and pressing C-cC-c on the block (or exporting) would result in the
insertion of a link to the resulting image into the org-mode buffer
behind a results line as follows -- only in org most of the hash is
hidden.

#+results[bdffac60833c9f925a52bd6617dace39832b7dda]:
[[file:procedure.png]]

My problem with your proposed solution is that it entails storing
information *outside* of the org-mode buffer -- which granted we are
already doing although in a less dramatic way.  The creation of external
files whose sole purpose is to save a hash in their filename strikes me
as wasteful/messy.

Best -- Eric

Francesco Pizzolante <fpz@missioncriticalit.com> writes:

> Hi,
>
>> I just pulled, reloaded, and re-ran my simple tests and the patch
>> appears to have been applied successfully.
>
> This idea of caching images is really great and works well.
>
> Nevertheless, I have a remark about the way it is implemented.
>
> When I write the following code:
>
> #+BEGIN_ditaa procedure.png
>
> I expect that:
>
> - the system generates a `procedure.png' file in order for me to check it in
>   the repository;
>
> - that the `procedure.png' file is referenced everywhere (i.e. in the exported
>   HTML code or LaTeX code).
>
> The current way it works is:
>
> 1. The `procedure_XXX.png' is committed to the repository;
>
> 2. I alter my ditaa code, and export the new document;
>
> 3. The `procedure_XXX.png' file has been removed from my working copy and a
>    new file `procedure_YYY.png' has been added in my working copy.
>
> Thus, in order to commit the new files as they are, I need to:
>
> 1. revert the `procedure_XXX.png' file, in order to delete it afterward;
>
> 2. add the `procedure_YYY.png' file to the repository.
>
> A nice solution would have been that all ditaa exports are done under the
> filename `procedure.png', and that empty files such as
> `procedure.png.ZZZ.cache' are created to "store" the fact that they have
> already been generated.
>
> That way, we even can ignore all the `*.cache' files from our working copy,
> when checking for updates.
>
> What do you think?
>
> Francesco
>
>>
>> Thanks -- Eric
>>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Hi Eric,
>>>
>>> I had a problem while pushing, please verify that the patch got in
>>> correctly.  Thanks!
>>>
>>> - Carsten
>>>
>>> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:
>>>
>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>>
>>>>> Wow, this is fantastic!
>>>>>
>>>>> Do you think it is ready to be included (because you say first
>>>>> pass...)
>>>>>
>>>>
>>>> Yes,
>>>>
>>>> I said first pass because I had only done minimal testing.  However
>>>> all
>>>> indications are that it works, and there are no further changes I
>>>> would
>>>> like to make, so if it looks good to you I would say "yes", please
>>>> apply
>>>> it.
>>>>
>>>> Thanks -- Eric
>>>>
>>>>>
>>>>> - Carsten
>>>>>
>>>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>>>>
>>>>>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>>>>>
>>>>>>> Hi Carsten,
>>>>>>>
>>>>>>> Thanks for the feedback, I have comments inline below
>>>>>>>
>>>>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>>> Now, I am sure that you are already planning to do the same
>>>>>>>> for ditaa images etc?
>>>>>>>
>>>>>>> of course :)
>>>>>>
>>>>>> A first pass at a patch implementing caching of ditaa and dot images
>>>>>> generated by org-exp-blocks is attached.  It seems to work in all
>>>>>> initial tests, and it will only remove files which match the
>>>>>> following
>>>>>> pattern
>>>>>>
>>>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>>>>
>>>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>>>>> characters.
>>>>>>
>>>>>> Best -- Eric
>>>>>>
>>>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00
>>>>>> 2001
>>>>>> From: Eric Schulte <schulte.eric@gmail.com>
>>>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>>>>> when exporting through org-exp-blocks
>>>>>>
>>>>>> ---
>>>>>> lisp/ChangeLog         |    6 ++++
>>>>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>>>>> +---------
>>>>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>>>>
>>>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>>>>> index 5f83aaa..c2d44fa 100755
>>>>>> --- a/lisp/ChangeLog
>>>>>> +++ b/lisp/ChangeLog
>>>>>> @@ -1,3 +1,9 @@
>>>>>> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
>>>>>> +
>>>>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>>>>> +	hash keys to cache and re-use images generated by the
>>>>>> +	org-exp-blocks interface to ditaa and dot.
>>>>>> +
>>>>>> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>>>>>>
>>>>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>>>>> index 71e1608..72fe6c4 100644
>>>>>> --- a/lisp/org-exp-blocks.el
>>>>>> +++ b/lisp/org-exp-blocks.el
>>>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>>>>> saved as the first
>>>>>> element of headers, any additional elements of headers will be
>>>>>> passed to the ditaa utility as command line arguments."
>>>>>>  (message "ditaa-formatting...")
>>>>>> -  (let ((out-file (if headers (car headers)))
>>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>>> " ")))
>>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>>> +         (raw-out-file (if headers (car headers)))
>>>>>> +         (out-file-parts (if (string-match
>>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>>> $" raw-out-file)
>>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>>> +                                   (match-string 2 raw-out-file))
>>>>>> +                           (cons raw-out-file "png")))
>>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>>> out-file-parts))))
>>>>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>>>>> path)))
>>>>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>>>>> arguments."
>>>>>> 			    "\n")))
>>>>>>    (cond
>>>>>>     ((or htmlp latexp docbookp)
>>>>>> -      (with-temp-file data-file (insert body))
>>>>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>>>>> data-file " " out-file))
>>>>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>>> args " " data-file " " out-file))
>>>>>> +      (unless (file-exists-p out-file)
>>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>>> +         (lambda (file)
>>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>>> file-parts))
>>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>>> \."
>>>>>> +                                            (regexp-quote (cdr out-
>>>>>> file-parts)))
>>>>>> +                                    file)
>>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>>> +             (delete-file (expand-file-name file
>>>>>> +                                            (file-name-directory
>>>>>> out-file)))))
>>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>>> +                              default-directory)))
>>>>>> +        (with-temp-file data-file (insert body))
>>>>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>>>>> " data-file " " out-file))
>>>>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>>> args " " data-file " " out-file)))
>>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>>     (t (concat
>>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>>>>> }
>>>>>> #+end_dot"
>>>>>>  (message "dot-formatting...")
>>>>>> -  (let ((out-file (if headers (car headers)))
>>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>>> " ")))
>>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>>> +         (raw-out-file (if headers (car headers)))
>>>>>> +         (out-file-parts (if (string-match
>>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>>> $" raw-out-file)
>>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>>> +                                   (match-string 2 raw-out-file))
>>>>>> +                           (cons raw-out-file "png")))
>>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>>> out-file-parts))))
>>>>>>    (cond
>>>>>>     ((or htmlp latexp docbookp)
>>>>>> -      (with-temp-file data-file (insert body))
>>>>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>>>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>>>>> file))
>>>>>> +      (unless (file-exists-p out-file)
>>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>>> +         (lambda (file)
>>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>>> file-parts))
>>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>>> \."
>>>>>> +                                            (regexp-quote (cdr out-
>>>>>> file-parts)))
>>>>>> +                                    file)
>>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>>> +             (delete-file (expand-file-name file
>>>>>> +                                            (file-name-directory
>>>>>> out-file)))))
>>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>>> +                              default-directory)))
>>>>>> +        (with-temp-file data-file (insert body))
>>>>>> +        (message (concat "dot " data-file " " args " -o " out-
>>>>>> file))
>>>>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>>>>> out-
>>>>>> file)))
>>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>>     (t (concat
>>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>>> --
>>>>>> 1.6.4.73.gc144
>>>>>>
>>>>>
>>>>> - Carsten
>>>
>>> - Carsten
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Francesco Pizzolante]

Eric,

> Would it be possible to switch from using org-exp-blocks to using
> org-babel?  If so then you could use org-babel's caching which does
> *not* affect the exported file name, but rather saves a sha1 key as
> (mostly) hidden text in the org-mode buffer.
>
> so for example
>
> #+BEGIN_ditaa procedure.png
>      +-----------------+
>      |     Example     |
>      |                 |
>      |   +---------+   |
>      |   |         |   |
>      |   |         |   |
>      |   +---------+   |
>      +-----------------+
> #+END_ditaa
>
> would be written as
>
> #+begin_src ditaa :file procedure.png :cache :exports results
>      +-----------------+
>      |     Example     |
>      |                 |
>      |   +---------+   |
>      |   |         |   |
>      |   |         |   |
>      |   +---------+   |
>      +-----------------+
> #+end_src
>
> and pressing C-cC-c on the block (or exporting) would result in the
> insertion of a link to the resulting image into the org-mode buffer
> behind a results line as follows -- only in org most of the hash is
> hidden.
>
> #+results[bdffac60833c9f925a52bd6617dace39832b7dda]:
> [[file:procedure.png]]

I don't see any problem in using org-babel. The only issues I see with this
solution are the following:

- when you export, no #+results line is added to the org-mode buffer;

- you, thus, have to do the C-cC-c at least once in your block to generate
  the #+results line;

- when you export (to LaTeX or PDF), the image is always re-computed (no
  caching mechanism is used), while doing C-cC-c, the caching is used.

Can you confirm these points?


> My problem with your proposed solution is that it entails storing
> information *outside* of the org-mode buffer -- which granted we are
> already doing although in a less dramatic way.  The creation of external
> files whose sole purpose is to save a hash in their filename strikes me
> as wasteful/messy.

I agree with your point. I was "simply" trying to reuse what has been done for
the caching of images in org. The org-babel solution is more elegant.

Thanks,
F.


_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Eric Schulte]

Hi Francesco,

Francesco Pizzolante <fpz@missioncriticalit.com> writes:

>>
>> and pressing C-cC-c on the block (or exporting) would result in the
>> insertion of a link to the resulting image into the org-mode buffer
>> behind a results line as follows -- only in org most of the hash is
>> hidden.
>>
>> #+results[bdffac60833c9f925a52bd6617dace39832b7dda]:
>> [[file:procedure.png]]
>
> I don't see any problem in using org-babel. The only issues I see with this
> solution are the following:
>
> - when you export, no #+results line is added to the org-mode buffer;
>
> - you, thus, have to do the C-cC-c at least once in your block to generate
>   the #+results line;
>

Yes, this is true, and this is how I generally do development with
Org-babel.  In most cases I'd rather generate the image with C-cC-c and
preview the image with C-cC-o before exporting the document.  My most
common setup is to construct source-code blocks like the following

--8<---------------cut here---------------start------------->8---
#+begin_src ditaa :file data/example.png :exports none
    +------------------+
    |  ditaa example   |
    |                  |
    |                  |
    +------------------+
#+end_src

#+results:
[[file:data/example.png]]
--8<---------------cut here---------------end--------------->8---

in which case the caching isn't even used as *no* action is taken on the
source-code block during export

>
> - when you export (to LaTeX or PDF), the image is always re-computed (no
>   caching mechanism is used), while doing C-cC-c, the caching is used.
>
> Can you confirm these points?
>

Yes, these points are both very valid.  Exporting should not change the
buffer being exported, however, that leaves caching no way to save it's
results aside from the somewhat hackey out-of-file solutions used
elsewhere.  When we were initially working on caching Dan had a solution
which saved cache information in memory which allowed caching on export
without any file/buffer changes but which saved information during the
current Emacs session.  I believe he also worked out a caching system
which stored this same information in a single system-wide file.  If
there is enough interest perhaps it would be worth resurrecting these
ideas.

Thanks -- Eric

Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

[Carsten Dominik]

Hi Eric,

looks great now, I have made a few minor changes and applied it.

- Carsten

On Nov 17, 2009, at 1:11 AM, Eric Schulte wrote:

> Delivered-To: carsten.dominik@gmail.com
> Received: by 10.90.33.18 with SMTP id g18cs184746agg;
>         Mon, 16 Nov 2009 16:14:16 -0800 (PST)
> Received: by 10.115.103.17 with SMTP id f17mr8915518wam. 
> 166.1258416855542;
>         Mon, 16 Nov 2009 16:14:15 -0800 (PST)
> Return-Path: <schulte.eric@gmail.com>
> Received: from mail-pz0-f194.google.com (mail-pz0-f194.google.com  
> [209.85.222.194])
>         by mx.google.com with ESMTP id 32si16386502pzk. 
> 110.2009.11.16.16.14.14;
>         Mon, 16 Nov 2009 16:14:14 -0800 (PST)
> Received-SPF: pass (google.com: domain of schulte.eric@gmail.com  
> designates 209.85.222.194 as permitted sender) client- 
> ip=209.85.222.194;
> Authentication-Results: mx.google.com; spf=pass (google.com: domain  
> of schulte.eric@gmail.com designates 209.85.222.194 as permitted  
> sender) smtp.mail=schulte.eric@gmail.com; dkim=pass (test mode) header.i=@gmail.com
> Received: by mail-pz0-f194.google.com with SMTP id 32so4183666pzk.21
>         for <carsten.dominik@gmail.com>; Mon, 16 Nov 2009 16:14:14  
> -0800 (PST)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>         d=gmail.com; s=gamma;
>         h=domainkey- 
> signature:received:received:from:to:cc:subject:date
>          :references:message-id:user-agent:mime-version:content-type;
>         bh=ljxQthw1QhSpvXDshKqdl0Hmi2nMWV522o9FyWQIilY=;
>         b=JWQ//xkTNTI4hck3U/DCNEnBYADht03DHIfHIpu/O3sUVCX7vECFDVV/ 
> YiboCVdziZ
>          R4Uy6vQO2/PIB 
> +m5VhNXtx9xQoVrZMZCkfsoNjXtg5iWUzvKPon0sP9Hu7x7iC48+bc3
>          nHT82nwLQxD8AfjPRnrHWxVJE0V6PeFBl2zrk=
> DomainKey-Signature: a=rsa-sha1; c=nofws;
>         d=gmail.com; s=gamma;
>         h=from:to:cc:subject:date:references:message-id:user-agent
>          :mime-version:content-type;
>         b=PsUXGek+vgAXULkt/6iP9BZQVaBqpCb8cB8bPp8suG4lT2ZAdTHti3K/ 
> QKt3ZKlUrp
>          uVYHXPt1lustTNapWXvGPCK269E9xLkzU0fiFtyE8InqF 
> +tOn86drUHSbDmSFC5hh3uJ
>          sXgMAXWAMMe7J1y89K1H/NdV61cXAm/AOclC4=
> Received: by 10.115.101.18 with SMTP id d18mr8602604wam. 
> 191.1258416853669;
>         Mon, 16 Nov 2009 16:14:13 -0800 (PST)
> Return-Path: <schulte.eric@gmail.com>
> Received: from eschulte (adaptive.cs.unm.edu [64.106.21.179])
>         by mx.google.com with ESMTPS id 23sm1871553pxi. 
> 1.2009.11.16.16.14.11
>         (version=TLSv1/SSLv3 cipher=RC4-MD5);
>         Mon, 16 Nov 2009 16:14:12 -0800 (PST)
> From: "Eric Schulte" <schulte.eric@gmail.com>
> To: Carsten Dominik <carsten.dominik@gmail.com>
> Cc: Org Mode <emacs-orgmode@gnu.org>
> Subject: Re: [Orgmode] [PATCH] sha1 hash of latex fragments to avoid  
> regeneration
> Date: Mon, 16 Nov 2009 17:11:03 -0700
> References: <m2my2n8h8i.fsf@gmail.com>
> 	<FD51078D-BA1E-40F5-ABC7-4D226A336F53@gmail.com>
> Message-ID: <m21vjy80tq.fsf@gmail.com>
> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (darwin)
> MIME-Version: 1.0
> Content-Type: multipart/mixed; boundary="=-=-="
>
> --=-=-=
>
> Hi Carsten,
>
> Thanks for the feedback, I have comments inline below
>
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
> > Hi Eric,
> >
> > this is fantastic, thank you for implementing it.  I have wanted  
> some
> > speedup
> > for this for a long time.
> >
> > I think your implementation still suffers from one issue:
> >
> > The produced image also depends on the variables org-format-latex-
> > options,
> > org-format-latex-header, org-export-latex-package-alist,
> > and on the `forbuffer' flag (because images made for display in
> > the buffer and fo HTML export generaly need different resolution).
> >
> > One way to deal with this would be to make a list containing the  
> values
> > of these four variables and using prin1-to-string to convert this  
> list
> > into a string, and then to prepend this string to TXT when creating
> > the hash.
> >
>
> That sounds like the best solution.  I have made this change in the
> newly attached patch.
>
> >
> > Now, I am sure that you are already planning to do the same
> > for ditaa images etc?
>
> of course :)
>
> > That would be a treat, because ditaa can be terribly slow for  
> complex
> > figures, and this would speed up the cycle when writing document by
> > quite a bit.
> >
>
> Dan and I have been working on general caching solution for org-babel.
> Once we get that sorted out it should provide for the caching of all
> org-babel results which would include ditaa, dot, gnuplot, etc...
>
> I am currently more interested in making these changes in org-babel  
> than
> in org-exp-blocks, but in this case it may be worth implementing  
> caching
> in both cases.
>
> >
> > There is one further issue:  Cleaning up images that are no
> > longer used.
> >
> > With the LaTeX fragments it is not a big problem, because there
> > live in a special directory.  This would be a bigger concern for
> > ditaa images etc which tend to live in the same directory as the
> > source.  Maybe that could be solved by
> >
> > 1. Making sure that each image still have a name like "blue", so
> >    that the name now would be "blus_loooooonghashvalue.png" or so.
> > 2. Maybe creating a command that will look for orphaned images
> >    and remove them, by looking for the hash in the name and
> >    checking access times.  I am not sure if this is needed,
> >    and not sure what would be the best way to implement it.
> >
>
> Yes, this will not be an issue in the org-babel implementation as the
> hash key is stored separate from the file name, but I can see how this
> would need to be considered for any org-exp-blocks hash-based image
> caching.  The first option you propose above sounds very doable, as  
> long
> as we are comfortable removing any files that match regular  
> expressions
> like
>
>   blue_[[:alnum:]]+\.png
>
> which seems safe enough...
>
> >
> > After looking at these things, I would be *very* happy to accept
> > this patch.
> >
>
> I'll give this some more thought, but perhaps the latex image fragment
> patch is now viable.
>
> Best -- Eric
>
>
> --=-=-=
> Content-Type: text/x-patch
> Content-Disposition: inline;
>  filename=0001-latex-fragment-images-cached-using-sha1-hash-keys.patch
>
> From 0e9a359c1d5e8f67c20066533171fb1edc11ba61 Mon Sep 17 00:00:00 2001
> From: Eric Schulte <schulte.eric@gmail.com>
> Date: Mon, 16 Nov 2009 16:53:34 -0700
> Subject: [PATCH] latex fragment images cached using sha1 hash keys
>
>   Latex fragment images are now saved in files named by the sha1 hash
>   of the latex text used to create the image.  By checking if files
>   exist before images generation the regeneration of identical latex
>   images is avoided.
> ---
>  lisp/ChangeLog |    6 ++++++
>  lisp/org.el    |   22 +++++++++++-----------
>  2 files changed, 17 insertions(+), 11 deletions(-)
>
> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
> index 5f83aaa..b581931 100755
> --- a/lisp/ChangeLog
> +++ b/lisp/ChangeLog
> @@ -1,3 +1,9 @@
> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
> +
> +	* org.el (org-format-latex): Latex images are now saved to files
> +	named by the sha1 hash of the latex source text avoiding
> +	regeneration of identical images.
> +
>  2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>
>  	* org-html.el (org-export-html-home/up-format): Add an ID to the
> diff --git a/lisp/org.el b/lisp/org.el
> index bf6573b..15a8f9e 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -14550,15 +14550,9 @@ Some of the options can be changed using  
> the variable
>  	 (opt org-format-latex-options)
>  	 (matchers (plist-get opt :matchers))
>  	 (re-list org-latex-regexps)
> -	 (cnt 0) txt link beg end re e checkdir
> +	 (cnt 0) txt hash link beg end re e checkdir
>  	 executables-checked
>  	 m n block linkfile movefile ov)
> -    ;; Check if there are old images files with this prefix, and  
> remove them
> -    (when (file-directory-p todir)
> -      (mapc 'delete-file
> -	    (directory-files
> -	     todir 'full
> -	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
>      ;; Check the different regular expressions
>      (while (setq e (pop re-list))
>        (setq m (car e) re (nth 1 e) n (nth 2 e)
> @@ -14576,9 +14570,14 @@ Some of the options can be changed using  
> the variable
>  	    (setq txt (match-string n)
>  		  beg (match-beginning n) end (match-end n)
>  		  cnt (1+ cnt)
> -		  linkfile (format "%s_%04d.png" prefix cnt)
> -		  movefile (format "%s_%04d.png" absprefix cnt)
>  		  link (concat block "[[file:" linkfile "]]" block))
> +            (setq hash (sha1 (prin1-to-string
> +                              (list org-format-latex-header
> +                                    (if (boundp 'org-export-latex- 
> package-alist)
> +                                        org-export-latex-package- 
> alist)
> +                                    forbuffer txt)))
> +		  linkfile (format "%s_%s.png" prefix hash)
> +		  movefile (format "%s_%s.png" absprefix hash))
>  	    (if msg (message msg cnt))
>  	    (goto-char beg)
>  	    (unless checkdir ; make sure the directory exists
> @@ -14592,8 +14591,9 @@ Some of the options can be changed using the  
> variable
>  	       "dvipng" "needed to convert LaTeX fragments to images")
>  	      (setq executables-checked t))
>
> -	    (org-create-formula-image
> -	     txt movefile opt forbuffer)
> +            (unless (file-exists-p movefile)
> +              (org-create-formula-image
> +               txt movefile opt forbuffer))
>  	    (if overlays
>  		(progn
>  		  (mapc (lambda (o)
> -- 
> 1.6.4.73.gc144
>
>
> --=-=-=
>
>
> >
> > - Carsten
> >
> > On Nov 16, 2009, at 1:07 AM, Eric Schulte wrote:
> >
> >> Hi,
> >>
> >> The attached patch changes the latex fragment image generation so  
> that
> >> it saves images into files named by the sha1 hash of the latex  
> source
> >> code.  By checking for the existence of image files before image
> >> generation the regeneration of identical images is avoided.
> >>
> >> In practice I find that this greatly speeds up export to html and  
> the
> >> `org-preview-latex-fragment' command.
> >>
> >> Cheers -- Eric
> >>
> >> From 13e1c48fa6cac43b0c87ca0fbc8e349f7a9fa864 Mon Sep 17 00:00:00  
> 2001
> >> From: Eric Schulte <schulte.eric@gmail.com>
> >> Date: Sun, 15 Nov 2009 17:00:09 -0700
> >> Subject: [PATCH] latex fragment images cached using sha1 hash keys
> >>
> >>  Latex fragment images are now saved in files named by the sha1  
> hash
> >>  of the latex text used to create the image.  By checking if files
> >>  exist before images generation the regeneration of identical latex
> >>  images is avoided.
> >> ---
> >> lisp/ChangeLog |    6 ++++++
> >> lisp/org.el    |   18 +++++++-----------
> >> 2 files changed, 13 insertions(+), 11 deletions(-)
> >>
> >> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
> >> index 339f248..f18755c 100755
> >> --- a/lisp/ChangeLog
> >> +++ b/lisp/ChangeLog
> >> @@ -1,3 +1,9 @@
> >> +2009-11-16  Eric Schulte  <schulte.eric@gmail.com>
> >> +
> >> +	* org.el (org-format-latex): Latex images are now saved to files
> >> +	named by the sha1 hash of the latex source text avoiding
> >> +	regeneration of identical images.
> >> +
> >> 2009-11-15  Carsten Dominik  <carsten.dominik@gmail.com>
> >>
> >> 	* org-wl.el (org-wl-store-link): Handle the case that
> >> diff --git a/lisp/org.el b/lisp/org.el
> >> index bf6573b..46348fc 100644
> >> --- a/lisp/org.el
> >> +++ b/lisp/org.el
> >> @@ -14550,15 +14550,9 @@ Some of the options can be changed using
> >> the variable
> >> 	 (opt org-format-latex-options)
> >> 	 (matchers (plist-get opt :matchers))
> >> 	 (re-list org-latex-regexps)
> >> -	 (cnt 0) txt link beg end re e checkdir
> >> +	 (cnt 0) txt hash link beg end re e checkdir
> >> 	 executables-checked
> >> 	 m n block linkfile movefile ov)
> >> -    ;; Check if there are old images files with this prefix, and
> >> remove them
> >> -    (when (file-directory-p todir)
> >> -      (mapc 'delete-file
> >> -	    (directory-files
> >> -	     todir 'full
> >> -	     (concat (regexp-quote prefixnodir) "_[0-9]+\\.png$"))))
> >>     ;; Check the different regular expressions
> >>     (while (setq e (pop re-list))
> >>       (setq m (car e) re (nth 1 e) n (nth 2 e)
> >> @@ -14576,9 +14570,10 @@ Some of the options can be changed using
> >> the variable
> >> 	    (setq txt (match-string n)
> >> 		  beg (match-beginning n) end (match-end n)
> >> 		  cnt (1+ cnt)
> >> -		  linkfile (format "%s_%04d.png" prefix cnt)
> >> -		  movefile (format "%s_%04d.png" absprefix cnt)
> >> 		  link (concat block "[[file:" linkfile "]]" block))
> >> +            (setq hash (sha1 txt)
> >> +		  linkfile (format "%s_%s.png" prefix hash)
> >> +		  movefile (format "%s_%s.png" absprefix hash))
> >> 	    (if msg (message msg cnt))
> >> 	    (goto-char beg)
> >> 	    (unless checkdir ; make sure the directory exists
> >> @@ -14592,8 +14587,9 @@ Some of the options can be changed using  
> the
> >> variable
> >> 	       "dvipng" "needed to convert LaTeX fragments to images")
> >> 	      (setq executables-checked t))
> >>
> >> -	    (org-create-formula-image
> >> -	     txt movefile opt forbuffer)
> >> +            (unless (file-exists-p movefile)
> >> +              (org-create-formula-image
> >> +               txt movefile opt forbuffer))
> >> 	    (if overlays
> >> 		(progn
> >> 		  (mapc (lambda (o)
> >> --
> >> 1.6.4.73.gc144
> >>
> >> _______________________________________________
> >> Emacs-orgmode mailing list
> >> Remember: use `Reply All' to send replies to the list.
> >> Emacs-orgmode@gnu.org
> >> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> >
> > - Carsten
>
> --=-=-=--

- Carsten