Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

From: Francesco Pizzolante <fpz-djc/iPCCuDYQheJpep6IedvLeJWuRmrY@public.gmane.org>
To: mailing-list-org-mode <emacs-orgmode-mXXj517/zsQ@public.gmane.org>
Subject: Re: [PATCH] sha1 hash of latex fragments to avoid regeneration
Date: Wed, 02 Dec 2009 11:35:05 +0100	[thread overview]
Message-ID: <87ws15zmsm.fsf@missioncriticalit.com> (raw)
In-Reply-To: <m23a4dozic.fsf-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> (Eric Schulte's message of "Tue, 17 Nov 2009 10:02:51 -0700")

Hi,

> I just pulled, reloaded, and re-ran my simple tests and the patch
> appears to have been applied successfully.

This idea of caching images is really great and works well.

Nevertheless, I have a remark about the way it is implemented.

When I write the following code:

--8<---------------cut here---------------start------------->8---
#+BEGIN_ditaa procedure.png
--8<---------------cut here---------------end--------------->8---

I expect that:

- the system generates a `procedure.png' file in order for me to check it in
  the repository;

- that the `procedure.png' file is referenced everywhere (i.e. in the exported
  HTML code or LaTeX code).

The current way it works is:

1. The `procedure_XXX.png' is committed to the repository;

2. I alter my ditaa code, and export the new document;

3. The `procedure_XXX.png' file has been removed from my working copy and a
   new file `procedure_YYY.png' has been added in my working copy.

Thus, in order to commit the new files as they are, I need to:

1. revert the `procedure_XXX.png' file, in order to delete it afterward;

2. add the `procedure_YYY.png' file to the repository.

A nice solution would have been that all ditaa exports are done under the
filename `procedure.png', and that empty files such as
`procedure.png.ZZZ.cache' are created to "store" the fact that they have
already been generated.

That way, we even can ignore all the `*.cache' files from our working copy,
when checking for updates.

What do you think?

Francesco

>
> Thanks -- Eric
>
> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> Hi Eric,
>>
>> I had a problem while pushing, please verify that the patch got in
>> correctly.  Thanks!
>>
>> - Carsten
>>
>> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:
>>
>>> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>
>>>> Wow, this is fantastic!
>>>>
>>>> Do you think it is ready to be included (because you say first
>>>> pass...)
>>>>
>>>
>>> Yes,
>>>
>>> I said first pass because I had only done minimal testing.  However
>>> all
>>> indications are that it works, and there are no further changes I
>>> would
>>> like to make, so if it looks good to you I would say "yes", please
>>> apply
>>> it.
>>>
>>> Thanks -- Eric
>>>
>>>>
>>>> - Carsten
>>>>
>>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>>>
>>>>> "Eric Schulte" <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>>>
>>>>>> Hi Carsten,
>>>>>>
>>>>>> Thanks for the feedback, I have comments inline below
>>>>>>
>>>>>> Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>>> Now, I am sure that you are already planning to do the same
>>>>>>> for ditaa images etc?
>>>>>>
>>>>>> of course :)
>>>>>
>>>>> A first pass at a patch implementing caching of ditaa and dot images
>>>>> generated by org-exp-blocks is attached.  It seems to work in all
>>>>> initial tests, and it will only remove files which match the
>>>>> following
>>>>> pattern
>>>>>
>>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>>>
>>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>>>> characters.
>>>>>
>>>>> Best -- Eric
>>>>>
>>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00
>>>>> 2001
>>>>> From: Eric Schulte <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>>>> when exporting through org-exp-blocks
>>>>>
>>>>> ---
>>>>> lisp/ChangeLog         |    6 ++++
>>>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>>>> +---------
>>>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>>>
>>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>>>> index 5f83aaa..c2d44fa 100755
>>>>> --- a/lisp/ChangeLog
>>>>> +++ b/lisp/ChangeLog
>>>>> @@ -1,3 +1,9 @@
>>>>> +2009-11-17  Eric Schulte  <schulte.eric-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>> +
>>>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>>>> +	hash keys to cache and re-use images generated by the
>>>>> +	org-exp-blocks interface to ditaa and dot.
>>>>> +
>>>>> 2009-11-16  Carsten Dominik  <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>>
>>>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>>>> index 71e1608..72fe6c4 100644
>>>>> --- a/lisp/org-exp-blocks.el
>>>>> +++ b/lisp/org-exp-blocks.el
>>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>>>> saved as the first
>>>>> element of headers, any additional elements of headers will be
>>>>> passed to the ditaa utility as command line arguments."
>>>>>  (message "ditaa-formatting...")
>>>>> -  (let ((out-file (if headers (car headers)))
>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>> " ")))
>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>> +         (raw-out-file (if headers (car headers)))
>>>>> +         (out-file-parts (if (string-match
>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>> $" raw-out-file)
>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>> +                                   (match-string 2 raw-out-file))
>>>>> +                           (cons raw-out-file "png")))
>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>> out-file-parts))))
>>>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>>>> path)))
>>>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>>>> arguments."
>>>>> 			    "\n")))
>>>>>    (cond
>>>>>     ((or htmlp latexp docbookp)
>>>>> -      (with-temp-file data-file (insert body))
>>>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>>>> data-file " " out-file))
>>>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>> args " " data-file " " out-file))
>>>>> +      (unless (file-exists-p out-file)
>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>> +         (lambda (file)
>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>> file-parts))
>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>> \."
>>>>> +                                            (regexp-quote (cdr out-
>>>>> file-parts)))
>>>>> +                                    file)
>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>> +             (delete-file (expand-file-name file
>>>>> +                                            (file-name-directory
>>>>> out-file)))))
>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>> +                              default-directory)))
>>>>> +        (with-temp-file data-file (insert body))
>>>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>>>> " data-file " " out-file))
>>>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>> args " " data-file " " out-file)))
>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>     (t (concat
>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>>>> }
>>>>> #+end_dot"
>>>>>  (message "dot-formatting...")
>>>>> -  (let ((out-file (if headers (car headers)))
>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>> " ")))
>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>> +         (raw-out-file (if headers (car headers)))
>>>>> +         (out-file-parts (if (string-match
>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>> $" raw-out-file)
>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>> +                                   (match-string 2 raw-out-file))
>>>>> +                           (cons raw-out-file "png")))
>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>> out-file-parts))))
>>>>>    (cond
>>>>>     ((or htmlp latexp docbookp)
>>>>> -      (with-temp-file data-file (insert body))
>>>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>>>> file))
>>>>> +      (unless (file-exists-p out-file)
>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>> +         (lambda (file)
>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>> file-parts))
>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>> \."
>>>>> +                                            (regexp-quote (cdr out-
>>>>> file-parts)))
>>>>> +                                    file)
>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>> +             (delete-file (expand-file-name file
>>>>> +                                            (file-name-directory
>>>>> out-file)))))
>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>> +                              default-directory)))
>>>>> +        (with-temp-file data-file (insert body))
>>>>> +        (message (concat "dot " data-file " " args " -o " out-
>>>>> file))
>>>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>>>> out-
>>>>> file)))
>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>     (t (concat
>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>> --
>>>>> 1.6.4.73.gc144
>>>>>
>>>>
>>>> - Carsten
>>
>> - Carsten
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode-mXXj517/zsQ@public.gmane.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

-- 
________________________________________________
Francesco Pizzolante - fpz-djc/iPCCuDYQheJpep6IedvLeJWuRmrY@public.gmane.org
M i s s i o n   C r i t i c a l   I T
Phone ... +32 2-757.10.15
Fax ..... +32 2-759.27.60

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode