Re: [PATCH] sha1 hash of latex fragments to avoid regeneration

From: "Eric Schulte" <schulte.eric@gmail.com>
To: Francesco Pizzolante <fpz@missioncriticalit.com>
Cc: mailing-list-org-mode <emacs-orgmode@gnu.org>
Subject: Re: [PATCH] sha1 hash of latex fragments to avoid regeneration
Date: Sat, 05 Dec 2009 09:35:35 -0700	[thread overview]
Message-ID: <yn43a3pqsyw.fsf@gmail.com> (raw)
In-Reply-To: <87ws15zmsm.fsf@missioncriticalit.com> (Francesco Pizzolante's message of "Wed, 02 Dec 2009 11:35:05 +0100")

Hi Francesco,

You raise good points below, and I am not sure how best to respond to
them.  My initial reaction is that you should not be checking
automatically generated files (e.g. the results of ditaa exports) into a
version control repository, however I understand that there are times
when such measures are required.

Would it be possible to switch from using org-exp-blocks to using
org-babel?  If so then you could use org-babel's caching which does
*not* affect the exported file name, but rather saves a sha1 key as
(mostly) hidden text in the org-mode buffer.

so for example

#+BEGIN_ditaa procedure.png
     +-----------------+
     |     Example     |
     |                 |
     |   +---------+   |
     |   |         |   |
     |   |         |   |
     |   +---------+   |
     +-----------------+
#+END_ditaa

would be written as

#+begin_src ditaa :file procedure.png :cache :exports results
     +-----------------+
     |     Example     |
     |                 |
     |   +---------+   |
     |   |         |   |
     |   |         |   |
     |   +---------+   |
     +-----------------+
#+end_src

and pressing C-cC-c on the block (or exporting) would result in the
insertion of a link to the resulting image into the org-mode buffer
behind a results line as follows -- only in org most of the hash is
hidden.

#+results[bdffac60833c9f925a52bd6617dace39832b7dda]:
[[file:procedure.png]]

My problem with your proposed solution is that it entails storing
information *outside* of the org-mode buffer -- which granted we are
already doing although in a less dramatic way.  The creation of external
files whose sole purpose is to save a hash in their filename strikes me
as wasteful/messy.

Best -- Eric

Francesco Pizzolante <fpz@missioncriticalit.com> writes:

> Hi,
>
>> I just pulled, reloaded, and re-ran my simple tests and the patch
>> appears to have been applied successfully.
>
> This idea of caching images is really great and works well.
>
> Nevertheless, I have a remark about the way it is implemented.
>
> When I write the following code:
>
> #+BEGIN_ditaa procedure.png
>
> I expect that:
>
> - the system generates a `procedure.png' file in order for me to check it in
>   the repository;
>
> - that the `procedure.png' file is referenced everywhere (i.e. in the exported
>   HTML code or LaTeX code).
>
> The current way it works is:
>
> 1. The `procedure_XXX.png' is committed to the repository;
>
> 2. I alter my ditaa code, and export the new document;
>
> 3. The `procedure_XXX.png' file has been removed from my working copy and a
>    new file `procedure_YYY.png' has been added in my working copy.
>
> Thus, in order to commit the new files as they are, I need to:
>
> 1. revert the `procedure_XXX.png' file, in order to delete it afterward;
>
> 2. add the `procedure_YYY.png' file to the repository.
>
> A nice solution would have been that all ditaa exports are done under the
> filename `procedure.png', and that empty files such as
> `procedure.png.ZZZ.cache' are created to "store" the fact that they have
> already been generated.
>
> That way, we even can ignore all the `*.cache' files from our working copy,
> when checking for updates.
>
> What do you think?
>
> Francesco
>
>>
>> Thanks -- Eric
>>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Hi Eric,
>>>
>>> I had a problem while pushing, please verify that the patch got in
>>> correctly.  Thanks!
>>>
>>> - Carsten
>>>
>>> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote:
>>>
>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>>
>>>>> Wow, this is fantastic!
>>>>>
>>>>> Do you think it is ready to be included (because you say first
>>>>> pass...)
>>>>>
>>>>
>>>> Yes,
>>>>
>>>> I said first pass because I had only done minimal testing.  However
>>>> all
>>>> indications are that it works, and there are no further changes I
>>>> would
>>>> like to make, so if it looks good to you I would say "yes", please
>>>> apply
>>>> it.
>>>>
>>>> Thanks -- Eric
>>>>
>>>>>
>>>>> - Carsten
>>>>>
>>>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote:
>>>>>
>>>>>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>>>>>
>>>>>>> Hi Carsten,
>>>>>>>
>>>>>>> Thanks for the feedback, I have comments inline below
>>>>>>>
>>>>>>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>>> Now, I am sure that you are already planning to do the same
>>>>>>>> for ditaa images etc?
>>>>>>>
>>>>>>> of course :)
>>>>>>
>>>>>> A first pass at a patch implementing caching of ditaa and dot images
>>>>>> generated by org-exp-blocks is attached.  It seems to work in all
>>>>>> initial tests, and it will only remove files which match the
>>>>>> following
>>>>>> pattern
>>>>>>
>>>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension"
>>>>>>
>>>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40
>>>>>> characters.
>>>>>>
>>>>>> Best -- Eric
>>>>>>
>>>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00
>>>>>> 2001
>>>>>> From: Eric Schulte <schulte.eric@gmail.com>
>>>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700
>>>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images
>>>>>> when exporting through org-exp-blocks
>>>>>>
>>>>>> ---
>>>>>> lisp/ChangeLog         |    6 ++++
>>>>>> lisp/org-exp-blocks.el |   60 +++++++++++++++++++++++++++++++++++++
>>>>>> +---------
>>>>>> 2 files changed, 54 insertions(+), 12 deletions(-)
>>>>>>
>>>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
>>>>>> index 5f83aaa..c2d44fa 100755
>>>>>> --- a/lisp/ChangeLog
>>>>>> +++ b/lisp/ChangeLog
>>>>>> @@ -1,3 +1,9 @@
>>>>>> +2009-11-17  Eric Schulte  <schulte.eric@gmail.com>
>>>>>> +
>>>>>> +	* org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1
>>>>>> +	hash keys to cache and re-use images generated by the
>>>>>> +	org-exp-blocks interface to ditaa and dot.
>>>>>> +
>>>>>> 2009-11-16  Carsten Dominik  <carsten.dominik@gmail.com>
>>>>>>
>>>>>> 	* org-html.el (org-export-html-home/up-format): Add an ID to the
>>>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el
>>>>>> index 71e1608..72fe6c4 100644
>>>>>> --- a/lisp/org-exp-blocks.el
>>>>>> +++ b/lisp/org-exp-blocks.el
>>>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be
>>>>>> saved as the first
>>>>>> element of headers, any additional elements of headers will be
>>>>>> passed to the ditaa utility as command line arguments."
>>>>>>  (message "ditaa-formatting...")
>>>>>> -  (let ((out-file (if headers (car headers)))
>>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>>> " ")))
>>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>>> +         (raw-out-file (if headers (car headers)))
>>>>>> +         (out-file-parts (if (string-match
>>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>>> $" raw-out-file)
>>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>>> +                                   (match-string 2 raw-out-file))
>>>>>> +                           (cons raw-out-file "png")))
>>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>>> out-file-parts))))
>>>>>>    (unless (file-exists-p org-ditaa-jar-path)
>>>>>>      (error (format "Could not find ditaa.jar at %s" org-ditaa-jar-
>>>>>> path)))
>>>>>>    (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body)
>>>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line
>>>>>> arguments."
>>>>>> 			    "\n")))
>>>>>>    (cond
>>>>>>     ((or htmlp latexp docbookp)
>>>>>> -      (with-temp-file data-file (insert body))
>>>>>> -      (message (concat "java -jar " org-ditaa-jar-path " " args " "
>>>>>> data-file " " out-file))
>>>>>> -      (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>>> args " " data-file " " out-file))
>>>>>> +      (unless (file-exists-p out-file)
>>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>>> +         (lambda (file)
>>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>>> file-parts))
>>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>>> \."
>>>>>> +                                            (regexp-quote (cdr out-
>>>>>> file-parts)))
>>>>>> +                                    file)
>>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>>> +             (delete-file (expand-file-name file
>>>>>> +                                            (file-name-directory
>>>>>> out-file)))))
>>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>>> +                              default-directory)))
>>>>>> +        (with-temp-file data-file (insert body))
>>>>>> +        (message (concat "java -jar " org-ditaa-jar-path " " args "
>>>>>> " data-file " " out-file))
>>>>>> +        (shell-command (concat "java -jar " org-ditaa-jar-path " "
>>>>>> args " " data-file " " out-file)))
>>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>>     (t (concat
>>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>>> @@ -259,14 +277,32 @@ digraph data_relationships {
>>>>>> }
>>>>>> #+end_dot"
>>>>>>  (message "dot-formatting...")
>>>>>> -  (let ((out-file (if headers (car headers)))
>>>>>> -	(args (if (cdr headers) (mapconcat 'identity (cdr headers) " ")))
>>>>>> -	(data-file (make-temp-file "org-ditaa")))
>>>>>> +  (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers)
>>>>>> " ")))
>>>>>> +         (data-file (make-temp-file "org-ditaa"))
>>>>>> +         (hash (sha1 (prin1-to-string (list body args))))
>>>>>> +         (raw-out-file (if headers (car headers)))
>>>>>> +         (out-file-parts (if (string-match
>>>>>> "\\(.+\\)\\.\\([^\\.]+\\)
>>>>>> $" raw-out-file)
>>>>>> +                             (cons (match-string 1 raw-out-file)
>>>>>> +                                   (match-string 2 raw-out-file))
>>>>>> +                           (cons raw-out-file "png")))
>>>>>> +         (out-file (concat (car out-file-parts) "_" hash "." (cdr
>>>>>> out-file-parts))))
>>>>>>    (cond
>>>>>>     ((or htmlp latexp docbookp)
>>>>>> -      (with-temp-file data-file (insert body))
>>>>>> -      (message (concat "dot " data-file " " args " -o " out-file))
>>>>>> -      (shell-command (concat "dot " data-file " " args " -o " out-
>>>>>> file))
>>>>>> +      (unless (file-exists-p out-file)
>>>>>> +        (mapc ;; remove old hashed versions of this file
>>>>>> +         (lambda (file)
>>>>>> +           (when (and (string-match (concat (regexp-quote (car out-
>>>>>> file-parts))
>>>>>> +                                            "_\\([[:alnum:]]+\\)\
>>>>>> \."
>>>>>> +                                            (regexp-quote (cdr out-
>>>>>> file-parts)))
>>>>>> +                                    file)
>>>>>> +                      (= (length (match-string 1 out-file)) 40))
>>>>>> +             (delete-file (expand-file-name file
>>>>>> +                                            (file-name-directory
>>>>>> out-file)))))
>>>>>> +         (directory-files (or (file-name-directory out-file)
>>>>>> +                              default-directory)))
>>>>>> +        (with-temp-file data-file (insert body))
>>>>>> +        (message (concat "dot " data-file " " args " -o " out-
>>>>>> file))
>>>>>> +        (shell-command (concat "dot " data-file " " args " -o "
>>>>>> out-
>>>>>> file)))
>>>>>>      (format "\n[[file:%s]]\n" out-file))
>>>>>>     (t (concat
>>>>>> 	 "\n#+BEGIN_EXAMPLE\n"
>>>>>> --
>>>>>> 1.6.4.73.gc144
>>>>>>
>>>>>
>>>>> - Carsten
>>>
>>> - Carsten
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode