From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Eric Schulte" Subject: Re: [PATCH] sha1 hash of latex fragments to avoid regeneration Date: Sat, 05 Dec 2009 09:35:35 -0700 Message-ID: References: <6978E5D0-BBB7-4B45-8270-11FDE93F6726@gmail.com> <87ws15zmsm.fsf@missioncriticalit.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NGxbu-0003Er-0U for emacs-orgmode@gnu.org; Sat, 05 Dec 2009 11:35:54 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1NGxbp-0003Dc-9e for emacs-orgmode@gnu.org; Sat, 05 Dec 2009 11:35:53 -0500 Received: from [199.232.76.173] (port=35377 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NGxbp-0003DZ-3n for emacs-orgmode@gnu.org; Sat, 05 Dec 2009 11:35:49 -0500 Received: from mail-pw0-f47.google.com ([209.85.160.47]:51852) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1NGxbo-0004mu-PD for emacs-orgmode@gnu.org; Sat, 05 Dec 2009 11:35:49 -0500 Received: by pwi9 with SMTP id 9so2981289pwi.26 for ; Sat, 05 Dec 2009 08:35:47 -0800 (PST) In-Reply-To: <87ws15zmsm.fsf@missioncriticalit.com> (Francesco Pizzolante's message of "Wed, 02 Dec 2009 11:35:05 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Francesco Pizzolante Cc: mailing-list-org-mode Hi Francesco, You raise good points below, and I am not sure how best to respond to them. My initial reaction is that you should not be checking automatically generated files (e.g. the results of ditaa exports) into a version control repository, however I understand that there are times when such measures are required. Would it be possible to switch from using org-exp-blocks to using org-babel? If so then you could use org-babel's caching which does *not* affect the exported file name, but rather saves a sha1 key as (mostly) hidden text in the org-mode buffer. so for example #+BEGIN_ditaa procedure.png +-----------------+ | Example | | | | +---------+ | | | | | | | | | | +---------+ | +-----------------+ #+END_ditaa would be written as #+begin_src ditaa :file procedure.png :cache :exports results +-----------------+ | Example | | | | +---------+ | | | | | | | | | | +---------+ | +-----------------+ #+end_src and pressing C-cC-c on the block (or exporting) would result in the insertion of a link to the resulting image into the org-mode buffer behind a results line as follows -- only in org most of the hash is hidden. #+results[bdffac60833c9f925a52bd6617dace39832b7dda]: [[file:procedure.png]] My problem with your proposed solution is that it entails storing information *outside* of the org-mode buffer -- which granted we are already doing although in a less dramatic way. The creation of external files whose sole purpose is to save a hash in their filename strikes me as wasteful/messy. Best -- Eric Francesco Pizzolante writes: > Hi, > >> I just pulled, reloaded, and re-ran my simple tests and the patch >> appears to have been applied successfully. > > This idea of caching images is really great and works well. > > Nevertheless, I have a remark about the way it is implemented. > > When I write the following code: > > #+BEGIN_ditaa procedure.png > > I expect that: > > - the system generates a `procedure.png' file in order for me to check it in > the repository; > > - that the `procedure.png' file is referenced everywhere (i.e. in the exported > HTML code or LaTeX code). > > The current way it works is: > > 1. The `procedure_XXX.png' is committed to the repository; > > 2. I alter my ditaa code, and export the new document; > > 3. The `procedure_XXX.png' file has been removed from my working copy and a > new file `procedure_YYY.png' has been added in my working copy. > > Thus, in order to commit the new files as they are, I need to: > > 1. revert the `procedure_XXX.png' file, in order to delete it afterward; > > 2. add the `procedure_YYY.png' file to the repository. > > A nice solution would have been that all ditaa exports are done under the > filename `procedure.png', and that empty files such as > `procedure.png.ZZZ.cache' are created to "store" the fact that they have > already been generated. > > That way, we even can ignore all the `*.cache' files from our working copy, > when checking for updates. > > What do you think? > > Francesco > >> >> Thanks -- Eric >> >> Carsten Dominik writes: >> >>> Hi Eric, >>> >>> I had a problem while pushing, please verify that the patch got in >>> correctly. Thanks! >>> >>> - Carsten >>> >>> On Nov 17, 2009, at 4:24 PM, Eric Schulte wrote: >>> >>>> Carsten Dominik writes: >>>> >>>>> Wow, this is fantastic! >>>>> >>>>> Do you think it is ready to be included (because you say first >>>>> pass...) >>>>> >>>> >>>> Yes, >>>> >>>> I said first pass because I had only done minimal testing. However >>>> all >>>> indications are that it works, and there are no further changes I >>>> would >>>> like to make, so if it looks good to you I would say "yes", please >>>> apply >>>> it. >>>> >>>> Thanks -- Eric >>>> >>>>> >>>>> - Carsten >>>>> >>>>> On Nov 17, 2009, at 3:42 AM, Eric Schulte wrote: >>>>> >>>>>> "Eric Schulte" writes: >>>>>> >>>>>>> Hi Carsten, >>>>>>> >>>>>>> Thanks for the feedback, I have comments inline below >>>>>>> >>>>>>> Carsten Dominik writes: >>>>>> >>>>>> [...] >>>>>> >>>>>>>> Now, I am sure that you are already planning to do the same >>>>>>>> for ditaa images etc? >>>>>>> >>>>>>> of course :) >>>>>> >>>>>> A first pass at a patch implementing caching of ditaa and dot images >>>>>> generated by org-exp-blocks is attached. It seems to work in all >>>>>> initial tests, and it will only remove files which match the >>>>>> following >>>>>> pattern >>>>>> >>>>>> "beginning-of-file-name_\\([[:alnum:]]+\\)\\.extension" >>>>>> >>>>>> such that the length of the part matched by \\([[:alnum:]]+\\) is 40 >>>>>> characters. >>>>>> >>>>>> Best -- Eric >>>>>> >>>>>> From bd90d519a7d3e16732f4f377ccce51bdb97cef90 Mon Sep 17 00:00:00 >>>>>> 2001 >>>>>> From: Eric Schulte >>>>>> Date: Mon, 16 Nov 2009 19:33:11 -0700 >>>>>> Subject: [PATCH] use sha1 hash keys to cache ditaa and dot images >>>>>> when exporting through org-exp-blocks >>>>>> >>>>>> --- >>>>>> lisp/ChangeLog | 6 ++++ >>>>>> lisp/org-exp-blocks.el | 60 +++++++++++++++++++++++++++++++++++++ >>>>>> +--------- >>>>>> 2 files changed, 54 insertions(+), 12 deletions(-) >>>>>> >>>>>> diff --git a/lisp/ChangeLog b/lisp/ChangeLog >>>>>> index 5f83aaa..c2d44fa 100755 >>>>>> --- a/lisp/ChangeLog >>>>>> +++ b/lisp/ChangeLog >>>>>> @@ -1,3 +1,9 @@ >>>>>> +2009-11-17 Eric Schulte >>>>>> + >>>>>> + * org-exp-blocks.el (org-export-blocks-format-ditaa): Use sha1 >>>>>> + hash keys to cache and re-use images generated by the >>>>>> + org-exp-blocks interface to ditaa and dot. >>>>>> + >>>>>> 2009-11-16 Carsten Dominik >>>>>> >>>>>> * org-html.el (org-export-html-home/up-format): Add an ID to the >>>>>> diff --git a/lisp/org-exp-blocks.el b/lisp/org-exp-blocks.el >>>>>> index 71e1608..72fe6c4 100644 >>>>>> --- a/lisp/org-exp-blocks.el >>>>>> +++ b/lisp/org-exp-blocks.el >>>>>> @@ -217,9 +217,15 @@ Specify the path at which the image should be >>>>>> saved as the first >>>>>> element of headers, any additional elements of headers will be >>>>>> passed to the ditaa utility as command line arguments." >>>>>> (message "ditaa-formatting...") >>>>>> - (let ((out-file (if headers (car headers))) >>>>>> - (args (if (cdr headers) (mapconcat 'identity (cdr headers) " "))) >>>>>> - (data-file (make-temp-file "org-ditaa"))) >>>>>> + (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers) >>>>>> " "))) >>>>>> + (data-file (make-temp-file "org-ditaa")) >>>>>> + (hash (sha1 (prin1-to-string (list body args)))) >>>>>> + (raw-out-file (if headers (car headers))) >>>>>> + (out-file-parts (if (string-match >>>>>> "\\(.+\\)\\.\\([^\\.]+\\) >>>>>> $" raw-out-file) >>>>>> + (cons (match-string 1 raw-out-file) >>>>>> + (match-string 2 raw-out-file)) >>>>>> + (cons raw-out-file "png"))) >>>>>> + (out-file (concat (car out-file-parts) "_" hash "." (cdr >>>>>> out-file-parts)))) >>>>>> (unless (file-exists-p org-ditaa-jar-path) >>>>>> (error (format "Could not find ditaa.jar at %s" org-ditaa-jar- >>>>>> path))) >>>>>> (setq body (if (string-match "^\\([^:\\|:[^ ]\\)" body) >>>>>> @@ -229,9 +235,21 @@ passed to the ditaa utility as command line >>>>>> arguments." >>>>>> "\n"))) >>>>>> (cond >>>>>> ((or htmlp latexp docbookp) >>>>>> - (with-temp-file data-file (insert body)) >>>>>> - (message (concat "java -jar " org-ditaa-jar-path " " args " " >>>>>> data-file " " out-file)) >>>>>> - (shell-command (concat "java -jar " org-ditaa-jar-path " " >>>>>> args " " data-file " " out-file)) >>>>>> + (unless (file-exists-p out-file) >>>>>> + (mapc ;; remove old hashed versions of this file >>>>>> + (lambda (file) >>>>>> + (when (and (string-match (concat (regexp-quote (car out- >>>>>> file-parts)) >>>>>> + "_\\([[:alnum:]]+\\)\ >>>>>> \." >>>>>> + (regexp-quote (cdr out- >>>>>> file-parts))) >>>>>> + file) >>>>>> + (= (length (match-string 1 out-file)) 40)) >>>>>> + (delete-file (expand-file-name file >>>>>> + (file-name-directory >>>>>> out-file))))) >>>>>> + (directory-files (or (file-name-directory out-file) >>>>>> + default-directory))) >>>>>> + (with-temp-file data-file (insert body)) >>>>>> + (message (concat "java -jar " org-ditaa-jar-path " " args " >>>>>> " data-file " " out-file)) >>>>>> + (shell-command (concat "java -jar " org-ditaa-jar-path " " >>>>>> args " " data-file " " out-file))) >>>>>> (format "\n[[file:%s]]\n" out-file)) >>>>>> (t (concat >>>>>> "\n#+BEGIN_EXAMPLE\n" >>>>>> @@ -259,14 +277,32 @@ digraph data_relationships { >>>>>> } >>>>>> #+end_dot" >>>>>> (message "dot-formatting...") >>>>>> - (let ((out-file (if headers (car headers))) >>>>>> - (args (if (cdr headers) (mapconcat 'identity (cdr headers) " "))) >>>>>> - (data-file (make-temp-file "org-ditaa"))) >>>>>> + (let* ((args (if (cdr headers) (mapconcat 'identity (cdr headers) >>>>>> " "))) >>>>>> + (data-file (make-temp-file "org-ditaa")) >>>>>> + (hash (sha1 (prin1-to-string (list body args)))) >>>>>> + (raw-out-file (if headers (car headers))) >>>>>> + (out-file-parts (if (string-match >>>>>> "\\(.+\\)\\.\\([^\\.]+\\) >>>>>> $" raw-out-file) >>>>>> + (cons (match-string 1 raw-out-file) >>>>>> + (match-string 2 raw-out-file)) >>>>>> + (cons raw-out-file "png"))) >>>>>> + (out-file (concat (car out-file-parts) "_" hash "." (cdr >>>>>> out-file-parts)))) >>>>>> (cond >>>>>> ((or htmlp latexp docbookp) >>>>>> - (with-temp-file data-file (insert body)) >>>>>> - (message (concat "dot " data-file " " args " -o " out-file)) >>>>>> - (shell-command (concat "dot " data-file " " args " -o " out- >>>>>> file)) >>>>>> + (unless (file-exists-p out-file) >>>>>> + (mapc ;; remove old hashed versions of this file >>>>>> + (lambda (file) >>>>>> + (when (and (string-match (concat (regexp-quote (car out- >>>>>> file-parts)) >>>>>> + "_\\([[:alnum:]]+\\)\ >>>>>> \." >>>>>> + (regexp-quote (cdr out- >>>>>> file-parts))) >>>>>> + file) >>>>>> + (= (length (match-string 1 out-file)) 40)) >>>>>> + (delete-file (expand-file-name file >>>>>> + (file-name-directory >>>>>> out-file))))) >>>>>> + (directory-files (or (file-name-directory out-file) >>>>>> + default-directory))) >>>>>> + (with-temp-file data-file (insert body)) >>>>>> + (message (concat "dot " data-file " " args " -o " out- >>>>>> file)) >>>>>> + (shell-command (concat "dot " data-file " " args " -o " >>>>>> out- >>>>>> file))) >>>>>> (format "\n[[file:%s]]\n" out-file)) >>>>>> (t (concat >>>>>> "\n#+BEGIN_EXAMPLE\n" >>>>>> -- >>>>>> 1.6.4.73.gc144 >>>>>> >>>>> >>>>> - Carsten >>> >>> - Carsten >> >> >> _______________________________________________ >> Emacs-orgmode mailing list >> Remember: use `Reply All' to send replies to the list. >> Emacs-orgmode@gnu.org >> http://lists.gnu.org/mailman/listinfo/emacs-orgmode